A data enhancement and frame selection method for multi-frame two-dimensional object detection tasks

Abstract

The invention discloses a data enhancement and frame selection method for multi-frame two-dimensional target detection tasks, belonging to the technical field of computer vision. The method comprises: constructing a feature extraction model and a target detection model, selecting historical frames in sequence for current video frames and performing data enhancement, and performing feature extraction on the historical frames after data enhancement to obtain feature vectors; then using a clustering algorithm to group the feature vectors and select reference frames; using the reference frames and the current frames as input data, inputting them into a target detection model for training and continuous iteration; image frame selection can adaptively select more effective historical frames, thereby providing a powerful reference for the target detection process of the current frame and improving the final detection accuracy; image enhancement improves the model's ability to extract information from historical frames through random data enhancement, thereby improving the accuracy of the final classification and positioning of the target object.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a data enhancement and frame selection method for multi-frame two-dimensional target detection tasks. Background Art

[0002] With the development of deep learning, fields such as autonomous driving, smart transportation, and smart security are gradually being realized. Object detection technology, especially object detection in video sequences, has become particularly important. It aims to locate and classify key objects in the picture (such as vehicles, pedestrians, and bicycles). Video object detection faces multiple challenges:

[0003] Autonomous driving: The vehicle is equipped with a high-definition camera to collect environmental images in real time. Due to position and size limitations, the images are often blocked, resulting in missing features of the target object, increasing the difficulty of detection.

[0004] Smart Transportation: Cameras are installed at high positions to collect vehicle information and detect illegal behaviors. On expressways or highways, vehicles travel at high speeds (over 100 kilometers per hour), which causes motion blur in the images and increases the difficulty of detection.

[0005] Smart security: In crowded areas, cameras are used to monitor and collect information about the flow of people. Due to the dense and fast movement of target objects and the movement of the camera, the image may be out of focus or distorted, increasing the difficulty of feature extraction.

[0006] To solve the problems of occlusion, blur and out-of-focus, researchers use the continuous characteristics of video frames to improve detection accuracy. The video captured by the camera is a continuous two-dimensional image sequence with 10 to 30 frames per second. These frames are input into the neural network in sequence for target detection. When a frame is occluded, the previous frame may be more complete. Therefore, using historical frames as references to improve the detection accuracy of the current frame is a key issue in video target detection.

[0007] Existing methods usually focus more on how to design a fusion method for multiple frames of images, how to associate target objects in different frames, or how to post-process the results of multiple frame detection, but tend to ignore the important issue of how to select image frames in the sequence. Summary of the invention

[0008] In order to solve the above technical problems, the present invention proposes a data enhancement and frame selection method for multi-frame two-dimensional target detection tasks to solve the problems existing in the above-mentioned prior art.

[0009] To achieve the above object, the present invention provides a data enhancement and frame selection method for a multi-frame two-dimensional target detection task, comprising the following steps:

[0010] Constructing a target detection network and a neural network model for feature extraction, inputting a pre-trained data set into the neural network model for iterative training, and obtaining a trained feature extraction network;

[0011] Setting the number of historical video frames to be selected, selecting historical video frames in order according to the set number, and then performing data enhancement on the selected historical video frames;

[0012] Inputting the enhanced historical video frames into the trained neural network model to extract features, and obtaining a feature vector for each historical video frame;

[0013] The feature vectors are grouped, and a feature vector with a sequence number closest to the current video frame is selected from each group, and a frame corresponding to the feature vector closest to the current video frame is used as a reference frame of the current video frame to obtain a reference frame set;

[0014] Inputting the reference frame set and the current video frame into the target detection network to perform target detection and obtain a target detection result;

[0015] Based on the target detection results, feedforward-backward propagation calculations are performed on the target detection network to obtain a target detection model;

[0016] Valid historical video frames are selected through the target detection model.

[0017] Preferably, the method for constructing a neural network model for feature extraction includes:

[0018] Construct basic convolution, pooling, and fully connected layers, combine the convolution, pooling, and fully connected layers to obtain a feature extraction network part and a pre-training task part, and splice the feature extraction network with the pre-training task part to obtain a neural network model for feature extraction.

[0019] Preferably, the method for performing iterative training includes:

[0020] Perform feedforward inference on the neural network model through historical data, compare the feedforward inference result with the real annotation, obtain the classification loss value, sum the classification loss value and perform back propagation to obtain the weight gradient, and perform gradient descent based on the weight gradient, iterate according to the set number of training times, and obtain the initial model;

[0021] The initial model is segmented, the fully connected layer and the classification task layer are removed, and the backbone network is retained to obtain a trained feature extraction network.

[0022] Preferably, the method for performing data enhancement on the selected historical video frames comprises:

[0023] Performing Color Jitter data enhancement on the historical video frame to obtain enhanced data of the historical video frame;

[0024] Adjust the pixel value of the historical video frame enhancement data so that the adjusted pixel value is within the range of 0 to 255;

[0025] The historical video frames after pixel adjustment are enhanced twice by using the Dropout method to obtain enhanced historical video frames.

[0026] Preferably, the data enhancement method in the Color Jitter mode includes: performing brightness transformation, contrast transformation, saturation transformation and hue transformation on the image.

[0027] Preferably, the method for obtaining a reference frame set comprises:

[0028] Taking N feature vectors as data points, K-Means randomly selects n initial centers, and then assigns each data point to the nearest center to form n groups;

[0029] Calculate the average value of each group as the new center, iterate until convergence, and obtain the final grouping;

[0030] The frame corresponding to the first feature vector in each group of the final grouping is selected as the reference frame, and a total of n frames are selected to form a reference frame set.

[0031] Preferably, before inputting the reference frame set and the current video frame into the object detection network, the reference frame set and the current video frame need to be sequentially spliced ​​in terms of the channel dimension.

[0032] Preferably, the method for performing feedforward-backward propagation calculation includes:

[0033] The spliced ​​video frames are input into the target detection network, and the prediction results are obtained through network layer calculations. The loss value is calculated based on the prediction results and the true labels. The target detection network parameters are updated through the back propagation algorithm to obtain the target detection model.

[0034] Compared with the prior art, the present invention has the following advantages and technical effects:

[0035] First, the present invention uses a clustering-based frame selection strategy CFS to perform more effective historical frame selection to better assist the current frame in target detection, solving the problem that the current method ignores the important issue of how to select image frames in a sequence. At the same time, a random historical frame enhancement strategy is used to perform Color Jitter and Dropout enhancement on all frames within the selected frame range, and the model's ability to extract information from historical frames is improved through random data enhancement, thereby helping the model improve the accuracy of the final classification and positioning of target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 is a schematic diagram of the structure of the method of an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a data enhancement process according to an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of a frame selection process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Embodiment 1

[0043] like Figure 1 As shown, this embodiment provides a data enhancement and frame selection method for a multi-frame two-dimensional target detection task. The experimental dataset in this embodiment uses ImageNet VID 2015 and Wanji video datasets.

[0044] Step 1: Build a neural network model for feature extraction, input the pre-trained data set into the model, perform multiple iterative calculations, and obtain the pre-trained feature extraction network;

[0045] Step 1.1: Construct basic convolution, pooling, and fully connected layers, and then combine the multi-layer network layers to obtain the feature extraction network part and the pre-training task head part, splice the feature extraction network with the pre-training task part (the pre-training task in the present invention is a classification task) to obtain a visual image classification model.

[0046] Step 1.2: Read data from the ImageNet dataset, sample a certain amount of images each time as a batch of data, input into the model obtained in step 1.1, perform feedforward inference, and compare the inference results with the true annotations. For the classification results, use the binary cross entropy algorithm to obtain the classification loss value. Sum the loss values ​​and perform backpropagation, calculate the gradient of each weight in the model, and perform gradient descent to complete one iteration. Repeat step 1.2, complete multiple iterations, and obtain a pre-trained classification model;

[0047] Step 1.3: Segment the generated model, remove the fully connected layer and classification task layer, retain the backbone network as the feature extraction function, input the final image data into the model, and output its feature vector;

[0048] Step 2 (such as Figure 2 ): Set the number of historical frames to be selected, select historical video frames in order for the current video frame, and then perform data enhancement through the Color Jitter and Dropout methods;

[0049] Step 2.1: Get the current video frame image t. Assume that there are N historical frames in total. Then select the last historical frame t-1 and the second last historical frame t-2 in order until the tNth historical frame is obtained in sequence. Perform data enhancement on the N historical frames t-1 to tN respectively.

[0050] Step 2.2: Perform Color Jitter data augmentation on N historical frames, and randomly select one of the following methods for data augmentation:

[0051] Brightness transformation. First, convert the image from RGB color space to HSV color space, adjust the brightness channel (V channel) in HSV color space, and change the image brightness by multiplying the value of the brightness channel by a random number (usually 0-2). The new calculation formula for the brightness value is: V new =V old × scaling factor. Finally, convert the adjusted image from HSV color space back to RGB color space.

[0052] Contrast transformation. Calculate the average value of each channel (R, G, B) of the image to get the average brightness. Adjust each pixel value of the image to increase or decrease the difference relative to the average brightness through the formula:

[0053] I new =I mean +(I old -I mean )×contrast scaling factor

[0054] in N is the total number of pixels in the image, I i is the brightness value of each pixel. The contrast scaling factor is a random number between 0 and 2. Finally, the adjusted image is converted from the HSV color space back to the RGB color space.

[0055] Saturation conversion. Convert the image from RGB color space to HSV color space, scale the saturation channel (S channel) in the HSV color space, and change the image saturation by multiplying the value of the saturation channel by a random number between 0 and 2. The new calculation formula for the saturation value is: S new =S old × scaling factor.

[0056] Finally, convert the adjusted image from HSV color space back to RGB color space.

[0057] Hue transformation. First, convert the image from RGB color space to HSV color space, and adjust the hue channel (H channel) in HSV color space. The hue can be changed by adding or subtracting a random angle value. Specifically, the new calculation formula for the hue value is: H new =(H old +ΔH) mod 180, where ΔH is a randomly generated hue value. Finally, the adjusted image is converted from the HSV color space back to the RGB color space.

[0058] No changes are made to the image.

[0059] Step 2.3: Make sure the adjusted pixel values ​​are in the range of 0 to 255 to avoid exceeding the representable color range,

[0060] I new =min(max(I new ,0),255)

[0061] Among them I new is the adjusted image;

[0062] Step 2.4: After the historical frames adjusted by the Color Jitter method in step 2.2 and the pixel values ​​adjusted in step 2.3, the Dropout method is applied to them. First, a mask matrix with the same size as the image is generated. Each element in the matrix is ​​a randomly generated binary value (0 or 1). These values ​​can be generated according to the set Dropout probability.

[0063] The generation of the mask matrix can be achieved by the following steps:

[0064] 1) For each pixel position, generate a random number between 0 and 1. If the random number is less than the set Dropout probability, the mask value of the position is set to 0 (indicating that the pixel is discarded); otherwise, it is set to 1 (indicating that the pixel is retained).

[0065] 2) Apply the generated Dropout mask to the image, setting the pixel values ​​at positions where the mask value is 0 to black.

[0066] Step 3: Let X be the historical frame sequence after data enhancement, X = {X 1 ,X 2 ,…,X n}, where Xi represents the i-th historical video frame. These enhanced historical frames are input into the neural network model f(·) for feature extraction to obtain the feature vector F of each historical video frame.

[0067] F i =f(X i ), i=1,2,…,n

[0068] The feature vector of the entire historical frame sequence is expressed as: F = {F 1 ,F 2 ,…,F n};

[0069] Step 4 (such as Figure 3 ): Set the specified number of groups to perform feature vector F on all historical frames in this round 1 ,F 2 ,…,F n}, use the K-means clustering method to group them, and each group obtained by clustering selects the frame corresponding to the feature vector closest to the current video frame as the reference frame of the current video frame;

[0070] Step 4.1: Assume that the number of groups is n, and use the K-Means clustering algorithm to cluster all the feature vectors F = {F 1 ,F 2 ,…,F n}Clustering is performed, and the clustering steps are as follows:

[0071] 1) Initial center selection: Randomly select n feature vectors as the initial clustering centers {C 1 ,C 2 ,…,C k}.

[0072] 2) Assign samples to the nearest center: Assign each feature vector to the nearest cluster center to form n clusters, and assign samples to the nearest center:

[0073]

[0074] 3) Update the cluster center: Calculate the average value of all feature vectors in each cluster and use the average value as the new cluster center:

[0075]

[0076] 4) Iteration: Repeat the steps of assigning and updating centers until the cluster centers no longer change or change very little, reaching convergence.

[0077] Step 4.2: After clustering is completed, traverse each cluster and select the frame corresponding to the feature vector closest to the current video frame from each cluster as the reference frame of the current video frame. The reference frame set obtained is R = {R 1 ,R 2 ,…,R k};

[0078] Step 5: Concatenate the reference frame set R with the current video frame in the channel dimension, and input the concatenated result into the target detection network for target detection.

[0079] Step 5.1: Concatenate the current frame t and the reference frame R in the channel dimension. Assuming that the size of each frame is H×W×C, where H is the height, W is the width, and C is the number of channels, the size of the concatenated input data will be H×W×(C×(n+1)), where n is the number of reference frames.

[0080] Step 5.2: Assume that the concatenated input is X input =Concat(t,R 1 ,R 2 ,…,R n ), where Concat is a concatenation operation;

[0081] Step 6: Perform feedforward-backward propagation calculations to train the target detection model network;

[0082] Step 6.1: Place X input Feedforward inference is performed as the input of the model, and the prediction results include the category and location of the target object.

[0083] Step 6.2: Compare the model's prediction results with the true labels and calculate the loss function. The loss function usually includes two parts: classification loss and regression loss.

[0084] Step 6.3: Through the back propagation algorithm, the gradient of the loss function is transferred from the output layer to the input layer layer by layer, and the back propagation calculates the gradient of each parameter to the loss to update the parameters;

[0085] Step 7: When the model training is completed, end the training, otherwise use the new two-dimensional video frame as training data and go to step 2. The criterion for completing the training is that the loss function of the target detection model is lower than a certain value or does not change much; select the valid historical video frame through the target detection model.

[0086] This method first selects multiple historical frames from the current frame in sequence and performs data enhancement. It then extracts features from the historical frames after data enhancement to obtain feature vectors for each historical video frame. It uses the K-Means clustering algorithm to group the feature vectors and selects a reference frame from each group. These reference frames are then used together with the current frame as input data and input into the target detection model for training. For multi-frame two-dimensional target detection tasks, this method increases the diversity of training data by introducing data enhancement and frame selection strategies, and selects representative reference frames through clustering algorithms, providing more contextual information to help the model obtain more accurate results in the target detection process of the current frame. During the model training process, the model parameters are gradually adjusted by optimizing the loss function so that it can maintain high detection accuracy in various complex scenarios.

[0087] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data enhancement and frame selection method for multi-frame two-dimensional object detection tasks, characterized in that: The following steps are involved: Constructing a target detection network and a neural network model for feature extraction, inputting a pre-trained data set into the neural network model for iterative training, and obtaining a trained feature extraction network; Setting the number of historical video frames to be selected, selecting historical video frames in order according to the set number, and then performing data enhancement on the selected historical video frames; The method for performing data enhancement on the selected historical video frames comprises: Performing Color Jitter data enhancement on the historical video frame to obtain enhanced data of the historical video frame; Adjust the pixel value of the historical video frame enhancement data so that the adjusted pixel value is within the range of 0 to 255; The historical video frames after pixel adjustment are enhanced twice by using the Dropout method to obtain enhanced historical video frames; Inputting the enhanced historical video frames into the trained neural network model to extract features, and obtaining a feature vector for each historical video frame; The feature vectors are grouped, and a feature vector with a sequence number closest to the current video frame is selected from each group, and a frame corresponding to the feature vector closest to the current video frame is used as a reference frame of the current video frame to obtain a reference frame set; The method for obtaining a reference frame set comprises: Taking N feature vectors as data points, K-Means randomly selects n initial centers, and then assigns each data point to the nearest center to form n groups; Calculate the average value of each group as the new center, iterate until convergence, and obtain the final grouping; Select the frame corresponding to the first feature vector from each group in the final grouping as the reference frame, and select n frames in total to form a reference frame set; Inputting the reference frame set and the current video frame into the target detection network to perform target detection and obtain a target detection result; Based on the target detection results, feedforward-backward propagation calculations are performed on the target detection network to obtain a target detection model; Valid historical video frames are selected through the target detection model.

2. The data enhancement and frame selection method for multi-frame two-dimensional target detection tasks according to claim 1, characterized in that: Methods for building a neural network model for feature extraction include: Construct basic convolution, pooling, and fully connected layers, combine the convolution, pooling, and fully connected layers to obtain a feature extraction network part and a pre-training task part, and splice the feature extraction network with the pre-training task part to obtain a neural network model for feature extraction.

3. The data enhancement and frame selection method for multi-frame two-dimensional target detection tasks according to claim 1, characterized in that: The method for performing iterative training comprises: Perform feedforward inference on the neural network model through historical data, compare the feedforward inference result with the real annotation, obtain the classification loss value, sum the classification loss value and perform back propagation to obtain the weight gradient, and perform gradient descent based on the weight gradient, iterate according to the set number of training times, and obtain the initial model; The initial model is segmented, the fully connected layer and the classification task layer are removed, and the backbone network is retained to obtain a trained feature extraction network.

4. The data enhancement and frame selection method for multi-frame two-dimensional target detection tasks according to claim 1, characterized in that: The data enhancement method of the Color Jitter method includes: performing brightness transformation, contrast transformation, saturation transformation and hue transformation on the image.

5. The data enhancement and frame selection method for multi-frame two-dimensional target detection tasks according to claim 1, characterized in that: Before inputting the reference frame set and the current video frame into the target detection network, the reference frame set and the current video frame need to be sequentially spliced ​​in terms of the channel dimension.

6. The data enhancement and frame selection method for multi-frame two-dimensional target detection tasks according to claim 5, characterized in that: Methods for performing feedforward-backward propagation calculations include: The spliced ​​video frames are input into the target detection network, and the prediction results are obtained through network layer calculations. The loss value is calculated based on the prediction results and the true labels. The target detection network parameters are updated through the back propagation algorithm to obtain the target detection model.

Images