A video object detection method based on time domain information transfer

By introducing a time domain information transfer module and GRU unit in video object detection, combined with the YOLOV5 model, the problem of insufficient utilization of context information in video object detection is solved, and more efficient detection accuracy and speed are achieved.

CN114758279BActive Publication Date: 2024-11-15ANHUI UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210443675.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-24
Publication Date
2024-11-15
Estimated Expiration
2042-04-24

AI Technical Summary

Technical Problem

The prior art is difficult to effectively utilize the context information between video frames in video object detection, resulting in unsatisfactory detection accuracy and speed. Especially in short videos and live videos, traditional methods such as single frame recognition or keyframe selection have difficulty in identifying. .

Method used

A video object detection method based on time domain information transmission is designed, video frames are grouped and merged through the information transfer module, features are extracted using convolutional neural network, and context information is realized through Gate Recurrent Unit (GRU) gated loop unit. The transmission of the YOLOV5 model is combined with the object detection.

Benefits of technology

It improves the accuracy and speed of video object detection, makes full use of the time domain information between video frames, enhances detection quality and speed, especially in the processing of short videos and live videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114758279B_ABST
    Figure CN114758279B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of computer vision, and provides a video target detection method based on time domain information transmission, comprising the following steps: 1) acquiring and inputting video data; 2) dividing video frames into m groups; 3) first merging each group of continuous frames through an information transmission module, acquiring features and realizing the transmission of context information through a gated recurrent unit; 4) performing subsequent detection on the data using YOLOV5, and training the model using Focal Loss cross entropy loss and DIoU Loss function. Compared with the existing technology, the present invention improves the YOLOV5 model based on the YOLOV5 model so that it pays more attention to the information of adjacent frames in continuous frames, and can effectively improve the quality of target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical field:

[0001] The present invention relates to the fields of image processing and computer vision, and in particular to a salient object detection algorithm based on an efficient multi-scale context exploration network. Background technology:

[0002] The statements in this section merely relate to the background technology related to the present invention and do not necessarily constitute the prior art.

[0003] Object detection has always been a long-standing research direction in the field of computer vision. Object detection is also a subjective process. For humans, it is easy to locate the target through observation. But for computers that can only receive RGB pixel matrices, it is extremely difficult. Traditional object detection algorithms are not ideal in terms of prediction accuracy and speed. However, since deep learning algorithms have made great achievements in the field of computer vision, the field of object detection has gradually begun to use deep learning for research.

[0004] So far, the object detection technology based on RGB images has become mature, and has reached a high level in terms of accuracy and speed. However, there is still a lot of room for development in object detection based on videos. Most of the short videos and live videos carry characters, scenes, actions and voice content. How to use features to express their content is the key to solving such problems. One of the most direct methods is the recognition method based on a single frame, that is, to truncate the video frame and then perform deep learning based on the single frame image. However, an image is a very small part of the entire video, especially when this frame does not have a good distinction, or some images are irrelevant to the video theme, the classifier cannot perform a good recognition. Another most commonly used method is to divide the video frame into two parts: key frames and non-key frames. There are two ways to select key frames, one is to select a fixed number of frames at intervals, and the other is to select in an adaptive way. Compared with images, the biggest feature of object detection in videos is the addition of contextual information. Each frame of the video has a connection correspondence and similarity of the context. Therefore, how to learn the expression of the video in the time domain is the main factor in improving video recognition. Of course, this not only requires rich time domain information, but also relies on ordinary image-based object detection technology. Summary of the invention:

[0005] In order to solve the above problems, the present invention proposes a video target detection method and system based on time domain information transmission, designs an information transmission module, groups and merges video frames, extracts features of the merged frames through a convolutional neural network, and transmits context information through a Gate Recurrent Unit (GRU), thereby realizing the flow of time domain information in the video.

[0006] The technical solution of the present invention is to provide a video target detection method based on time domain information transmission, the method comprising the following steps:

[0007] 1. Obtain and input video data;

[0008] 1.1) Collect commonly used datasets in the field of target detection, including KITTI dataset, Waymo dataset, NuScenes dataset, BDD 100K dataset, and H3D dataset.

[0009] 1.2) Use the KITTI dataset as the detection benchmark.

[0010] 2. Cut and group the frame data in the video into m adjacent frames;

[0011] 2.1) Divide the multi-frame video data into a group according to each m adjacent frames, merge each group of data into one frame, and then perform the Focus slicing operation on it. Then perform the convolution operation with a convolution kernel size of 3×3 on the sliced ​​image.

[0012] 2.2) The feature h generated in step 3.1 is input into the GRU unit for information transfer between adjacent frames, which includes a reset gate r and an update gate z.

[0013]

[0014]

[0015] Where σ is the sigmoid function, x t is the input of the current node, h t-1 is the last transmitted state, W r and W z are the weights of the reset gate and update gate respectively.

[0016] After getting the gate signal, first use the reset gate to get the data after reset h t-1 ′, and then h t-1 ′ and input x t After concatenation, the data is scaled to the range of -1 to 1 through a Tanh activation function, and finally h′ is obtained.

[0017]

[0018] h t-1 ′=h t-1 ⊙r, (4)

[0019] Where W is the weight and ⊙ is the XOR operation.

[0020] 3. Through the information transfer module, each group of consecutive frames is first merged to obtain features and the context information is transferred through the gated recurrent unit;

[0021] 3.1) The feature y after information transfer t The first feature of the input image is obtained by sequentially performing CSP1_1 operation, CSL operation and CSP1_2 operation, as shown in the following formula.

[0022] Y=CSP(CLB(y t ))where CLB=Conv+BN+ReLU, (5)

[0023] Where CSP stands for feature fusion.

[0024] 3.2) The first feature is then used to obtain the second feature of the input feature.

[0025] 3.3) The obtained second feature is sequentially subjected to CBL operation, SSP operation, and CSP2_1 operation to obtain the third feature of the input feature.

[0026] 3.4) The third feature is upsampled and concatenated with the second feature, and then subjected to CSP2_1 and CBL operations to obtain the fourth feature of the input feature.

[0027] 3.5) The fourth feature is upsampled and concatenated with the first feature, and then subjected to CSP2_1 operation to obtain the fifth feature of the input feature. The fifth feature is subjected to Conv convolution operation to obtain the sixth feature map, as shown in the following formula.

[0028] Y′=F(Concat, CSP, Conv), (6)

[0029] The F function represents feature extraction, and Concate represents concatenation.

[0030] 3.6) Input the fifth feature obtained into formula (6) to obtain the eighth feature map.

[0031] 3.7) The obtained seventh feature is used through formula (6) to obtain the ninth feature to complete target detection and target selection.

[0032] 4. Use YOLOV5 to conduct subsequent testing on the data;

[0033] 4.1) At the output end, the loss function of the target Bounding Box is calculated for the ninth feature obtained, and the DIoU method is used to suppress the maximum value.

[0034] 4.2) Focal Loss cross entropy loss function is used for the loss generated by classification.

[0035] FL(p t )=-(1-p t ) γ log(p t ), (7)

[0036] where 1-p t is a variable balancing factor, and γ is an adjustment factor and is greater than zero.

[0037] 4.3) The DIoU Loss loss function is used for the loss generated by position regression.

[0038]

[0039] Where ρ represents the distance between the center point of the predicted box and the real box, p and g are the center points of the two boxes, and c represents the diagonal length of the minimum bounding box of the two boxes.

[0040] Based on the above, the information transmission module consists of three parts. The Focus layer improves the receptive field of each point and reduces the loss of original information. The convolution layer (Conv) extracts features from the previous information. The GRU layer updates data through the reset gate and update gate to achieve information transmission between adjacent frames, so as to improve detection accuracy and speed.

[0041] Advantages of the present invention: The present invention makes full use of GRU to realize the gradient problem of long-term memory and back propagation. First, the state h transmitted from the previous t-1 and the current input x t To obtain the status of the update gate and the reset gate, the time domain information in the continuous video frames can be utilized to improve the quality and speed of target detection. Description of the drawings:

[0042] Figure 1 Network Flowchart

[0043] Figure 2 GRU Module

[0044] Figure 3 YOLOV5 model

[0045] Figure 4 Improving the YOLOV5 model Specific implementation method:

[0046] The following will be combined with the drawings in the examples of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. In addition, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this research direction without creative work are within the scope of protection of the present invention.

[0047] The flowchart framework of the present invention is as follows Figure 1 As shown, the present invention provides a video target detection method based on time domain information transmission, and its specific operation is described as follows:

[0048] 1. Obtain and input video data

[0049] Collect public video data sets, set training data and test data, and use Mosaic to perform data enhancement processing such as random scaling and cropping on the training data to enhance the robustness of the network.

[0050] The specific steps are as follows:

[0051] 1.1 Collect commonly used data sets in the field of target detection, including KITTI data set, Waymo data set, NuScenes data set, BDD 100K data set, and H3D data set.

[0052] 1.2 Use the KITTI dataset as the detection benchmark.

[0053] 2. Cut and group the frame data in the video into m adjacent frames

[0054] Since video target detection has rich contextual connections and similarities compared to image target detection, the detection results of previous and next frames can be used to improve the detection accuracy of the current frame; since previous and next frames have similar continuity, the detection speed of each frame can be accelerated by using redundant information. The present invention adds a GRU unit to the original YOLOV5 model to solve this problem by transferring information between adjacent frames. Figure 4 shown.

[0055] The specific steps are as follows:

[0056] 2.1 Divide the multi-frame video data into a group according to each m adjacent frames, merge each group of data into one frame, and then perform the Focus slicing operation on it. Then perform the convolution operation with a convolution kernel size of 3×3 on the sliced ​​image.

[0057] 2.2 The feature h generated in step 3.1 is input into the GRU unit to transfer information between adjacent frames, such as Figure 2 As shown in Figure 1, it includes a reset gate r and an update gate z. Figure 2 shown.

[0058]

[0059]

[0060] Where σ is the sigmoid function, x t is the input of the current node, h t-1 is the last transmitted state, W r and W z are the weights of the reset gate and update gate respectively.

[0061] After getting the gating signal, first use the reset gating to get the data after reset h t-1 ′, and then h t-1 ′ and input x t After concatenation, a tanh activation function is used to scale the data to the range of -1 to 1, and finally h′ is obtained.

[0062]

[0063] h t-1′ =h t-1 ⊙r, (4)

[0064] Where W is the weight and ⊙ is the XOR operation.

[0065] 3. Through the information transfer module, each group of consecutive frames is first merged to obtain features and the context information is transferred through the gated recurrent unit.

[0066] Based on YOLOV5 Figure 3 As shown, it has high performance in target detection and still maintains its original structure for subsequent feature processing.

[0067] The specific steps are as follows:

[0068] 3.1 The feature y after information transfer t The first feature of the input image is obtained by sequentially performing CSP1_1 operation, CSL operation and CSP1_2 operation, as shown in the following formula.

[0069] Y=CSP(CLB(y t ))where CLB=Conv+BN+ReLU, (5)

[0070] Where CSP stands for feature fusion.

[0071] 3.2 The first feature is obtained by using formula (5) to obtain the second feature of the input feature.

[0072] 3.3 The obtained second feature is subjected to CBL operation, SSP operation, and CSP2_1 operation in sequence to obtain the third feature of the input feature.

[0073] 3.4 The third feature is upsampled and concatenated with the second feature, and then subjected to CSP2_1 and CBL operations to obtain the fourth feature of the input feature.

[0074] 3.5 The fourth feature is upsampled and concatenated with the first feature, and then subjected to CSP2_1 operation to obtain the fifth feature of the input feature. The fifth feature is subjected to Conv convolution operation to obtain the sixth feature map, as shown in the following formula.

[0075] Y′=F(Concat, CSP, Conv), (6)

[0076] The F function represents feature extraction, and Concate represents concatenation.

[0077] 3.6 Input the fifth feature obtained into formula (6) to obtain the eighth feature map.

[0078] 3.7 The seventh feature is obtained through formula (6) to obtain the ninth feature to complete target detection and target selection.

[0079] 4. Use YOLOV5 to perform subsequent testing on the data.

[0080] The specific steps are as follows:

[0081] 4.1 At the output end, the loss function of the target Bounding Box is calculated for the ninth feature obtained, and the DIoU method is used to suppress the maximum value.

[0082] 4.2 The Focal Loss cross entropy loss function is used for the loss generated by classification.

[0083] FL(p t )=-(1-p t ) γ log(p t ), (7)

[0084] where 1-p t is a variable balancing factor, and γ is an adjustment factor and is greater than zero.

[0085] 4.3 The DIoU Loss loss function is used for the loss generated by position regression.

[0086]

[0087] The above is for the preferred implementation of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video target detection method based on time domain information transmission, characterized in that: The method comprises the following steps: 1) Input video data; 2) Cut and group the frame data in the video into m adjacent frames; 3.1) Divide the multi-frame video data into a group according to every m adjacent frames, merge each group of data into one frame, perform Focus slicing operation on it, and then perform convolution operation with a convolution kernel size of 3×3 on the sliced ​​image; 3.2) The feature h generated in step 3.1 is input to the GRU unit for information transfer between adjacent frames, which includes a reset gate r and an update gate z. Where σ is the sigmoid function, x t is the input of the current node, h t-1 is the last transmitted state, W r and W z They are the weights of the reset gate and the update gate respectively; After getting the gating signal, first use the reset gating to get the data after reset h t-1′ , and then h t-1′ With input x t After concatenation, a tanh activation function is used to scale the data to the range of -1 to 1, and finally h′ is obtained. h t-1′ =h t-1 ⊙r, (4) Where W is the weight, ⊙ is the XOR operation; 3) Obtain information from adjacent frames through information transfer mechanism to enhance the detection of the current frame; 4.1) The feature y after information transfer t The first feature of the input image is obtained by sequentially performing CSP1_1 operation, CBL operation and CSP1_2 operation, as shown in the following formula: Y = CPS (CBL (y t ))where CBL=Conv+BN+ReLU, (5)where CSP represents feature fusion; 4.2) The first feature is then used to obtain the second feature of the input feature through formula (5); 4.3) The obtained second feature is sequentially subjected to CBL operation, SPP operation, and CSP2_1 operation to obtain the third feature of the input feature; 4.4) The third feature is subjected to an upsampling operation and concatenated with the second feature, and then subjected to a CSP2_1 operation and a CBL operation to obtain a fourth feature of the input feature; 4.5) The fourth feature is upsampled and concatenated with the first feature, and then subjected to CSP2_1 operation to obtain the fifth feature of the input feature. The fifth feature is convolved to obtain the sixth feature map, as shown in the following formula: Y′=F(Concat, CSP, Conv), (6) The F function represents feature extraction, and Concate represents concatenation; 4.6) Input the obtained sixth feature into formula (6) to obtain the seventh feature map; 4.7) The obtained seventh feature is used through formula (6) to obtain the eighth feature to complete target detection and target selection; 4) Use the YOLOV5 model to further detect the processed frame data and obtain the detection results; 5.1) At the output end, the loss function of the target Bounding Box is calculated for the ninth feature obtained, and the DIoU method is used to suppress the maximum value; 5.2) For the loss generated by classification, the Focal Loss cross entropy loss function is used. FL(p t )=-(1-p t ) γ log(p t ), (7) where 1-p t is a variable balance factor, γ is an adjustment factor and is greater than zero; 5.3) The DIoU Loss function is used for the loss caused by position regression. Where ρ represents the distance between the center point of the predicted box and the real box, p and g are the center points of the two boxes, and c represents the diagonal length of the minimum bounding box of the two boxes.

2. The video target detection method based on time domain information transmission according to claim 1 is characterized in that: The specific method of step 1) is: 2.1) Collect commonly used datasets in the field of target detection; 2.2) Use the KITTI dataset as the detection benchmark.