A joint detection and tracking method based on image enhancement and image fusion

Abstract

This invention discloses a joint detection and tracking method based on image enhancement and image fusion. This invention combines traditional detection and tracking algorithms, with both tasks sharing a feature extraction network. Based on the ResNet-50 backbone feature extraction network, this invention proposes an image enhancement module, which effectively solves the problem of insufficient feature extraction in multi-target tracking. Furthermore, based on the ResNet-50 backbone feature extraction network and the image enhancement module, this invention proposes an image fusion module, which effectively fuses multi-scale features of the image, mitigating the conflict between the different feature depth requirements of detection and tracking.

Description

Technical Field

[0001] This invention relates to a video target detection method, belonging to the field of intelligent analysis of surveillance videos. Background Technology

[0002] In recent years, with the development of video surveillance technology, intelligent surveillance has received increasing attention, with people hoping that it will enhance the functionality and automation of video surveillance systems. Pedestrians and vehicles are important monitoring target categories in video surveillance systems, and the importance of these targets varies depending on the scenario. For example, in monitoring squares or airport terminals, pedestrians are the most important target. On highways, vehicles are the most important target. Therefore, intelligent monitoring technology for different targets in video is crucial. As a fundamental step in intelligent video target monitoring technology, the main purpose of video target extraction methods is to extract targets from the video and input their images into a database for later retrieval and analysis.

[0003] Generally, there are two methods for target tracking in videos. One is detection-based methods, which detect targets in images; the other is tracking-based methods, which continuously track targets in the video after they are known. Detection-based methods can be further divided into image-based detection methods and motion-based detection methods. Image-based detection methods use target features to detect targets in images, while motion-based detection methods detect changing (moving) regions in the video and use certain methods to determine whether they are specific targets. Regardless of the method, both suffer from insufficient accuracy and robustness. Summary of the Invention

[0004] To address the differences between detection and tracking tasks and the inadequacy of target feature extraction in the field of multi-target tracking, this invention proposes a joint detection and tracking method based on image enhancement and image fusion.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by the present invention includes the following steps:

[0006] Step 1: Pass the input image through the Resnet-50 basic backbone network.

[0007] Step 2: The different layer features obtained from the input image through ResNet-50 are processed by the image enhancement module.

[0008] Step 3: The different layer features from the image enhancement module are then processed by the image fusion module.

[0009] Step 4: Extract the features output by the image enhancement module through the detection branch and the appearance feature extraction branch, respectively.

[0010] Step 5: Input the appearance features obtained in Step 5 into the data association module to perform data association.

[0011] Compared with the prior art, the beneficial effects of the present invention are:

[0012] (1) Compared with traditional joint detection and tracking algorithms, it has better performance in scenes where the target is partially occluded.

[0013] (2) It can better detect and track targets with motion blur. Attached Figure Description

[0014] Figure 1 Here is the overall flowchart of the algorithm.

[0015] Figure 2 For: Algorithm image enhancement module.

[0016] Figure 3 For: Algorithm Image Fusion Module. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0018] like Figure 1 As shown in this specific embodiment, the tracking method based on the joint detection and tracking network includes the following processing steps:

[0019] Step 1: Extract basic backbone features from the input frame image.

[0020] like Figure 3 As shown, the input frame image is first processed by the ResNet-50 backbone feature extraction network. The network is divided into 5 different layers according to the feature dimension, and its output features are named C1, C2, C3, C4 and C5. The corresponding output features will be further processed in subsequent steps.

[0021] Step 2: Pass the corresponding output features through the image enhancement module.

[0022] The features from different levels output in the first step are then passed through the image enhancement module. Step 2 uses the ResNet-50 algorithm to obtain different levels of output features from the image obtained in the first step, and then passes these output features through the image enhancement module. For example... Figure 2 As shown, the image enhancement module first compresses the number of channels in the input feature layer using a 1*1*1 convolution, and then performs a linear mapping to obtain the corresponding features. Then calculate the reshaped feature layers. The covariance matrix between them, specifically, is obtained by... Transpose it and combine it with Perform a dot product operation to obtain the correlation between any pixel and all other pixels in each frame. Then, use sofmax to obtain the corresponding weighting coefficients from the output covariance matrix. The weighting coefficient With characteristic matrix The results are multiplied and combined with the input feature layer in the form of residual edges to obtain the output of the image enhancement module.

[0023] Step 3: Pass the output features of the image enhancement module through the image fusion module.

[0024] Detection tasks focus more on deep feature information, while tracking tasks focus more on shallow feature information. Therefore, fusing features at different levels is beneficial for balancing the two tasks. For example... Figure 3 As shown, the basic network feature fusion process is mainly divided into two modules: RNUP and IDAUP.

[0025] RNUP primarily performs feature fusion between modules, similar in form to residual connections. Its specific implementation process is illustrated by fusing level 2 and level 3 output features as shown in the diagram. The second output feature layer has a dimension of 128*128*256, and the third output feature layer has a dimension of 64*64*512. First, the third-layer output feature layer C3 is passed through an upsampling module. Then, C3 is sequentially passed through convolution, normalization, and a ReLU activation function, and finally through a deconvolution module to obtain the upsampled feature map. 3( The upsampling module is used to obtain... After 3, 3. The feature layer C2 output from the second layer is fused together using the add (matrix addition) method. This is the fusion process between the second-layer output feature layer C2 and the third-layer output feature layer C3. In the RNUP module, the second-layer output feature layer C2 and the third-layer output feature layer C3, the third-layer output feature layer C3 and the fourth-layer output feature layer C4, and the fourth-layer output feature layer C4 and the fifth-layer output feature layer C5 are fused in a recursive manner. The structure of the entire RNUP module is similar to an inverted pyramid structure, and the feature layers output from different dimension modules are input into the IDAUP module for further fusion between modules at different levels.

[0026] In the IDAUP module, the input is the output of the three fusion modules of RNUP. , , The output is As the output of the improved backbone feature extraction network, it also serves as the input to the subsequent detection and appearance feature extraction branches. For example... Figure 3As shown, the second-level output feature layer and the third-level output feature layer Taking this as an example, let me explain the specific fusion process in detail. First... An upsampling module is used, with a design identical to the upsampling module design in the RNUP described above. After passing through the upsampling module, the result is... Then input it with the IDAUP module. Perform the add operation (add matrix elements by element).

[0027] Step 4: Extract the output features from the image fusion module through the appearance feature extraction branch and the detection branch, respectively.

[0028] like Figure 1 As shown, the output features of the image fusion module are processed through an appearance feature extraction branch and a detection branch, respectively. The detection branch is implemented based on an anchorless detection algorithm, outputting the features from the backbone feature extraction network to three parallel heads, which are used to estimate the heatmap, object center offset, and detection box size, respectively. Each head is implemented through a 3*3*256 convolution, and finally, a 1*1 convolutional layer is used to generate the final target.

[0029] Step 5: Input the obtained appearance features into the data association module.

[0030] The obtained appearance features are input into the data association module. The cosine similarity between the appearance features of the current frame and the historical appearance features is calculated. If the highest similarity exceeds the threshold, trajectory association is performed; otherwise, a new trajectory is assigned to the object.

[0031] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this description may be replaced by other equivalent or similar alternative features unless otherwise specified. All features or steps in the disclosed methods or processes may be combined in any way, except for mutually exclusive features and / or steps.

Claims

1. A joint detection and tracking method based on image enhancement and image fusion, characterized in that, Includes the following steps: Step 1: Pass the input image through the ResNet-50 basic backbone network; Step 2: The features obtained from different layers of ResNet-50 are enhanced by the image enhancement module; Step 3: The features from different layers processed by the image enhancement module are fused using the image fusion module. The image fusion module includes an RNUP module and an IDAUP module. The RNUP module recursively fuses the output features of the second, third, fourth, and fifth layers of the ResNet-50 network, and the features output by different dimension modules are input into the IDAUP module for further fusion to obtain the output features of the backbone feature extraction network. ; Step 4: Convert the output features Apparent features are obtained by using the detection branch and the appearance feature extraction branch, respectively. Step 5: Input the appearance features obtained in step 4 into the data association module to perform data association.

2. The joint detection and tracking method based on image enhancement and image fusion according to claim 1, characterized in that, In the second step, the image enhancement module operates on the outputs of the second, third, fourth, and fifth layers of the ResNet-50 feature extraction network.

3. The joint detection and tracking method based on image enhancement and image fusion according to claim 1, characterized in that, In the third step, the RNUP module recursively fuses the output features of the second and third layers, the third and fourth layers, and the fourth and fifth layers to form an inverted pyramid structure; the IDAUP module uses the output of the RNUP module as its input.

Images