Cooperative repair guidance-based occlusion target identification system and method

Abstract

The invention belongs to the technical field of computer vision and deep learning, and discloses an occlusion target recognition system based on collaborative repair guidance, which comprises a gating dual-channel repair network, a multi-scale feature fusion detection network, an occlusion image and mask input gating dual-channel repair network, a multi-scale feature fusion detection network and a multi-scale feature fusion detection network, image restoration is carried out through a global path and a local path in the gated dual-path restoration network; and the repaired image is input into the multi-scale feature fusion detection network, a detection result is output through a feature injection module and a feature fusion module in the multi-scale feature fusion detection network, the detection result is fed back to the gated dual-channel repair network, and the repair network is guided to carry out training optimization in a direction beneficial to target detection. Through the joint training strategy and the feedback loss, the whole system can learn how to repair for detection in an end-to-end mode, the synergistic effect that 1 plus 1 is larger than 2 is achieved, and extremely high robustness and accuracy are shown in a severe shielding scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and relates to an occluded target recognition system and method based on collaborative repair guidance, which is used to solve the problem of severe target occlusion in complex scenes. By fusing an improved image repair network and an improved target detection network, it achieves accurate end-to-end occluded target recognition. Background Technology

[0002] Currently, deep learning-based object detection algorithms (such as YOLO and DETR) have achieved excellent performance on publicly available datasets. However, in practical applications (such as traffic monitoring, robot navigation, and dense crowd analysis), mutual occlusion between objects or between objects and the scene is one of the main reasons for the sharp decline in model performance. Occlusion causes the loss of object feature information, introduces a large amount of background noise, and causes the detector to miss detections or have localization errors.

[0003] There are two main directions for existing solutions: 1. Improving model robustness by simulating occlusion through data augmentation: This involves artificially adding occlusions to the training data (data augmentation) or introducing attention mechanisms (such as Transformer or SE modules) to teach the model to focus on the unoccluded parts of the target. These methods are considered "passive adaptation" and do not recover lost information, thus having limited effectiveness against severe occlusion.

[0004] 2. Image Inpainting Before Detection: This approach employs image inpainting or completion techniques, such as pix2pix or GAN-based models, to first repair occluded areas before feeding the repaired image into the detector. This two-stage method has significant drawbacks: the inpainting process is independent and blind, and its optimization objectives (such as pixel-level L1 loss or perceptual loss) are inconsistent with the objectives of higher-level tasks (object detection). This can lead to visually plausible but ultimately unhelpful image textures, or even misleading results. Furthermore, errors accumulate between the two stages, and the overall computational efficiency is low.

[0005] Therefore, there is an urgent need for an end-to-end framework that can tightly couple and collaboratively optimize image inpainting and target detection, so that the inpainting process is driven and guided by the detection task, thereby fundamentally improving the recognition accuracy of occluded targets. Summary of the Invention

[0006] (a) Purpose of the invention The purpose of this invention is to address the problem of significantly reduced target recognition accuracy in complex scenarios where targets are severely occluded. This invention proposes an occluded target recognition system and method based on collaborative repair guidance. By employing a repair-detection bidirectional guidance mechanism, deep feature fusion, and end-to-end collaborative optimization, the invention solves the problem of low target recognition accuracy in heavily occluded scenarios.

[0007] (II) Technical Solution To address the aforementioned technical problems, this invention provides an occluded target recognition system based on collaborative repair guidance. The system includes a gated dual-path repair network and a multi-scale feature fusion detection network. The occluded image and mask are input into the gated dual-path repair network, where image repair is performed through global and local pathways. The repaired image is then input into the multi-scale feature fusion detection network, passing through a feature injection module and a feature fusion module to output detection results. These results are simultaneously fed back to the gated dual-path repair network, guiding its training and optimization in a direction conducive to target detection.

[0008] This invention also provides a method for identifying occluded targets based on collaborative repair guidance, which includes the following steps: S1: Construct a simulation occlusion dataset; S2: Construct a gated dual-path repair network; S3: Construct a multi-scale feature fusion detection network; S4: Design collaborative training and optimization strategies; S5: Inference output results.

[0009] Further, in step S1, a public dataset containing rich targets is obtained, and an irregular occlusion region is generated on the target instance in the image using a binary random mask generation algorithm, forming a four-tuple training sample pair of "original image-occluded image-binary occlusion mask-ground annotation".

[0010] Furthermore, in step S1, the proportion of the occlusion area of ​​the irregular occlusion area is 10% to 40%.

[0011] Furthermore, in step S2, the process of constructing the gated dual-path repair network is as follows: First, occlude the image and the corresponding binary occlusion mask As input; Secondly, depthwise separable convolutions are used to replace the conventional convolutions in the U-Net encoder to construct a global path. The entire global path is responsible for capturing the global contextual semantic information of the image and performing preliminary and smooth restoration on the entire image. Then, a lightweight network consisting of depthwise separable convolutions is used to construct local pathways, which are masked. Using the initial detection bounding box as a gating signal, gated convolution is used to dynamically select the regions that need fine repair, restoring the edge and texture key features of the target; Finally, at each layer of the decoder, the contextual features of the global pathway and the detail features of the local pathway are weighted and fused to output the restored image. .

[0012] Furthermore, in step S2, the global pathway uses a pre-trained MobileNetv3 and replaces the ordinary convolutions in it with depthwise separable convolutions; the local pathway uses a 4-layer lightweight gated convolutional neural network.

[0013] Furthermore, in step S3, the process of constructing the multi-scale feature fusion detection network is as follows: First, the repaired image output by the gated dual-path repair network... As input; Secondly, a repair feature guidance module is added to the end of the YOLOv8 backbone network. This module injects the high-dimensional repair features of the corresponding layer of the decoder of the gated dual-path repair network into the feature pyramid of YOLOv8 through skip connections. Then, the bidirectional feature pyramid network technique is used to deeply fuse the multi-scale features extracted by YOLOv8 with the repair features to enhance the feature representation of the repaired target. Finally, the network outputs the final prediction results, including the category, confidence score, and bounding box; at the same time, the preliminary prediction results are fed back to the gated dual-path inpainting network as priors for the target region, guiding the fine-tuning in the next round of training.

[0014] Furthermore, in step S3, using YOLOv8s as the baseline model, the feature injection module uses 1×1 convolution to adjust the channels and then directly splices them, while the feature fusion module uses a weighted bidirectional feature pyramid network.

[0015] Furthermore, in step S4, during the pre-training phase, the gated dual-path repair network and the multi-scale feature fusion detection network are trained independently to provide quantized initialization parameters for subsequent collaborative training; during the joint training phase, the entire network is unfrozen for end-to-end training; the total loss function... Damage caused by repair Detection loss and feedback loss It consists of three parts, and the calculation method is as follows:

[0016] in , and To balance the hyperparameters; The restoration loss consists of pixel loss, perceptual loss, and occlusion mask loss. Among them, the pixel loss uses L1 loss to calculate the pixel difference between the restored area and the real unoccluded area, ensuring the consistency of color and texture in the restoration result. The perceptual loss extracts high-level features between the restored image and the real image through a pre-trained encoding network, calculates feature differences, and improves the visual rationality of the restoration result. The occlusion mask loss introduces a binary occlusion mask, which increases the loss weight of the occluded area by 2-3 times and reduces the loss weight of the unoccluded area to 0.5 times, strengthening the network's attention to the restoration of the occluded area. The detection loss consists of classification loss and regression loss; the classification loss uses the binary cross-entropy loss function, and the regression loss uses the cross-union ratio loss and the cross-union ratio loss. Feedback loss is the detection loss between the initial detection results generated by the detection network on the restored image and the ground truth annotation.

[0017] Furthermore, in step S5, during the forward inference stage, the input occlusion image is... If there is no readily available mask A lightweight mask prediction network is connected in parallel to estimate the occlusion region; occluded image After being repaired by the gated dual-path repair network, the final prediction result is directly output by the multi-scale feature fusion detection network without intermediate feedback.

[0018] (III) Beneficial Effects The above-mentioned technical solution provides an occluded target recognition system and method based on collaborative repair guidance, which has the following beneficial effects: 1. Detection Feedback Repair: The repair network receives direct feedback and guidance from the detection network. Its repair process takes "whether it is conducive to accurate detection" as the optimization goal, rather than just pursuing visual rationality, thus solving the problem of inconsistency between repair and detection goals.

[0019] 2. Deep Feature-Level Fusion: By using the repair feature injection module, repair information and detection information are deeply fused at the feature level rather than just the pixel level, enabling the detector to understand the "repair process" and greatly improving the feature expression capability.

[0020] 3. End-to-end collaborative optimization: Through joint training strategies and specially designed feedback loss, the entire system can learn how to repair for detection in an end-to-end manner, achieving a synergistic effect of 1+1 greater than 2, and exhibiting extremely high robustness and accuracy in heavily occluded scenarios. Attached Figure Description

[0021] Figure 1 This is a diagram of the occluded target recognition system based on collaborative repair guidance, as described in this invention. Detailed Implementation

[0022] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0023] This invention integrates image inpainting and object detection by constructing an end-to-end deep learning network. The inpainting network employs a dual-path structure, performing refined inpainting of the target region based on global inpainting and feedback from the detection network. The detection network enhances its ability to recognize the inpainted target by injecting and fusing high-level features from the inpainting process. Both are collaboratively optimized through a joint loss function that includes a feedback mechanism, ensuring that the inpainting process directly aims to improve detection accuracy. This effectively addresses the issues of missed and false detections of targets in heavily occluded scenarios, significantly improving the model's robustness and accuracy.

[0024] like Figure 1 As shown, this embodiment is based on an occluded target recognition system guided by collaborative repair, which includes a gated dual-path repair network and a multi-scale feature fusion detection network. The occluded image and mask are input into the gated dual-path repair network, and image repair is performed through the global and local paths within the gated dual-path repair network. The repaired image is input into the multi-scale feature fusion detection network, and after passing through the feature injection module and feature fusion module within the multi-scale feature fusion detection network, the detection result is output. At the same time, the detection result is fed back to the gated dual-path repair network to guide the repair network to be trained and optimized in a direction that is conducive to target detection.

[0025] The occluded target recognition method based on collaborative repair proposed in this invention includes the following steps: (1) Construct a simulated occlusion dataset. Obtain a public dataset containing a variety of targets, and use a binary random mask generation algorithm to generate irregular occlusion regions on target instances in the image, forming a four-tuple training sample pair of "original image-occluded image-binary occlusion mask-real annotation".

[0026] (2) Construct a gated dual-path repair network. First, the occluded image is... and the corresponding binary occlusion mask As input, and secondly, due to the requirements of real-time inference, depthwise separable convolutions are used to replace the regular convolutions in the U-Net encoder to construct the global pathway. This global pathway is responsible for capturing the global contextual semantic information of the image, performing preliminary, smooth inpainting of the entire image. Then, a lightweight network composed of depthwise separable convolutions is used to construct the local pathway, which uses a mask... Using the initial detection bounding box as a gating signal, gated convolution is used to dynamically select regions requiring fine-grained restoration, focusing on restoring key features such as the object's edges and textures. Finally, at each layer of the decoder, the contextual features of the global pathway and the detailed features of the local pathway are weighted and fused to output the restored image. .

[0027] (3) Construct a multi-scale feature fusion detection network. First, the repaired image output by the gated dual-path repair network is processed. As input, a repair feature guidance module is added to the end of the YOLOv8 backbone network. This module injects the high-dimensional repair features of the corresponding layer of the decoder in the gated dual-path repair network into the YOLOv8 feature pyramid through skip connections. Then, a bidirectional feature pyramid network technique is used to deeply fuse the multi-scale features extracted by YOLOv8 with the repair features, enhancing the feature representation of the repaired target. Finally, the network outputs the final prediction results, including the class, confidence score, and bounding box. Simultaneously, the preliminary prediction results are fed back to the gated dual-path repair network as a priori for the target region, guiding the refined repair in the next round of training.

[0028] (4) Design a collaborative training and optimization strategy. During the pre-training phase, a gated dual-path inpainting network (using pixel loss, perceptual loss, and occlusion mask loss) and a multi-scale feature fusion detection network (on an occlusion-free dataset) are trained independently to provide quantized initialization parameters for subsequent collaborative training. During the joint training phase, the entire network is unfrozen for end-to-end training. Total loss function Damage caused by repair Detection loss and feedback loss It consists of three parts, and the specific calculation method is as follows:

[0029] in , and To balance the hyperparameters.

[0030] The restoration loss consists of pixel loss, perceptual loss, and occlusion mask loss. Pixel loss uses L1 loss to calculate the pixel difference between the restored region and the real unoccluded region, ensuring color and texture consistency in the restored result. Perceptual loss extracts high-level features between the restored and real images through a pre-trained encoding network, calculates feature differences, and improves the visual plausibility of the restored result. Occlusion mask loss introduces a binary occlusion mask, increasing the loss weight for occluded regions by 2-3 times and decreasing the loss weight for unoccluded regions to 0.5 times, strengthening the network's focus on restoring occluded regions.

[0031] The detection loss consists of classification loss and regression loss. The classification loss uses the binary cross-entropy loss function, while the regression loss uses an improved cross-union ratio (CUP) loss and an improved CUP loss.

[0032] Feedback loss is the detection loss between the initial detection results generated by the detection network on the restored image and the ground truth annotation.

[0033] (5) Reasoning process: In the forward reasoning stage, the input is only the occluded image. If there is no readily available mask A lightweight mask prediction network can be connected in parallel to estimate the occluded region. Occluded image After being repaired by the gated dual-path repair network, the final prediction result is directly output by the multi-scale feature fusion detection network without intermediate feedback, thus ensuring efficiency.

[0034] Example (1) Experimental Environment. Hardware: CPU is Intel Xeon Gold 5320, GPU is 4 Nvidia A100 (40GB), memory is 64G; Embedded information processing board: Rockchip RK3588, computing power: 6TOPS, memory is 8G. Software: Operating system is Ubuntu 20.04, deep learning framework is PyTorch 2.0, programming language is Python 3.8, image processing library is OpenCV 4.7.

[0035] (2) Data preparation. First, LabelImage is used to label the targets in the dataset; then, a binary random mask generation algorithm is used to randomly apply occlusion to the targets within the labeled boxes in the image, with the occlusion area ratio ranging from 10% to 40%; finally, the occlusion image, mask image and label file are obtained.

[0036] (3) Parameter settings of the gated dual-path repair network. The global path of the gated dual-path repair network adopts the pre-trained MobileNetv3, and the ordinary convolutions in it are replaced with depthwise separable convolutions; the local path uses a 4-layer lightweight gated convolutional neural network.

[0037] (4) Parameter settings for the multi-scale feature fusion detection network. Using YOLOv8s as the baseline model, the feature injection module adopts a 1... After channel adjustment, the convolutional layers are directly spliced ​​together, and the feature fusion module adopts a weighted bidirectional pyramid network.

[0038] (5) Co-training parameter settings. Pre-training: The initial learning rate of the repair network is 1e-4, the learning rate decay strategy is 0.5 times every 10 rounds, the optimizer is AdamW (weight decay 1e-3), the batch size is 32, the number of training rounds is 100, and the weights of pixel loss, perception loss and occlusion mask loss in the loss function of the repair network are 1.0, 0.5, and 2.0. The learning rate strategy of the detection network is cosine annealing, the initial learning rate is 1e-3, the minimum learning rate is 1e-5, the optimizer is SGD (momentum is 0.9, weight decay is 5e-4), the batch size is 64, the number of training rounds is 100, and the weights of improved crossover ratio loss and improved focus loss in the loss function of the detection network are 1.0 and 1.0.

[0039] Joint training optimization: Initial learning rate 1e-3, optimizer SGD, batch size 32, training 100 epochs. Total loss weights are set to... =1.0, =1.0, =0.5.

[0040] (6) Experimental Results: The experimental results of the test dataset on Rockchip RK3588 are shown in the table below:

[0041] The experimental results show that, compared with models like U-Net+YOLOv8 that repair before recognition, this invention improves mAP@0.5 by 4.9% while maintaining a similar number of parameters and inference speed, which is significantly better than existing technologies.

[0042] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An occluded target recognition system based on collaborative repair guidance, characterized in that, It includes a gated dual-path restoration network and a multi-scale feature fusion detection network. The occluded image and mask are input into the gated dual-path restoration network, and the image is restored through the global and local paths inside the gated dual-path restoration network. The restored image is input into the multi-scale feature fusion detection network, and after passing through the feature injection module and feature fusion module inside the multi-scale feature fusion detection network, the detection result is output. At the same time, the detection result is fed back to the gated dual-path restoration network to guide the restoration network to be trained and optimized in a direction that is conducive to target detection.

2. An occluded target recognition method based on collaborative repair guidance, characterized in that, Includes the following steps: S1: Construct a simulated occlusion dataset; S2: Construct a gated dual-path repair network; S3: Construct a multi-scale feature fusion detection network; S4: Design collaborative training and optimization strategies; S5: Inference output results.

3. The occluded target recognition method based on collaborative repair guidance as described in claim 2, characterized in that, In step S1, a public dataset containing a variety of targets is obtained. A binary random mask generation algorithm is used to generate irregular occlusion regions on the target instances in the image, forming a four-tuple training sample pair of "original image - occluded image - binary occlusion mask - ground truth annotation".

4. The occluded target recognition method based on collaborative repair guidance as described in claim 3, characterized in that, In step S1, the proportion of the occluded area of ​​the irregular occlusion area is 10% to 40%.

5. The occluded target recognition method based on collaborative repair guidance as described in claim 3, characterized in that, In step S2, the process of constructing the gated dual-path repair network is as follows: First, occlude the image and the corresponding binary occlusion mask As input; Secondly, depthwise separable convolutions are used to replace the conventional convolutions in the U-Net encoder to construct a global path. The entire global path is responsible for capturing the global contextual semantic information of the image and performing preliminary and smooth restoration on the entire image. Then, a lightweight network consisting of depthwise separable convolutions is used to construct local pathways, which are masked. Using the initial detection bounding box as a gating signal, gated convolution is used to dynamically select the regions that need fine repair, restoring the edge and texture key features of the target; Finally, at each layer of the decoder, the contextual features of the global pathway and the detail features of the local pathway are weighted and fused to output the restored image. .

6. The occluded target recognition method based on collaborative repair guidance as described in claim 5, characterized in that, In step S2, the global pathway uses a pre-trained MobileNetv3, and the ordinary convolutions in it are replaced with depthwise separable convolutions; the local pathway uses a 4-layer lightweight gated convolutional neural network.

7. The occluded target recognition method based on collaborative repair guidance as described in claim 5, characterized in that, In step S3, the process of constructing the multi-scale feature fusion detection network is as follows: First, the repaired image output by the gated dual-path repair network... As input; Secondly, a repair feature guidance module is added to the end of the YOLOv8 backbone network. This module injects the high-dimensional repair features of the corresponding layer of the decoder of the gated dual-path repair network into the feature pyramid of YOLOv8 through skip connections. Then, the bidirectional feature pyramid network technique is used to deeply fuse the multi-scale features extracted by YOLOv8 with the repair features to enhance the feature representation of the repaired target. Finally, the network outputs the final prediction results, including the category, confidence score, and bounding box; at the same time, the preliminary prediction results are fed back to the gated dual-path inpainting network as priors for the target region, guiding the fine-tuning in the next round of training.

8. The occluded target recognition method based on collaborative repair guidance as described in claim 7, characterized in that, In step S3, using YOLOv8s as the baseline model, the feature injection module uses 1×1 convolution to adjust the channels and then directly splices them together, while the feature fusion module uses a weighted bidirectional feature pyramid network.

9. The occluded target recognition method based on collaborative repair guidance as described in claim 8, characterized in that, In step S4, during the pre-training phase, the gated dual-path repair network and the multi-scale feature fusion detection network are trained independently to provide quantized initialization parameters for subsequent collaborative training; during the joint training phase, the entire network is unfrozen and end-to-end training is performed; the total loss function... Damage caused by repair Detection loss and feedback loss It consists of three parts, and the calculation method is as follows: in , and To balance the hyperparameters; The restoration loss consists of pixel loss, perceptual loss, and occlusion mask loss. Among them, the pixel loss uses L1 loss to calculate the pixel difference between the restored area and the real unoccluded area, ensuring the consistency of color and texture in the restoration result. The perceptual loss extracts high-level features between the restored image and the real image through a pre-trained encoding network, calculates feature differences, and improves the visual rationality of the restoration result. The occlusion mask loss introduces a binary occlusion mask, which increases the loss weight of the occluded area by 2-3 times and reduces the loss weight of the unoccluded area to 0.5 times, strengthening the network's attention to the restoration of the occluded area. The detection loss consists of classification loss and regression loss; the classification loss uses the binary cross-entropy loss function, and the regression loss uses the cross-union ratio loss and the cross-union ratio loss. Feedback loss is the detection loss between the initial detection results generated by the detection network on the restored image and the ground truth annotation.

10. The occluded target recognition method based on collaborative repair guidance as described in claim 9, characterized in that, In step S5, during the forward inference stage, the occluded image is input. If there is no readily available mask A lightweight mask prediction network is connected in parallel to estimate the occlusion region; occluded image After being repaired by the gated dual-path repair network, the final prediction result is directly output by the multi-scale feature fusion detection network without intermediate feedback.

Images