Real-time detection-segmentation integrated modeling method based on double-constraint mechanism

Abstract

The invention discloses a real-time target detection and image segmentation integrated modeling method and a real-time target detection and image segmentation integrated modeling system based on a double-constraint mechanism, which are suitable for an intelligent terminal and are executed by a processor. The method comprises the following steps: constructing a unified neural network model, introducing a Mask prediction branch into a detection module, and generating a space attention map by using the predicted Mask to restrain the generation of a bounding box so as to improve the target positioning precision; a boundary sensing mechanism is introduced into the segmentation module, and the definition and stability of the segmentation boundary are improved through edge detection, edge enhanced convolution and distance field supervision; and collaborative optimization of detection and segmentation is realized by sharing shallow layer features and separating high layer features. The system comprises an image acquisition module, a feature extraction module, a detection module, a segmentation module and a control module, and can efficiently complete detection and segmentation tasks. According to the method, high precision is guaranteed, the light weight and the real-time performance of the model are considered, and the small target detection and boundary segmentation performance is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence visual perception technology, and in particular to a real-time detection-segmentation integrated modeling method based on a double constraint mechanism. BACKGROUND

[0002] Target detection and image segmentation are two basic and key tasks in computer vision. The former focuses on "finding out what exists in the image", and the latter emphasizes "accurately outlining the target boundary". In practical application scenarios such as autonomous driving, industrial detection and security monitoring, both often appear at the same time and put higher requirements on the response speed and accuracy of the system.

[0003] However, in the prior art, these two tasks are usually modeled and deployed independently, which not only leads to repeated calculation and waste of resources, but also causes information fragmentation between tasks. For example, traditional detection methods such as Faster R-CNN and YOLO series have a significant decrease in accuracy when dealing with occlusion, irregular shape or small targets; while semantic segmentation methods such as DeepLab or PSPNet often have problems such as blurred contours and structure breaks at boundaries, and rely heavily on post-processing (such as CRF) to improve the effect.

[0004] Especially in edge computing terminals, the computing power is limited, and the model parameter size must be simplified. Existing multi-task joint models attempt to integrate both, but most have high structural coupling, low module reuse rate and serious optimization path conflicts, resulting in complex models and unstable effects, making it difficult to truly deploy.

[0005] Therefore, there is an urgent need for an integrated solution that can realize collaborative optimization of detection and segmentation, clear structure, accurate boundaries, small target robustness and terminal deployment, breaking through the bottlenecks of existing solutions in task collaboration, model structure and real-time performance.

[0006] Therefore, the prior art still needs to be improved. SUMMARY

[0007] In view of the above deficiencies of the prior art, the purpose of the present application is to provide a real-time detection-segmentation integrated modeling method and system based on a double constraint mechanism, which solves the problems of unstable small target detection, fuzzy segmentation boundaries and low multi-task collaboration efficiency by introducing spatial attention and boundary perception mechanisms, thereby realizing high-precision, lightweight and deployable multi-task visual perception capabilities.

[0008] The technical solution of the present application is as follows: In one aspect, the present application provides a real-time detection-segmentation integrated modeling method based on a double constraint mechanism, which comprises: (1) constructing a unified neural network structure including a target detection module and an image segmentation module; (2) A Mask prediction branch is set in the target detection module, a spatial attention map is generated using the predicted Mask, and the bounding box generation is constrained through the attention map to enhance the small target detection precision; (3) A boundary perception mechanism is integrated in the image segmentation module, edge detection, edge enhancement convolution, and distance field supervision are used to improve the accuracy and robustness of the segmentation boundary; (4) By sharing shallow features and task-specific separation of high-level features, the detection and segmentation modules are jointly optimized to complete end-to-end joint training.

[0009] In one embodiment, the target detection module adopts a multi-task learning structure, including a classification branch, a bounding box regression branch, and a Mask prediction branch, wherein the Mask prediction result is processed by convolution and normalization to generate a spatial attention map.

[0010] In one embodiment, the boundary perception mechanism includes: a) The original image is subjected to initial edge extraction using a Canny operator; b) The initial edge is refined through an edge enhancement convolution module; c) A distance field is constructed based on the Euclidean distance of pixels to the nearest boundary and used as a supervision signal to guide the segmentation module training.

[0011] In one embodiment, the region of interest (ROI) output by the detection module is used to crop the input image feature map, and the cropped region is used as the prior input of the segmentation module to improve feature consistency and segmentation accuracy.

[0012] In one embodiment, the boundary information of the segmentation module is fed back to the detection module and used for dynamic adjustment of the detection box confidence, thereby improving the boundary accuracy of the detection result.

[0013] In one embodiment, the neural network structure has a module decoupling mechanism, and the target detection module, the image segmentation module, or both the cooperative module can be selectively enabled according to application requirements.

[0014] Another aspect of the present application provides an intelligent vision processing system based on a double constraint mechanism, comprising: an image acquisition module for acquiring input images; a feature extraction module for extracting shallow shared features; a target detection module configured with a Mask prediction branch and a spatial attention generation unit; an image segmentation module configured with an edge detection module, a distance field supervision module, and a boundary refinement module; an inference control module for coordinating the joint execution of the detection module and the segmentation module; The system is used for executing the method of any one of the above.

[0015] In one embodiment, the feature extraction module shares low-level feature maps with the detection module and the segmentation module through a unified neural network backbone.

[0016] Another aspect of the present application provides a terminal device integrated with the above-mentioned system, which comprises: an image acquisition device; a neural network processor; a memory and a model loading module; a communication module and a power supply control module; wherein the terminal device is suitable for real-time target detection and image segmentation applications in an edge computing environment.

[0017] In one embodiment, the neural network processor comprises an acceleration module for performing target detection and image segmentation inference, and supports dynamic switching of running modes to adapt to different task loads.

[0018] Compared with prior art solutions, the present application has the following outstanding technical advantages and unexpected effects: 1. More robust detection: By introducing a Mask constraint mechanism, the Box-Mask alignment accuracy is significantly improved, the IoU of Box and the real contour can reach 90.3%, and the false detection rate is reduced from 0.89 to 0.61, especially suitable for dense occlusion and small target scenes.

[0019] 2. More detailed segmentation: Based on the double boundary supervision of distance field and edge enhancement, the boundary F-score is improved to 0.83, which is 0.11 higher than the traditional softmax, and the structure details are more complete.

[0020] 3. More efficient inference: Through shallow sharing and high-level decoupling between modules, 20% of redundant calculations are reduced without sacrificing accuracy, and the model size is compressed by 10%, which is more suitable for edge terminals.

[0021] 4. Closer collaboration: The detection results participate in the segmentation area guidance, and the segmentation boundary feeds back to the detection scoring, which realizes the lightweight visual bidirectional feedback mechanism for the first time, and improves the overall system adaptive ability.

[0022] 5. More flexible deployment: Supports detection / segmentation running alone or jointly, meeting the needs of various scenarios such as intelligent vehicle, security camera, mobile robot, etc. BRIEF DESCRIPTION OF DRAWINGS

[0023] The present application will be further described below in conjunction with the drawings and embodiments, and the drawings are as follows: Figure 1A flow chart of the real-time detection-segmentation integrated modeling method based on the double constraint mechanism provided by the present application is shown in the figure. Figure 2 A schematic diagram of the mechanism of the real-time detection-segmentation integrated modeling method based on the double constraint mechanism provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. The embodiments of the present application are described below with reference to the accompanying drawings.

[0025] The real-time detection-segmentation integrated modeling method based on the double constraint mechanism provided by the present application is shown in the figure. Figure 1 The method comprises the following steps: S1, constructing a unified neural network structure comprising a target detection module and an image segmentation module; S2, setting a Mask prediction branch in the target detection module, using the predicted Mask to generate a spatial attention map, and using the attention map to constrain the generation of the bounding box to enhance the small target detection accuracy; S3, integrating a boundary perception mechanism in the image segmentation module, using edge detection, edge enhancement convolution and distance field supervision to improve the accuracy and robustness of the segmentation boundary; S4, jointly optimizing the detection and segmentation modules by sharing the shallow layer features and task-specific separation of the high layer features, and completing the end-to-end joint training.

[0026] Specifically, the present application provides an integrated modeling method, and the overall structure of the network is composed of a backbone feature extraction network, a target detection module and an image segmentation module. The feature extraction network is a shared shallow layer module, and the high layer features are output to the detection head and the segmentation head respectively. The detection module contains a Mask prediction branch, which generates an attention map to participate in the generation of the detection box, and the segmentation module is based on a boundary perception mechanism to model the target boundary in detail.

[0027] In a further embodiment, the target detection module adopts a multi-task learning structure, including a classification branch, a bounding box regression branch and a Mask prediction branch, wherein the Mask prediction result is processed by convolution and normalization to generate a spatial attention map.

[0028] In the detection module, a three-branch architecture is adopted: the classification branch is responsible for target category prediction, the regression branch is responsible for bounding box coordinates, and the Mask branch predicts the target region Mask. After Mask prediction, a spatial attention map is generated by convolution and Sigmoid activation, which is used as a target region guided feature and weighted input into the regression module to improve the spatial consistency of the detection result.

[0029] In further embodiments, the boundary-aware mechanism comprises: a) initial edge extraction on the original image using Canny operator; b) refining the initial edges by edge-enhancing convolution module; c) constructing distance field based on the Euclidean distance from pixels to the nearest boundary, and using it as a supervisory signal to guide the training of the segmentation module.

[0030] The segmentation module adopts a boundary-aware strategy. First, the initial edges are extracted using Canny algorithm and sent to a learnable edge-enhancing convolution for refinement. Then, a pixel-level distance field is generated by distance transformation, which is used as an auxiliary supervisory signal for training. The boundary features of each layer are fused and output as the final boundary prediction map after convolution fusion.

[0031] In further embodiments, the region of interest (ROI) output by the detection module is used to crop the input image feature map, and the cropped region is used as prior input to the segmentation module to improve feature consistency and segmentation accuracy.

[0032] The ROI output by the detection module is used to crop the image feature region and sent as prior information to the segmentation module to enhance the region recognition. This process improves the positioning accuracy of the segmentation module, especially in occluded targets or low-contrast images.

[0033] In further embodiments, the boundary information of the segmentation module is fed back to the detection module and used for dynamic adjustment of the detection box confidence, thereby improving the boundary accuracy of the detection result.

[0034] The boundary confidence map output by the segmentation module is used to feedback the detection module, and is used as a boundary weight item in weighted calculation when detecting scoring, thereby improving the accuracy of the detection boundary and the quality of the box fitting.

[0035] In further embodiments, the neural network structure has a module decoupling mechanism, which can selectively enable the target detection module, the image segmentation module, or the two modules in coordination according to application requirements.

[0036] The model supports on-demand enabling of the target detection module, the image segmentation module, or both modules working in parallel, adapting to different terminal deployment requirements.

[0037] Please refer to Figure 2 , the schematic diagram of the present application is specifically as Figure 2As shown, wherein Data represents the input picture, Backbone is the low-level shared feature convolution layer, Mask processing and Boundary processing respectively complete the high-level mask and boundary feature extraction, and the loss calculation with the labeled true value, Detection head and Segmentation head respectively complete the post-processing of the detection and segmentation results, and output the detection and segmentation results.

[0038] The application also provides an intelligent vision processing system running the neural network model of the method of the application, which comprises: An image acquisition module: which can be an industrial camera, a vehicle-mounted camera or an embedded image sensor, which acquires external images and converts them into network input format; A feature extraction module: which is usually realized by a deep convolutional neural network such as MobileNet, ResNet, SwinTransformer, etc., which is responsible for extracting shallow shared features for subsequent detection and segmentation branches; A target detection module: which includes a classification branch, a regression branch and a Mask prediction branch, and the Mask branch outputs an attention map for accurate positioning of the bounding box; An image segmentation module: which includes an edge extraction module, an edge enhancement convolution module and a boundary supervision module, and the edge information is further refined through multiple scale fusion modules; An inference control module: which is used to dynamically schedule the running mode of the detection module and the segmentation module during task execution, and coordinate the logic of feature sharing, data flow routing and output format matching, etc.

[0039] The modules are interconnected through a unified communication bus, and data flows between the modules in the form of tensors, and the control module controls the order, frequency and weight distribution of detection and segmentation based on the set task strategy.

[0040] In the above system, the connection structure between the feature extraction module and the detection module and the segmentation module has the following characteristics: Shallow feature sharing: the 1st-3rd layer convolution outputs are input to the detection and segmentation modules as shared parts to improve the calculation efficiency and feature reuse rate; High-level feature separation: the 4th layer and above features are decoupled and optimized for tasks, respectively connected to the detection branch and the segmentation branch, solving the problem of multi-task optimization conflict; There is a cross-module information guiding mechanism in the intermediate path, such as the ROI cropping path and the boundary confidence feedback path, to ensure efficient and non-redundant information cross-fusion.

[0041] The application also provides a terminal device for deploying the system, which is suitable for edge deployment and embedded execution scenarios, and specifically comprises: Image acquisition device: high-definition camera module is adopted, which has wide dynamic range and high sensitivity, and supports output image resolution of 1080P or above; Neural network processor: NPU, GPU or TPU structure, supporting ONNX / TensorRT model inference, and having a minimum operation capacity of 8 TOPS; Model storage and loading module: used for loading and dynamically scheduling detection and segmentation model parameters, supporting model hot updating and parameter compression storage; Communication module: supporting USB, CAN, Ethernet or 5G communication, facilitating edge result synchronization to a central server or interaction with other terminals; Power management and device management module: supporting voltage monitoring, fault detection and remote OTA updating.

[0042] Further, the neural network processor integrated in the terminal device is provided with a running mode control unit, which dynamically switches among the following three modes according to the current task pressure or system instruction: Detection mode: only the detection module is enabled, suitable for real-time response and low segmentation requirement scenarios; Segmentation mode: only the segmentation module is enabled, suitable for scenarios where the target area is known but high-precision contour labeling is required; Joint mode: both the detection and segmentation modules are enabled, and they run cooperatively and interact with each other, suitable for complex visual tasks such as city street analysis or industrial defect detection.

[0043] The control strategy supports automatic weight distribution, model load balancing and intermediate feature reuse optimization, and can automatically balance accuracy and speed under different frame rates and task targets.

[0044] The application provides a real-time detection-segmentation integrated modeling method based on a double constraint mechanism, and the modeling method has the following advantages for target detection and segmentation: I. Compared with the traditional detection method, the following advantages are obtained: (1) The box frame is generated by guiding the Mask information, which can improve the positioning accuracy.

[0045] (2) Compared with the traditional detection method, the Mask can indicate the visible area, avoiding interference from the occluded area.

[0046] (3) For irregular targets, the rectangular frame contains a large amount of background information, while the Mask can be closely fitted to the target contour for prediction, and the detection effect of irregularly shaped targets such as curves is significantly improved II. Compared with the traditional segmentation method, the following advantages are obtained: (1) Compared with the traditional method, the method has clear binary boundary.

[0047] (2) The traditional method performs poorly in fine structures, and the application increases edge enhancement to protect detailed information from loss.

[0048] (3) The traditional method is susceptible to texture interference, while the application corrects the boundary through morphology and enhances the stability of the boundary.

[0049] (4) The traditional method relies heavily on CRF post-processing, while the application can realize end-to-end learning and reduce the post-processing computing overhead.

[0050] III. Compared with the traditional multi-task method, the following advantages are obtained: (1) The traditional multi-task model has more repeated calculations and low computational efficiency, and the application uses a balance design of feature sharing and specialization to greatly improve the inference speed.

[0051] (2) The traditional multi-task model has redundant parameters and low parameter utilization, and the application reduces redundancy through Mask-boundary collaborative constraint, reducing the model size by 10% (3) The traditional multi-task model has a fixed structure, while the application can be disassembled and deployed to support detection / segmentation alone.

[0052] In summary, the application provides a real-time target detection and image segmentation integrated modeling method and system that fuses Mask constraint and boundary perception mechanism, which first realizes bidirectional information guidance and module decoupling collaboration in a lightweight structure, solving the core problems of unstable small target detection, unclear segmentation boundary and structural redundancy of multi-task model in the prior art.

[0053] By introducing a spatial attention driven Mask prediction branch in the detection module, the target region feature response ability is effectively enhanced; in the segmentation module, edge convolution and distance field joint supervision mechanism are introduced to realize boundary optimization at the detail level; at the same time, the structure design of backbone feature sharing and high-level feature separation is adopted to ensure that the detection and segmentation tasks still have high-precision inference ability in the limited computing resource scene.

[0054] The method described in the application not only has complete engineering implementability, but also is compatible with various terminal deployment requirements, and is suitable for automatic driving systems, intelligent security terminals, industrial vision edge devices and other types of actual scenes, has good expansibility and industrialization prospect, and can provide an integrated, efficient and high-precision solution for multi-task visual perception systems.

[0055] It is to be understood that the application is not limited to the examples described above, which can be modified or adapted in several ways by those skilled in the art without departing from the scope of the present application, as defined by the appended claims.

Claims

1. A real-time detection-segmentation integrated modeling method based on a double constraint mechanism, characterized in that, The method comprises: (1) constructing a unified neural network structure, including a target detection module and an image segmentation module; (2) setting a Mask prediction branch in the target detection module, using the predicted Mask to generate a spatial attention map, and constraining the generation of the bounding box through the attention map to enhance the detection accuracy of small targets; (3) integrating a boundary perception mechanism in the image segmentation module, using edge detection, edge enhancement convolution, and distance field supervision to improve the accuracy and robustness of the segmentation boundary; (4) jointly optimizing the detection and segmentation modules by sharing shallow features and task-specific separation of high-level features, and completing end-to-end joint training.

2. The real-time detection-segmentation integrated modeling method based on a double constraint mechanism according to claim 1, characterized in that, The target detection module uses a multi-task learning structure, including a classification branch, a bounding box regression branch, and a Mask prediction branch, wherein the Mask prediction result is processed by convolution and normalization to generate a spatial attention map.

3. The real-time detection-segmentation integrated modeling method based on a double constraint mechanism according to claim 1, characterized in that, The boundary perception mechanism comprises: a) using a Canny operator to perform initial edge extraction on the original image; b) refining the initial edges through an edge enhancement convolution module; c) constructing a distance field based on the Euclidean distance from the pixel to the nearest boundary, and using it as a supervision signal to guide the segmentation module training.

4. The real-time detection-segmentation integrated modeling method based on a double constraint mechanism according to claim 1, characterized in that, The region of interest (ROI) output by the detection module is used to crop the input image feature map, and the cropped region is used as the prior input of the segmentation module to improve feature consistency and segmentation accuracy.

5. The real-time detection-segmentation integrated modeling method based on a double constraint mechanism according to claim 1, characterized in that, The boundary information of the segmentation module is fed back to the detection module and used for dynamic adjustment of the detection box confidence, thereby improving the boundary accuracy of the detection result.

6. The real-time detection-segmentation integrated modeling method based on a double constraint mechanism according to claim 1, characterized in that, The neural network structure has a module decoupling mechanism, which can selectively enable the target detection module, the image segmentation module, or both modules according to application requirements.

7. An intelligent vision processing system based on dual constraint mechanism characterized in that, Comprises: An image acquisition module for acquiring input images; A feature extraction module for extracting shallow shared features; A target detection module configured with a Mask prediction branch and a spatial attention generation unit; An image segmentation module configured with edge detection, distance field supervision, and boundary refinement modules; An inference control module for coordinating the joint execution of the detection module and the segmentation module; The system is used to perform the method of any one of claims 1 to 6.

8. The dual constraint mechanism based intelligent vision processing system according to claim 7, wherein, The feature extraction module, the detection module, and the segmentation module all share low-level feature maps through a unified neural network backbone.

9. A terminal device integrated with the system of claim 7, characterized in that, The terminal device comprises: An image acquisition device; A neural network processor; A memory and a model loading module; A communication module and a power supply control module; The terminal device is suitable for real-time target detection and image segmentation applications in an edge computing environment.

10. The terminal device according to claim 9, characterized by The neural network processor includes an acceleration module for performing target detection and image segmentation inference, and supports dynamic switching of running modes to adapt to different task loads.

Images