Intelligent marking method and system for road occupation construction violation based on pixel-level semantic segmentation

Abstract

The invention discloses a pixel-level semantic segmentation-based road-occupying construction violation intelligent labeling method. The method comprises the following steps of S1, obtaining an original image of a road construction scene; s2, the obtained image is processed based on a semantic segmentation network, and the semantic segmentation network comprises a backbone network used for feature extraction, a pyramid pooling module which has a plurality of pooling sizes and is used for capturing multi-scale context information, and a feature fusion layer; the feature fusion layer is used for fusing multi-level semantic information and outputting segmentation thermodynamic diagrams of different construction elements; s3, based on a conditional random field algorithm, performing boundary optimization on the segmented thermodynamic diagram; and S4, based on the segmentation result of the plurality of construction elements, identifying and outputting a composite violation scene. The method can solve the problems that in the prior art, the manual labeling cost is high, multi-element collaborative recognition is difficult, and the recognition precision is insufficient in a complex scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent traffic monitoring technology, and in particular to a method and system for intelligent labeling of illegal road construction based on pixel-level semantic segmentation. Background Technology

[0002] During drone inspections, intelligent detection of violations typically requires manual image annotation. A large number of images are manually identified and annotated before being used for training with YOLOv5 or other technologies. This image annotation process consumes significant manpower and also has the following drawbacks: 1) Inefficient manual labeling: Traditional methods rely on manual labeling of construction areas, fences and other elements. Labeling 1km of road image takes 3 days (Traffic Engineering, 2023, 41(2):78), and labor costs account for more than 60% of the total cost; 2) Limitations of single-element recognition: Patent CN114219797A only detects construction equipment and does not associate it with complex violations such as missing fences; 3) Recognition accuracy bottleneck: The traditional YOLOv5 model has a recognition rate of ≤85% in complex urban scenarios (IEEE T-ITS2022 data).

[0003] Therefore, there is an urgent need for an efficient and accurate automated annotation method to reduce costs and improve the ability to identify complex violation scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for intelligent labeling of illegal road construction based on pixel-level semantic segmentation, in order to solve the problems of high cost of manual labeling, difficulty in multi-element collaborative identification, and insufficient identification accuracy in complex scenarios in the existing technology.

[0005] To achieve the above objectives, the following technical solution is adopted: A method for intelligent annotation of illegal road construction based on pixel-level semantic segmentation includes the following steps: S1: Acquire the original image of the road construction scene; S2: The image obtained by processing based on a semantic segmentation network, wherein the semantic segmentation network includes a backbone network for feature extraction, a pyramid pooling module with multiple pooling sizes for capturing multi-scale contextual information, and a feature fusion layer; the feature fusion layer is used to fuse multi-level semantic information and output segmentation heatmaps of different construction elements. S3: Boundary optimization of segmented heatmaps based on the conditional random field algorithm; S4: Based on the segmentation results of multiple construction elements, identify and output composite violation scenarios.

[0006] Furthermore, in S2, the semantic segmentation network is a PSPNet network, and the backbone network is a ResNet50 network; the pyramid pooling module performs pooling operations of four scales in parallel: 1×1, 2×2, 3×3, and 6×6, to obtain multi-scale contextual features covering the global layout to local small targets.

[0007] Furthermore, the formula for the segmentation heatmap output by the feature fusion layer is expressed as follows: .

[0008] Furthermore, in S2, the construction elements include the construction area, fence, water-filled barriers, and equipment.

[0009] Furthermore, in S3, an energy function is used. Boundary optimization is performed, where, Let i represent the class probability of pixel i. This represents the similarity constraint between adjacent pixels i and j.

[0010] Furthermore, in S4, a dynamic weighted fusion model is used to determine the spatial relationship of various construction elements in order to identify complex violation scenarios. For areas with a confidence level of less than 90%, a manual lightweight review is performed, while the labeling results are directly output for the remaining areas.

[0011] Furthermore, the dynamic weighted fusion model is based on FPN structure to align multi-scale outputs and uses a decision tree rule engine to determine the spatial relationships of elements.

[0012] A pixel-level semantic segmentation-based intelligent labeling system for illegal road construction is also provided to implement the above-mentioned method, including... Image acquisition module, used to acquire images of road construction via drone; The automatic annotation engine adopts the PSPNet semantic segmentation architecture, configures ResNet50 as the backbone network, and includes a pyramid pooling module to output segmentation heatmaps of different construction elements. The boundary optimization module is used to optimize the boundaries of the segmented heatmap; The violation decision output module includes a dynamic weighted fusion model for identifying complex violation scenarios, and the violation decision output module is used to output the annotation results.

[0013] Furthermore, the pooling sizes of the pyramid pooling modules are 1×1, 2×2, 3×3, and 6×6, respectively. Furthermore, it also includes a manual review interface for reviewing and correcting low-confidence areas.

[0014] By adopting the above solution, the beneficial effects of the present invention are: 1) Pixel-level automatic annotation is achieved through the PSPNet semantic segmentation network. Compared with manual annotation, this invention can greatly reduce annotation costs and significantly improve annotation efficiency. 2) The PSPNet architecture, which adopts pyramid pooling modules (1×1, 2×2, 3×3, 6×6), can automatically capture multi-scale contextual information from global layout to local small targets (such as water-filled barriers and fence seams). Combined with the CRF boundary optimization algorithm, it can effectively reduce edge blurring and greatly improve the accuracy of single-feature annotation. 3) Based on the dynamic weighted fusion model and decision tree rule engine, this invention can identify complex violation scenarios (such as "no fence in the construction area + equipment crossing the boundary"), solving the problem of difficulty in identifying multiple factors in the prior art. Attached Figure Description

[0015] Figure 1 This is a flowchart of the present invention; Figure 2 This is a diagram of the overall architecture of the present invention; Figure 3 This is a design diagram of the semantic segmentation network of the present invention; Figure 4 This is a schematic diagram of the pyramid pooling module of the present invention; Figure 5 This is a diagram illustrating the intelligent annotation implementation mechanism of the present invention. Detailed Implementation

[0016] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] Reference Figures 1 to 5 As shown, this invention provides a method for intelligent annotation of illegal road construction based on pixel-level semantic segmentation, comprising the following steps: S1: Obtain the original image of the road construction scene.

[0018] In this step, raw images of the road construction scene are acquired using drones or other imaging devices. The drones need to be equipped with high-resolution cameras (such as 1920×1080 RGB) to capture panoramic images of the road construction site from the air, ensuring wide coverage and clear details. The acquired images also need to meet the requirements of high definition and multiple angles to support subsequent semantic segmentation processing.

[0019] S2: The image obtained by processing based on a semantic segmentation network, wherein the semantic segmentation network includes a backbone network for feature extraction, a pyramid pooling module with multiple pooling sizes for capturing multi-scale contextual information, and a feature fusion layer; the feature fusion layer is used to fuse multi-level semantic information and output segmentation heatmaps of different construction elements. In S2, the semantic segmentation network is a PSPNet network and the backbone network is a ResNet50 network; the pyramid pooling module performs pooling operations of four scales in parallel: 1×1, 2×2, 3×3 and 6×6, to obtain multi-scale contextual features covering the global layout to local small targets. The formula for the segmentation heatmap output by the feature fusion layer is as follows: .

[0020] In S2, the construction elements include the construction area, fence, water-filled barriers, and equipment.

[0021] In this step, the acquired images are processed using a semantic segmentation network to generate segmentation heatmaps for different construction elements. The semantic segmentation network is PSPNet (Pyramid Scene Parsing Network), whose core advantage lies in its ability to recognize targets of varying sizes in complex urban scenes through multi-scale feature extraction. The structure of the PSPNet network includes the following key components: Backbone Network (ResNet50): ResNet50 is used as the feature extraction network. It extracts hierarchical features of the image through a series of convolutional layers (7×7, 3×3 and residual block sequence). The output feature map size is 1 / 32 (16×16) of the original image. The deep structure of ResNet50 ensures the robustness of feature extraction. Pyramid Pooling Module: This module performs pooling operations in parallel at four scales: 1×1, 2×2, 3×3, and 6×6. It captures multi-scale contextual information from global layout (such as the entire construction area) to local small targets (such as water-filled barriers and fence seams). The applicable scenarios for different pooling sizes are shown in the table below:

[0022] Table 1. Applicable Scenarios for Different Pooling Sizes Feature fusion layer: By upsampling and stitching, the multi-scale feature maps are restored to the original image resolution, generating segmented heatmaps of elements such as construction areas, fences, water barriers, and equipment.

[0023] S3: Boundary optimization of segmented heatmaps based on the conditional random field algorithm; In S3, an energy function is used. Boundary optimization is performed, where, Let i represent the class probability of pixel i. This represents the similarity constraint between adjacent pixels i and j.

[0024] In this step, the Conditional Random Field (CRF) algorithm is used to optimize the boundary of the segmentation heatmap to improve the labeling accuracy. The CRF algorithm simultaneously optimizes the pixel category probability and neighborhood similarity through the energy function mentioned above. At the same time, it minimizes the color / spatial distance difference through iterative calculation to solve the edge blurring problem of small targets such as fences and water-filled barriers, improves the single-feature labeling accuracy to over 98%, and significantly suppresses missegmentation.

[0025] S4: Based on the segmentation results of multiple construction elements, identify and output composite violation scenarios; In S4, a dynamic weighted fusion model is used to determine the spatial relationship of multiple construction elements in order to identify complex violation scenarios. For areas with a confidence level of less than 90%, a manual lightweight review is performed, while the labeling results are directly output for the remaining areas. The dynamic weighted fusion model is based on FPN structure to align multi-scale output and uses a decision tree rule engine to determine the spatial relationship of elements.

[0026] In this step, based on the segmentation results of multiple construction elements (construction area, fence, water-filled barriers, equipment), a dynamic weighted fusion model is used to identify complex violation scenarios. This model aligns multi-scale outputs based on an FPN (Feature Pyramid Network) structure and determines the spatial relationships between elements using a decision tree rule engine. For example: The test detects a combination of "no fence in the construction area" and "equipment crossing the boundary"; The dynamic weighted fusion model identifies violations involving "missing fences" and "no water-filled barriers" by weighting and fusing the segmentation results of each element and combining them with decision tree rules (such as spatial proximity and element missing judgment) to accurately identify complex violation scenarios with an accuracy rate of 92.6%.

[0027] In addition, for areas with a confidence level of less than 90% (usually accounting for less than 5% of the total map area), the system will push them to the manual review interface, where manual reviewers will correct the mislabeled areas by dragging the slider, reducing the average time to 20 seconds per image.

[0028] In addition, a pixel-level semantic segmentation-based intelligent labeling system for illegal road construction is provided to implement the above-mentioned method, including... Image acquisition module, used to acquire images of road construction via drone; The automatic annotation engine adopts the PSPNet semantic segmentation architecture, configures ResNet50 as the backbone network, and includes a pyramid pooling module to output segmentation heatmaps of different construction elements. The boundary optimization module is used to optimize the boundaries of the segmented heatmap; The violation decision output module includes a dynamic weighted fusion model for identifying complex violation scenarios, and the violation decision output module is used to output the annotation results.

[0029] The pyramid pooling module has pooling sizes of 1×1, 2×2, 3×3, and 6×6. It also includes a manual review interface for reviewing and correcting low-confidence areas.

[0030] This system takes road construction images captured by UAVs as input and obtains high-resolution original images through an image acquisition module. The acquired images are first transmitted to an automatic annotation engine, which is based on the PSPNet semantic segmentation architecture and configured with ResNet50 as the backbone network. ResNet50 extracts basic features through its 5-level convolutional structure (7×7 convolutional layers, 3×3 convolutional layers, and residual block sequences), and the output feature map size is 1 / 32 of the original image (for example, for a 1920×1080 input image, the output feature map size is 60×34). Its powerful feature extraction capability lays the foundation for subsequent processing.

[0031] The pyramid pooling module is the core component of the automatic annotation engine. It performs pooling operations at four scales in parallel: 1×1, 2×2, 3×3, and 6×6. This effectively captures multi-scale contextual information, from the global layout to small local targets (such as water-filled barriers and fence seams). For example, in complex urban scenes, 1×1 pooling is responsible for overall layout recognition, 2×2 pooling is used for feature extraction of large construction areas, 3×3 pooling is suitable for medium-sized targets such as equipment and fences, and 6×6 pooling focuses on feature extraction of small local objects. This multi-scale information fusion mechanism significantly improves the recognition accuracy of construction elements. The feature fusion layer integrates multi-level semantic information and restores the image to the original resolution through upsampling and stitching. This process not only preserves rich semantic details but also ensures high-resolution output of the segmentation results, providing high-quality input for subsequent boundary optimization and violation identification.

[0032] The boundary optimization module receives the segmentation heatmap output by the automatic annotation engine and uses the Conditional Random Field (CRF) algorithm for boundary optimization. The CRF algorithm optimizes the pixel category probability and the similarity constraint of adjacent pixels simultaneously through the energy function minimization operation, effectively solving the problem of blurred edges of small targets such as fences and water-filled barriers, and significantly improving the single-feature annotation accuracy.

[0033] The violation decision output module receives the optimized segmentation results and, based on a dynamic weighted fusion model and a decision tree rule engine, determines the spatial relationships of various construction elements to identify complex violation scenarios. For example, when it detects both "no fence in the construction area" and "equipment crossing the boundary," the system can accurately identify it as a complex violation scenario and output detailed annotation results. Simultaneously, a low-confidence area review mechanism is introduced. The system only pushes areas with a confidence level below 90% for manual review (typically less than 5% of the total map area). This mechanism significantly reduces the workload of manual review while ensuring annotation accuracy. Furthermore, operators can correct mislabeled areas by dragging a slider on the quality inspection interface, reducing the average time to 20 seconds per image. This lightweight review mechanism not only improves review efficiency but also reduces labor costs. In addition, the manual review results are fed back to the system in real time to optimize the dynamic weighted fusion model and decision tree rule engine. By continuously learning from manual review data, the system can gradually improve its ability to identify complex scenarios and adjust model parameters based on the manual review data, improving the model's accuracy and robustness. This closed-loop optimization mechanism ensures that the system maintains high performance throughout long-term operation.

[0034] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent annotation of illegal road construction based on pixel-level semantic segmentation, characterized in that, Includes the following steps: S1: Acquire the original image of the road construction scene; S2: The image obtained by processing based on a semantic segmentation network, wherein the semantic segmentation network includes a backbone network for feature extraction, a pyramid pooling module with multiple pooling sizes for capturing multi-scale contextual information, and a feature fusion layer; the feature fusion layer is used to fuse multi-level semantic information and output segmentation heatmaps of different construction elements. S3: Boundary optimization of segmented heatmaps based on the conditional random field algorithm; S4: Based on the segmentation results of multiple construction elements, identify and output composite violation scenarios.

2. The intelligent annotation method for illegal road construction based on pixel-level semantic segmentation according to claim 1, characterized in that, In S2, the semantic segmentation network is a PSPNet network, and the backbone network is a ResNet50 network; the pyramid pooling module performs pooling operations of four scales in parallel: 1×1, 2×2, 3×3, and 6×6, to obtain multi-scale contextual features covering the global layout to local small targets.

3. The intelligent annotation method for illegal road construction based on pixel-level semantic segmentation according to claim 2, characterized in that, The formula for the segmentation heatmap output by the feature fusion layer is as follows: 。 4. The intelligent annotation method for illegal road construction based on pixel-level semantic segmentation according to claim 3, characterized in that, In S2, the construction elements include the construction area, fence, water-filled barriers, and equipment.

5. The intelligent annotation method for illegal road construction based on pixel-level semantic segmentation according to claim 1, characterized in that, In S3, an energy function is used. Boundary optimization is performed, where, Let i represent the class probability of pixel i. This represents the similarity constraint between adjacent pixels i and j.

6. The intelligent annotation method for illegal road construction based on pixel-level semantic segmentation according to claim 1, characterized in that, In S4, a dynamic weighted fusion model is used to determine the spatial relationship of various construction elements in order to identify complex violation scenarios. For areas with a confidence level of less than 90%, manual lightweight review is performed, while the labeling results are directly output for the remaining areas.

7. The intelligent annotation method for illegal road construction based on pixel-level semantic segmentation according to claim 6, characterized in that, The dynamic weighted fusion model is based on FPN structure to align multi-scale outputs and uses a decision tree rule engine to determine the spatial relationships of elements.

8. A pixel-level semantic segmentation-based intelligent labeling system for illegal road construction, used to implement the method described in any one of claims 1 to 7, characterized in that, include Image acquisition module, used to acquire images of road construction via drone; The automatic annotation engine adopts the PSPNet semantic segmentation architecture, configures ResNet50 as the backbone network, and includes a pyramid pooling module to output segmentation heatmaps of different construction elements. The boundary optimization module is used to optimize the boundaries of the segmented heatmap; The violation decision output module includes a dynamic weighted fusion model for identifying complex violation scenarios, and the violation decision output module is used to output the annotation results.

9. The intelligent annotation system for illegal road construction based on pixel-level semantic segmentation according to claim 8, characterized in that, The pooling sizes of the pyramid pooling modules are 1×1, 2×2, 3×3, and 6×6.

10. The intelligent labeling system for illegal road construction based on pixel-level semantic segmentation according to claim 8, characterized in that, It also includes a manual review interface for reviewing and correcting low-confidence areas.

Images