Road green vision rate dynamic monitoring method based on target detection and semantic segmentation

Abstract

The invention relates to a road green vision rate dynamic monitoring method based on target detection and semantic segmentation. The method comprises the following steps: firstly, acquiring a road scene image through camera equipment mounted on a vehicle-mounted platform or a roadside fixing device, automatically identifying and framing a greening area in the image by using a YOLOv5s target detection model, and then cutting the area image in a detection frame; and inputting the obtained local image into a DeepLab + ResNeSt image semantic segmentation model which is pre-trained by adopting an ADE20K data set, and obtaining a pixel-level green plant area mask image. And counting the proportion of green pixels in the image based on the mask image, and further calculating a greening rate (green vision rate) index. In order to realize comparative analysis of images at different time or at different visual angles, an image feature point matching and registering method is further introduced, scene alignment and greening change difference analysis of different space-time images are realized, and finally greening identification and evaluation results are displayed in various visual modes such as image highlighting, numerical value labels and trend graphs.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a method for dynamic monitoring of road green visibility based on target detection and semantic segmentation. Background Technology

[0002] Urban greening is an important component of ecological civilization construction, and roadside green belts, as key units of the green urban space network, not only enhance the aesthetic appeal of the urban landscape but also have significant ecological benefits such as climate regulation, noise isolation, and air quality improvement. Currently, the assessment of green coverage is mainly applied to macro-ecological fields such as forestry and grassland resource monitoring. Its application in the identification and quantitative assessment of localized greening along urban roads is still in its early stages, with few practical application cases and a lack of efficient, automated, and comparable monitoring methods. Currently, the assessment of roadside green coverage relies heavily on traditional methods such as manual inspections, drone aerial photography, and manually labeled image analysis. These methods suffer from low identification efficiency, large subjective errors, and untimely data updates, making it difficult to support the large-scale, continuous greening supervision needs.

[0003] In response, Chinese patent CN118552598A discloses a method for calculating the green visibility rate of urban streets. The method includes the following steps: S1, dividing the streets to be evaluated into regions; S2, selecting sampling points; S3, acquiring images using a full-frame camera with a shooting height of 160cm±5cm, a 24mm fixed-focus lens, or a zoom lens with a 24mm focal length, and taking pictures of the streets according to the preset shooting angle of different types of sampling points; S4, preprocessing the acquired images to remove interference; S5, performing pixel traversal on the images; S6, analyzing the pixel RGB model of the image traversed in step S5, converting the pixel RGB color model into an HSV color model, and obtaining an HSV image. However, the above-mentioned prior art does not explicitly adopt an advanced deep learning model architecture, but only vaguely describes "image recognition model"; secondly, its green area recognition mainly relies on color channel histogram features, lacking a clear threshold definition in HSV space. Secondly, regarding multi-temporal image comparison, this patent uses histogram changes as the comparison basis, failing to consider spatial changes such as angles and positions between images, and thus lacks spatiotemporal alignment capabilities. Finally, it does not incorporate image registration technology and lacks modeling support for geometric transformations between images, limiting its accuracy in analyzing greening changes in complex scenarios. These shortcomings mainly stem from insufficient generality in the methodology's design and inadequate consideration of the unstructured characteristics of actual road images, resulting in significant deficiencies in its recognition robustness and comparative evaluation capabilities. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic monitoring method for road green visibility based on target detection and semantic segmentation to overcome the shortcomings of the existing technology.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A method for dynamic monitoring of road green view rate based on object detection and semantic segmentation includes:

[0007] Step S1: Acquire multi-source road images and the capture time of each image, and preprocess each road image;

[0008] Step S2: Select any two road images that correspond to the same road area but were taken at different times, and perform image registration and spatial alignment. The road image taken earlier is used as the reference image, and the road image taken later is used as the current image.

[0009] Step S3: Perform target detection on each reference image and the current image to obtain the green areas in the reference image and the current image, and crop the detection box area corresponding to the green area to use as the first image of the reference image and the current image respectively;

[0010] Step S4: Perform pixel-level semantic segmentation on each first image to obtain green vegetation regions, and generate a mask image of the same size as the first image corresponding to the green vegetation regions;

[0011] Step S5: For the first image corresponding to the same green area in the reference image and the current image, based on the corresponding mask image, traverse each pixel of the green vegetation area in any first image, determine whether each pixel is a green pixel, and take the ratio of the number of green pixels to the total number of pixels in the first image as the green visibility rate.

[0012] Step S6: Use the difference in green visibility between the current image and the reference image as the change in green visibility.

[0013] The preprocessing includes one or more of the following: brightness perturbation, contrast adjustment, random rotation, horizontal flipping, color perturbation, and slight blurring.

[0014] The target detection process in step S3 is implemented using the YOLOv5s model, and the YOLOv5s model training dataset is PascalVOC, a self-built greening annotation dataset, or a combination thereof.

[0015] The pixel-level semantic segmentation process in step S4 is implemented using the GluonCV platform and a pre-trained DeepLab-ResNeSt model. The DeepLab-ResNeSt model is pre-trained on public datasets such as COCO, Cityscapes, or ADE20K, and its parameters are loaded and deployed on a deep learning platform based on MXNet through GluonCV's Model Zoo.

[0016] The process of determining green pixels in step S5 includes:

[0017] Based on the mask image, each pixel of the first image is filtered, and the pixels in the first image other than the green vegetation area are set to 0 to obtain the segmentation mask image.

[0018] The segmentation mask image is converted to the HSV color space to obtain the second image;

[0019] Iterate through all pixels in the second image and designate all pixels with H∈[60,160], S>50, and V>40 as green pixels.

[0020] The image registration and spatial alignment process includes:

[0021] Step S2-1: Select a reference image and use SIFT, ORB, or SURF algorithms to extract feature points from the current image and the reference image;

[0022] Step S2-2: Based on the feature point matching results, use algorithms such as BFMatcher or FLANN for matching;

[0023] Step S2-3: By minimizing the reprojection error of the matching points, estimate the homography matrix H or affine matrix A to achieve registration between the current image and the reference image.

[0024] The homography matrix H is a 3×3 matrix used for perspective transformation registration, and the affine matrix A is a 2×3 matrix used for linear transformation registration.

[0025] The RANSAC algorithm is used to remove outliers from the matching feature points during the registration process.

[0026] A dynamic monitoring device for road green visibility based on target detection and semantic segmentation includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements the method described above.

[0027] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Through a multi-stage processing flow, the system enables functions such as green area identification, green visibility calculation, and historical comparative analysis, thereby improving the intelligence and precision of road greening management and achieving automated quantitative assessment of green coverage changes at different times or from multiple perspectives.

[0030] 2. Multi-source road video recording is adopted, including vehicle-mounted and roadside recordings, thus providing more data sources. On the one hand, the sampling cost is lower, and on the other hand, the data sources are richer.

[0031] 3. First detect the green area → then segment → then filter, so that the model calculation is more focused on the target area, which helps to reduce the false detection rate and optimize the overall processing efficiency.

[0032] 4. YOLOv5s detection and DeepLab segmentation are used in combination to enhance the accuracy and completeness of green area boundary recognition based on the division of labor between target localization and pixel segmentation.

[0033] 5. Employ multi-temporal image registration sets to assess changes in green visibility rate, thereby enabling more comparable time-series analysis in dynamic scenarios and enhancing the ability to judge road greening maintenance trends.

[0034] 6. Use the HSV color space and set explicit thresholds H∈[60,160], S>50, V>40 to improve the consistency of green pixel extraction and enhance the discrimination stability of images under different lighting and color conditions.

[0035] 7. Use methods such as ORB, BFMatcher, homography matrix H, and RANSAC for registration and alignment to ensure accurate alignment of images in different spatiotemporal regions in terms of geometric relationships, laying a spatially consistent foundation for subsequent index comparison.

[0036] 8. RANSAC is used to remove outliers in the matching process, thereby improving registration robustness and reducing evaluation bias caused by incorrect feature matching. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating the main steps of the method of the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0039] Example 1

[0040] A method for dynamic monitoring of road green view rate based on object detection and semantic segmentation, such as Figure 1 As shown, it includes:

[0041] Step S1: Acquire multi-source road images and the capture time of each image, and preprocess each road image;

[0042] In this embodiment, section A of the city's main road was selected as the test scenario. A sequence of forward road images was collected using a vehicle-mounted high-definition camera. The image resolution was 1280×720, and the acquisition time covered three time periods: morning, noon, and evening, ensuring that the sample was typical and diverse.

[0043] Extract still image frames from the captured video and perform the following processing steps in sequence:

[0044] (1) Size standardization: The original image is scaled to 512×512 pixels proportionally;

[0045] (2) Normalization: Map the RGB pixel values ​​of the image to the [0,1] interval;

[0046] (3) Data augmentation: Augmented images are constructed by methods such as brightness perturbation (±15%), rotation (±10°), and horizontal flipping to enhance the robustness of the model.

[0047] Step S2: Select any two road images that correspond to the same road area but were taken at different times, and perform image registration and spatial alignment. The road image taken earlier is used as the reference image, and the road image taken later is used as the current image.

[0048] To analyze the changes in greenery along the same road section at different times, images collected at two different times were selected for image registration and comparison:

[0049] (1) Key point extraction: The ORB (Oriented FAST and Rotated BRIEF) algorithm is used to extract feature points of the current and reference images, approximately 500 per image;

[0050] Feature point extraction and descriptor generation: The ORB algorithm (Oriented FAST and Rotated BRIEF) is used to extract feature points from the current and reference images. The ORB algorithm first finds feature points based on FAST corner detection and then calculates the corner strength using the Harris corner metric formula.

[0051] R = det(M) - k * (trace(M))² 2

[0052] Where M is the gray-level gradient matrix and k is the empirical coefficient.

[0053] Orientation calculation: To achieve rotation invariance, ORB calculates the principal orientation for each feature point:

[0054]

[0055] Where: x i yi The coordinates of the corner point's neighborhood, I(x) i ,y i () represents the grayscale value.

[0056] Descriptor generation: The orientation-compensated BRIEF algorithm generates rotation-invariant binary descriptors for feature points.

[0057] (2) Feature matching: Based on the BRIEF binary descriptor, similarity is calculated using Hamming distance (used to compare the number of different bits of binary feature points at each bit), and feature matching is completed using Brute-Force Matcher (BFMatcher).

[0058] Next, outlier removal is performed: To improve the accuracy and robustness of registration, the RANSAC (Random Sample Consensus) algorithm is used to remove outliers from the matched feature point pairs. RANSAC iteratively selects matching point pairs, estimates the homography matrix H, and determines the optimal model based on the number of interior points to effectively eliminate the impact of erroneous matches.

[0059] In another embodiment, the homography matrix H can be replaced by the affine matrix A to achieve registration between the current image and the reference image. The homography matrix H is a 3×3 matrix used for perspective transformation registration, and the affine matrix A is a 2×3 matrix used for linear transformation registration.

[0060] (3) Transformation estimation: The RANSAC algorithm is used to eliminate erroneous matches in the matching points and estimate the homography matrix H, which is used to achieve perspective or affine registration and spatially align the two images.

[0061] Step S3: Perform target detection on each reference image and the current image to obtain the green areas in the reference image and the current image, and crop the detection box area corresponding to the green area to use as the first image of the reference image and the current image respectively;

[0062] The target detection process in step S3 is implemented using the YOLOv5s model. The YOLOv5s model training dataset is PascalVOC, a self-built greening annotation dataset, or a combination thereof.

[0063] Specifically, in this embodiment, the main parameters are as follows:

[0064] Model architecture: YOLOv5s;

[0065] Training data: Cityscapes and a self-built green space annotation dataset;

[0066] Detection category: Set to a single category "greenery";

[0067] Input dimensions: 512×512;

[0068] Training parameters: learning rate 0.001, batch size 16, training 100 rounds;

[0069] Output: Each frame of the image outputs several detection boxes (formatted as x). center ,y center (width, height, confidence).

[0070] The model can efficiently identify green areas in areas such as sidewalks and green belts, with a target detection confidence level greater than 0.9.

[0071] Step S4: Perform pixel-level semantic segmentation on each first image to obtain green vegetation regions, and generate a mask image of the same size as the first image corresponding to the green vegetation regions;

[0072] The pixel-level semantic segmentation process in step S4 is implemented using the GluonCV platform and a pre-trained DeepLab-ResNeSt model. The DeepLab-ResNeSt model is pre-trained on public datasets such as COCO, Cityscapes, or ADE20K, and its parameters are loaded and deployed on a deep learning platform based on MXNet through GluonCV's Model Zoo.

[0073] The target bounding boxes identified by YOLO are cropped to generate several sub-images (e.g., 240×300, 180×250 pixels). The cropped images are then input into the DeepLabV3+ model for semantic segmentation.

[0074] Model configuration:

[0075] Platform tool: GluonCV;

[0076] Backbone network: ResNeSt-50;

[0077] Pre-trained weights: using the COCO or ADE20K dataset, with 20 rounds of transfer learning;

[0078] Input and output: The input is an RGB image, and the output is a semantic mask for each pixel.

[0079] In the mask image output by semantic segmentation, green vegetation areas are accurately segmented with clear boundaries, while non-green areas (such as roads, vehicles, buildings, etc.) are effectively removed.

[0080] Step S5: For the first image corresponding to the same green area in the reference image and the current image, based on the corresponding mask image, traverse each pixel of the green vegetation area in any first image, determine whether each pixel is a green pixel, and take the ratio of the number of green pixels to the total number of pixels in the first image as the green visibility rate.

[0081] The process of identifying green pixels includes:

[0082] Based on the mask image, each pixel of the first image is filtered, and the pixels in the first image other than the green vegetation area are set to 0 to obtain the segmentation mask image.

[0083] The segmentation mask image is converted to the HSV color space to obtain the second image;

[0084] Iterate through all pixels in the second image and designate all pixels with H∈[60,160], S>50, and V>40 as green pixels.

[0085] Step S6: Use the difference in green visibility between the current image and the reference image as the change in green visibility:

[0086] ΔG=G current -G reference

[0087] Where: ΔG is the change in green visibility, G current G represents the green view rate of the current image. reference The green view rate is used as a reference image.

[0088] For example, if the current image green view ratio is 0.378 and the reference image is 0.295, then ΔG = +0.083, and the system determines the greening status as "significantly improved". When |ΔG| > 0.05|, it is determined that the greening level has changed significantly.

[0089] Finally, the system generates the following visualization results:

[0090] (1) The green area in the original image is highlighted;

[0091] (2) Overlay green visibility rate labels on each detection area;

[0092] (3) Add a “Greenery Improvement ↑” or “Greenery Degradation ↓” marker below the image;

[0093] (4) Simultaneously export CSV format data files to record changes in green visibility rate for easy analysis and archiving.

[0094] Table 1 shows the consistency between the system evaluation results and the human judgment results in 20 typical image pairs.

[0095] Table 1

[0096]

[0097]

[0098] Experimental data show that when the absolute value of the green visibility change value ΔG exceeds 0.05, the consistency between the method of the present invention and the result of human judgment reaches 100%, indicating that the method has excellent accuracy, stability and practical application value in assessing the trend of road greening changes.

[0099] Comparative Example 1

[0100] This comparative example is largely the same as Example 1. The difference between this comparative example and Example 1 is that, in this comparative example, an image difference registration method based on pixel grayscale histograms is used to test road images taken at different times before and after the shooting. The results are as follows:

[0101] Test results show that in the presence of viewing angle shifts, obstructions, or changes in lighting, Comparative Example 1 exhibits registration errors, with an average green area overlap rate of less than 85%, leading to inaccuracies in the corresponding areas during subsequent analysis and significant errors in green visibility comparison. In contrast, Example 1, through feature point matching combined with RANSAC to remove outliers, achieves a stable area overlap rate of over 95% after registration, significantly improving spatial alignment accuracy and providing a reliable foundation for subsequent comparative analysis.

[0102] Comparative Example 2

[0103] This comparative example is largely the same as Example 1. The difference between this comparative example and Example 1 is that, in this comparative example,

[0104] In step S5, when determining the green pixels, the method based on color channel or histogram features, as used in the Chinese patent, was used for testing, and the results are as follows:

[0105] In complex scenes such as strong light and shadow, the recognition accuracy of Comparative Example 2 decreased, with an average false positive rate of 12% for green pixels, and significant over-detection or under-detection occurred in some images. In contrast, Example 1, which combines semantic segmentation masking with a three-channel HSV joint screening mechanism, controlled the false positive rate of green pixels to within 5%, and achieved better results in terms of lighting adaptability and extraction stability.

[0106] Comparative Example 3

[0107] This comparative example is largely the same as Example 1. The difference between this comparative example and Example 1 is that, in this comparative example, edge line detection is used for error processing during image registration and alignment, instead of using RANSAC to remove outliers. The test results are as follows:

[0108] The tests revealed that Comparative Example 3 was sensitive to feature point mismatches; when the in-place point ratio of the matched points was below 60%, the registration failure rate exceeded 25%, leading to misalignment of the evaluation image and poor mask alignment. In contrast, Example 1 used RANSAC to filter out mismatches, increasing the in-place point ratio during the registration process, achieving a registration success rate of over 95%, and demonstrating higher robustness in different scenarios.

[0109] Comparative Example 4

[0110] This comparative example is largely the same as Example 1. The difference between this comparative example and Example 1 is that in this comparative example, when determining green pixels in step S5, the entire image is traversed through HSV filters to obtain green pixels, instead of first detecting green areas → then segmenting → then filtering. The test results are as follows:

[0111] When the proportion of green areas is low or there are green distractions (such as vehicles or advertising signs), the overall green visibility rate in Example 4 is 5% to 10% higher, which is significantly different from the results of manual annotation. In contrast, Example 1 uses a three-level processing flow of "detection → segmentation → screening" to extract pixels only from the actual green areas, and the green visibility rate error is less than 3% on average, which greatly improves the accuracy and consistency of the evaluation.

[0112] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for dynamic monitoring of road green view rate based on object detection and semantic segmentation, characterized in that, include: Step S1: Acquire multi-source road images and the capture time of each image, and preprocess each road image; Step S2: Select any two road images that correspond to the same road area but were taken at different times, and perform image registration and spatial alignment. The road image taken earlier is used as the reference image, and the road image taken later is used as the current image. Step S3: Perform target detection on each reference image and the current image to obtain the green areas in the reference image and the current image, and crop the detection box area corresponding to the green area to use as the first image of the reference image and the current image respectively; Step S4: Perform pixel-level semantic segmentation on each first image to obtain green vegetation regions, and generate a mask image of the same size as the first image corresponding to the green vegetation regions; Step S5: For the first image corresponding to the same green area in the reference image and the current image, based on the corresponding mask image, traverse each pixel of the green vegetation area in any first image, determine whether each pixel is a green pixel, and take the ratio of the number of green pixels to the total number of pixels in the first image as the green visibility rate. Step S6: Use the difference in green visibility between the current image and the reference image as the change in green visibility.

2. The method for dynamic monitoring of road green view rate based on target detection and semantic segmentation according to claim 1, characterized in that, The preprocessing includes one or more of the following: brightness perturbation, contrast adjustment, random rotation, horizontal flipping, color perturbation, and slight blurring.

3. The method for dynamic monitoring of road green view rate based on target detection and semantic segmentation according to claim 1, characterized in that, The target detection process in step S3 is implemented using the YOLOv5s model, and the YOLOv5s model training dataset is PascalVOC, a self-built greening annotation dataset, or a combination thereof.

4. The method for dynamic monitoring of road green view rate based on target detection and semantic segmentation according to claim 1, characterized in that, The pixel-level semantic segmentation process in step S4 is implemented using the GluonCV platform and a pre-trained DeepLab-ResNeSt model. The DeepLab-ResNeSt model is pre-trained on public datasets such as COCO, Cityscapes, or ADE20K, and its parameters are loaded and deployed on a deep learning platform based on MXNet through GluonCV's Model Zoo.

5. The method for dynamic monitoring of road green view rate based on target detection and semantic segmentation according to claim 1, characterized in that, The process of determining green pixels in step S5 includes: Based on the mask image, each pixel of the first image is filtered, and the pixels in the first image other than the green vegetation area are set to 0 to obtain the segmentation mask image. The segmentation mask image is converted to the HSV color space to obtain the second image; Iterate through all pixels in the second image and designate all pixels with H∈[60,160], S>50, and V>40 as green pixels.

6. The method for dynamic monitoring of road green view rate based on target detection and semantic segmentation according to claim 1, characterized in that, The image registration and spatial alignment process includes: Step S2-1: Select a reference image and use SIFT, ORB, or SURF algorithms to extract feature points from the current image and the reference image; Step S2-2: Based on the feature point matching results, use algorithms such as BFMatcher or FLANN for matching; Step S2-3: By minimizing the reprojection error of the matching points, estimate the homography matrix H or affine matrix A to achieve registration between the current image and the reference image.

7. The method for dynamic monitoring of road green view rate based on target detection and semantic segmentation according to claim 6, characterized in that, The homography matrix H is a 3×3 matrix used for perspective transformation registration, and the affine matrix A is a 2×3 matrix used for linear transformation registration.

8. The method for dynamic monitoring of road green view rate based on target detection and semantic segmentation according to claim 6, characterized in that, The RANSAC algorithm is used to remove outliers from the matching feature points during the registration process.

9. A dynamic monitoring device for road green view rate based on target detection and semantic segmentation, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images