Road sign defect detection method and system based on Transform and medium

Abstract

The invention discloses a road sign defect detection method and system based on Transform and a medium, and the method comprises the steps: firstly, obtaining an image, carrying out the preprocessing of the image, carrying out the road sign region segmentation of the image through employing a Mask2Former algorithm, and generating a Mask contour of a road sign; based on the minimum enclosing rectangular frame of the road identification area, segmenting the original image to obtain a plurality of sub-images containing the identification area; then, a YOLO-WORLD multi-mode model is used for conducting defect detection on the sub-graph rectangular frame of the road mark, the defects of the road mark are detected on the sub-graph rectangular frame, and a defect positioning frame is generated; and finally, in combination with the Mask contour, filtering the pixel positions of the defects, and filtering out the defects outside the road marking area. According to the method, two excellent algorithm models of Mask2Former and YOLO-WORLD are creatively combined, accurate road marking area segmentation in a complex environment is realized, the detection precision and robustness are improved, the false alarm rate is remarkably reduced, and the detection reliability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of road sign defect detection, and in particular relates to a road sign defect detection method, system and medium based on Transformer. Background Art

[0002] Traditional daily inspections of urban roads mainly rely on inspectors patrolling on the road, or the public's voluntary supervision and reporting. For the defects of road traffic signs found, they are often reported manually through filling out paper forms or reporting manually by telephone feedback, which is inefficient. In addition, this method has strong subjectivity and lag. It is difficult for different inspectors or the public to maintain consistent cognition of the degree of the same traffic sign defect, and the public's consciousness is inconsistent, making it difficult to report in a timely manner. In addition, the length of roads inspected by inspectors every day is generally between 30-40km, which is a short mileage. Therefore, the use of computer technology to conduct intelligent analysis of road traffic sign defects is of great practical significance. However, the current road traffic sign defect detection methods have the following problems: 1. Insufficient segmentation accuracy. Traditional edge detection or simple convolutional network segmentation algorithms are difficult to accurately segment the road sign area, resulting in errors in subsequent defect detection. In addition, the single-stage defect detection algorithm is easily affected by environmental factors, and the defect detection accuracy is low.

[0003] 2. The existing detection methods do not accurately determine whether the defect is located in the marked area, resulting in the detection results being unable to be used for actual maintenance tasks and a high false alarm rate. Summary of the invention

[0004] The purpose of the present invention is to provide a road sign defect detection method, system and medium based on Transformer, aiming to solve the above-mentioned problems.

[0005] The present invention is mainly achieved through the following technical solutions: The road sign defect detection method based on Transformer includes the following steps: Step S1: First, after acquiring and preprocessing the image, the image is segmented into road sign areas using the Mask2Former algorithm to generate a Mask outline of the road sign; Step S2: Based on the minimum circumscribed rectangular frame of the road sign area, a number of sub-images containing the sign area are segmented from the original image; then, the YOLO-WORLD multimodal model is used to perform defect detection on the sub-image rectangular frame of the road sign, and the defects of the road sign are detected on the sub-image rectangular frame and a defect location frame is generated; Step S3: Finally, the pixel positions of the defects detected in step S2 are filtered in combination with the Mask contour obtained by segmentation in step S1 to filter out the defects outside the road marking area, ensuring that the detection results are only within the road marking area.

[0006] In order to better implement the present invention, further, in step S1, based on the image captured by the vehicle traveling on the road, the image preprocessing includes the following steps: Step S11: using Min-Max normalization algorithm to perform image normalization; Step S12: using Gaussian filtering or median filtering to remove noise; Step S13: Image enhancement: Adjust the contrast of the image to make the sign area more prominent and enhance the performance of the segmentation model. The histogram equalization method is used to make the grayscale distribution of the image more uniform, thereby highlighting the outline and details of the road sign. Then, adjust the brightness and contrast of the image to ensure that the image remains consistent under different lighting conditions. Step S14: Based on the reverse mapping distortion correction algorithm, the camera is calibrated to obtain its internal parameter matrix and distortion coefficient, and the image is subjected to distortion correction.

[0007] In order to better implement the present invention, further, step S2 includes the following steps: Step S21: sub-image cropping: based on the mask outline of the road sign generated in step S1, extract the minimum circumscribed rectangular frame of the sign area, and crop the area from the original image to obtain a sub-image, and the sub-image contains the complete sign area; Step S22: Use the YOLO-WORLD multimodal model to perform defect detection on the sub-image; Step S221: First, the YOLO-WORLD multimodal model is used to divide the sub-image into several sub-regions, and features are extracted for each region to generate possible defect candidate boxes; Step S222: Then, a multi-scale detection mechanism is used to identify defect types at different scales; Step S223: Finally, the detection results are output: the model outputs the category, confidence level and pixel position coordinates of each detected defect, where the pixel position coordinates represent the relative position of the defect in the sub-image of the road sign.

[0008] In order to better implement the present invention, further, step S3 includes the following steps: Step S31: Position mapping: convert the pixel coordinates of the defect output by the YOLO-WORLD multimodal model into coordinates relative to the original image, and ensure the accurate positioning of the defect position through the geometric mapping relationship between the sub-image and the original image; Step S32: Filtering algorithm: Defects within the Mask outline of the road sign are considered valid defects, and defects beyond the Mask outline are filtered out.

[0009] In order to better implement the present invention, further, step S4 is included: visually displaying the defect results finally detected, and generating a defect report including the defect type, location and detection confidence.

[0010] In order to better implement the present invention, further, in step S4, the defect report also includes a confidence score of the detection result and defect type statistics, showing the number and distribution of various types of defects; when visually displayed, different types of defects are marked with different colors in the output detection image, and their specific pixel positions are displayed.

[0011] The present invention is mainly achieved through the following technical solutions: The Transformer-based road sign defect detection system is based on the above-mentioned Transformer-based road sign defect detection method, and includes an image acquisition module, an image preprocessing module, a Mask contour generation module, a defect detection module and a visualization display module; the Mask contour generation module is used to perform road sign area segmentation on the image based on the Mask2Former algorithm to generate the Mask contour of the road sign; the defect detection module is used to perform defect detection on the sub-image based on the YOLO-WORLD multimodal model; the visualization display module is used to display the defect detection results and generate a defect report.

[0012] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned Transformer-based road sign defect detection method.

[0013] The beneficial effects of the present invention are as follows: (1) The present invention innovatively combines two excellent algorithm models, Mask2Former and YOLO-WORLD. First, Mask2Former realizes accurate road sign area segmentation in complex environments by combining visual Transformer with traditional semantic segmentation network. Subsequently, the segmented sign area is subjected to defect detection by the YOLO-WORLD multimodal detection model, further improving the detection accuracy and robustness. Through these two algorithms, the present invention ensures the accuracy of segmentation and realizes real-time and high-precision defect detection. The present invention compares and filters the position of the detected defect with the outline of the sign area to ensure that the detected defect is only located within the road sign area, thereby significantly reducing the false alarm rate and improving the reliability of detection.

[0014] (2) Accurate segmentation: The present invention uses the Mask2Former algorithm to accurately segment road signs in complex scenes and reduce the impact of environmental factors on the segmentation of sign contours. The visual Transformer in Mask2Former has a powerful global information capture capability and is particularly suitable for capturing the global contour information of road signs in complex environments. Traditional convolutional networks can usually only process local information, while the visual Transformer can capture long-distance dependencies in images through a multi-level self-attention mechanism. This enables it to more accurately segment the complete road sign area in situations such as shadows and uneven lighting.

[0015] (3) Multimodal detection: The YOLO-WORLD model can more accurately detect subtle defects by integrating multimodal data. It can effectively detect various types of marking defects and improve the accuracy and robustness of detection.

[0016] (4) Position-accurate filtering: By combining defect detection with segmentation results, the defect location is accurately filtered, false alarms are reduced, and detection accuracy is improved. The identification area segmented by Mask2Former provides an accurate positioning reference for YOLO-WORLD detection. By processing these areas, YOLO-WORLD can perform defect detection more efficiently and accurately in relatively small sub-images. This strategy of region cropping and focusing on specific areas reduces the computational burden of the model while avoiding an increase in the false alarm rate, which has significant advantages, especially in defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The figure is a flow chart of the road sign defect detection method based on Transformer of the present invention. DETAILED DESCRIPTION

[0018] Embodiment 1: A road sign defect detection method based on Transformer uses the Mask2Former algorithm to segment the sign area, uses the YOLO-WORLD model to detect defects, and combines the segmented contour to filter the defect position, thereby accurately locating the real defects on the road sign. Specifically, the road sign area is first segmented by the Mask2Former algorithm to obtain the outline of the road sign; the YOLO-WORLD model is used to detect the defects of the sign on the sub-image rectangular frame of the sign; finally, the pixel position of the detected defect is filtered in combination with the segmented contour to ensure that the detection result is only within the sign area, thereby accurately locating the real defects on the road sign.

[0019] Preferably, the present invention specifically comprises the following steps: Step 1: Obtain the corresponding image dataset by photographing vehicles traveling on the road. After preprocessing the image, use the Mask2Former algorithm to segment the image into road sign areas and generate the Mask contour of the road sign.

[0020] The Mask2Former algorithm is an algorithm that combines the visual Transformer with the traditional segmentation network, aiming to solve the problem of accurate segmentation in complex scenes. Its working mechanism includes multi-scale feature extraction, that is, through the Transformer model, Mask2Former can effectively capture global information from the image, and combine the local feature extraction ability of the semantic segmentation network to achieve fine segmentation. And adopts a multi-stage segmentation strategy. The algorithm segments road signs step by step from coarse to fine in a step-by-step manner to ensure that accurate segmentation results can still be obtained under complex backgrounds. The advantage of Mask2Former lies in its powerful global feature capture capability, which makes it particularly suitable for processing the segmentation of target objects such as road signs that have fixed forms but are easily disturbed by the environment.

[0021] Step 2: Use the YOLO-WORLD multimodal model to perform defect detection on the sub-image rectangular frame of the road sign and generate a defect location frame.

[0022] The YOLO-WORLD multimodal model is a multimodal detection model extended from the YOLO model, which is mainly used for defect detection. The innovation of this model is the introduction of multimodal information fusion technology, which can combine information from multiple data sources such as optical images, thermal images or depth maps to further improve detection accuracy. The specific features include the following: First, real-time detection. YOLO-WORLD retains the real-time detection capability of the YOLO model and can quickly process high-resolution road images. Second, multimodal fusion. By fusing different types of input data, YOLO-WORLD can still maintain a high detection accuracy when facing complex scenes such as uneven lighting and minor damage to signs.

[0023] Step 3: Filter and retain valid defects in the marked area.

[0024] After the defect detection is completed, the present invention further filters out those false detection results that are outside the sign area by combining the detection results with the previously generated sign Mask outline. The key to this step is to ensure that the detected defects are actually located on the road sign, thereby reducing the false alarm rate and improving the detection accuracy.

[0025] Step 4: Visualize the detection results.

[0026] The final detected defect results are displayed visually, and a defect report including defect type, location and detection confidence is generated to facilitate subsequent identification maintenance and repair work.

[0027] Embodiment 2: A Transformer-based road sign defect detection method that comprehensively utilizes the global feature capture capability of the Transformer in the Mask2Former algorithm and the advantages of the YOLO-WORLD multimodal detection model can achieve high-precision road sign defect detection in complex scenes. Figure 1 As shown, the specific steps include: Step 1: Road sign segmentation.

[0028] Collect images, and after preprocessing, input the images into the Mask2Former algorithm model to generate accurate segmentation contours (Mask) of the signs. The Mask2Former algorithm combines the global feature capture capability of the Transformer and the local detail processing capability of the traditional semantic segmentation network, and is particularly suitable for processing road sign segmentation problems in complex scenarios. Specific operations include: Step 1.1, image acquisition and preprocessing.

[0029] Vehicles traveling on the road are equipped with cameras to continuously capture road images. These images are transmitted to the inspection platform through the data acquisition system. Before defect detection, the images need to be preprocessed in a series of steps. These preprocessing steps are the basis for ensuring the stable operation of subsequent algorithms. They include: Image normalization: Image normalization aims to standardize images taken under different conditions so that the detection algorithm is robust to external factors such as lighting changes. The specific method used is the Min-Max normalization algorithm.

[0030] Noise removal: In road scenes, the vibration, illumination changes and other factors often cause noise in the image during vehicle driving, which affects the subsequent segmentation and detection results. In order to eliminate these noises, Gaussian filtering and median filtering methods are used.

[0031] Image enhancement: In order to improve the accuracy of subsequent segmentation and detection, the image is enhanced to a certain extent, including: (1) Histogram equalization: By adjusting the contrast of the image, the sign area is made more prominent and the performance of the segmentation model is enhanced. In this process, the histogram equalization technique is used to make the grayscale distribution of the image more uniform, thereby highlighting the outline and details of the road sign.

[0032] (2) Brightness and contrast adjustment: Automatically adjust the brightness and contrast of the image according to the lighting conditions of the shooting environment to ensure that the image remains consistent under different lighting conditions.

[0033] Distortion correction: Due to the camera installation location and the actual road conditions, image distortion may occur, especially fisheye distortion. This will cause deviations in the detection of subsequent algorithms, so we use a reverse mapping distortion correction algorithm. By calibrating the camera, obtaining its intrinsic parameter matrix and distortion coefficient, the image is corrected.

[0034] Step 1.2, feature extraction and segmentation.

[0035] Through the Mask2Former model, the multi-scale feature map of the image is extracted to capture the global features of the sign area. The Transformer module can adaptively capture the morphology and location information of the sign at different levels of the image. A multi-stage segmentation strategy is adopted to refine different areas in the image and gradually improve the accuracy of sign segmentation. The output result is the binary Mask outline of the sign, which represents the precise boundary of the road sign.

[0036] Step 2: Defect detection based on identification subgraph.

[0037] The minimum bounding rectangle of the sign is determined by the sign mask information obtained in the first step. The minimum bounding rectangle sub-image containing the sign area is cut out from the original image for further defect detection. The YOLO-WORLD multimodal model is used for defect detection. After fine-tuning and training, the model can identify multiple defect types on different road sign sub-images. The specific steps include: Sub-image cropping: Based on the logo outline generated by Mask2Former, the minimum bounding rectangle of the logo area is extracted and the area is cropped out from the original image. The sub-image contains the complete logo area and can be used for defect detection.

[0038] Model fine-tuning: For the segmented logo area, the YOLO-WORLD multimodal model is used to perform defect detection on the logo sub-image. The model works in the following way during the detection process: First, the YOLO-WORLD model divides the image into several sub-regions, extracts features for each region, and generates possible defect candidate boxes. Then, a multi-scale detection mechanism is used to identify defect types at different scales, such as cracks, blur, and damage. Finally, the detection results are output, and the model outputs the category, confidence, and pixel position coordinates of each detected defect. These coordinates represent the relative position of the defect in the logo sub-image.

[0039] Step 3: Defect location filtering.

[0040] In order to ensure the accuracy of defect detection, the present invention further performs position filtering on the detected defects. The specific method is to compare the pixel coordinates of the defects with the identification Mask contour generated in the first step to determine whether these defects are located in the identification area. The detailed steps are as follows: Position mapping: The defect position output by the YOLO-WORLD model is converted into coordinates relative to the original image. The geometric mapping relationship between the sub-image and the original image is used to ensure the accurate positioning of the defect position.

[0041] Filtering algorithm: Filter the detected defects. Only the defects inside the identified Mask outline are considered valid defects. Defects outside the outline will be filtered out to reduce false detection.

[0042] Step 4: Visualize and output the detection results.

[0043] Final output: Based on the filtered results, the precise defect location is output and a detailed defect report is generated, which includes the defect type, location and detection confidence. The present invention will finally display the detection results visually. Specifically including: Defect type statistics: The defect report includes the number and distribution of various types of defects, such as cracks, contamination, etc.

[0044] Defect location marking: In the output inspection image, different types of defects are marked with different colors and their specific pixel locations are displayed to facilitate subsequent maintenance and repair.

[0045] Detection confidence analysis: The report also includes the confidence score of the test results to help understand the reliability of the test results and provide a reference for subsequent identification maintenance decisions.

[0046] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A road sign defect detection method based on Transformer, characterized in that: The following steps are involved: Step S1: First, after acquiring and preprocessing the image, the image is segmented into road sign areas using the Mask2Former algorithm to generate a Mask outline of the road sign; Step S2: Based on the minimum circumscribed rectangular frame of the road sign area, a number of sub-images containing the sign area are segmented from the original image; then, the YOLO-WORLD multimodal model is used to perform defect detection on the sub-image rectangular frame of the road sign, and the defects of the road sign are detected on the sub-image rectangular frame and a defect location frame is generated; Step S3: Finally, the pixel positions of the defects detected in step S2 are filtered in combination with the Mask contour obtained by segmentation in step S1 to filter out the defects outside the road marking area, ensuring that the detection results are only within the road marking area.

2. The road sign defect detection method based on Transformer according to claim 1 is characterized in that: In step S1, based on capturing an image of a vehicle traveling on a road, image preprocessing includes the following steps: Step S11: using Min-Max normalization algorithm to perform image normalization; Step S12: using Gaussian filtering or median filtering to remove noise; Step S13: Image enhancement: Adjust the contrast of the image to make the sign area more prominent and enhance the performance of the segmentation model. The histogram equalization method is used to make the grayscale distribution of the image more uniform, thereby highlighting the outline and details of the road sign. Then, adjust the brightness and contrast of the image to ensure that the image remains consistent under different lighting conditions. Step S14: Based on the reverse mapping distortion correction algorithm, the camera is calibrated to obtain its internal parameter matrix and distortion coefficient, and the image is subjected to distortion correction.

3. The road sign defect detection method based on Transformer according to claim 1, characterized in that: The step S2 comprises the following steps: Step S21: sub-image cropping: based on the mask outline of the road sign generated in step S1, extract the minimum circumscribed rectangular frame of the sign area, and crop the area from the original image to obtain a sub-image, and the sub-image contains the complete sign area; Step S22: Use the YOLO-WORLD multimodal model to perform defect detection on the sub-image; Step S221: First, the YOLO-WORLD multimodal model is used to divide the sub-image into several sub-regions, and features are extracted for each region to generate possible defect candidate boxes; Step S222: Then, a multi-scale detection mechanism is used to identify defect types at different scales; Step S223: Finally, the detection results are output: the model outputs the category, confidence level and pixel position coordinates of each detected defect, where the pixel position coordinates represent the relative position of the defect in the sub-image of the road sign.

4. The road sign defect detection method based on Transformer according to claim 3 is characterized in that: The step S3 comprises the following steps: Step S31: Position mapping: convert the pixel coordinates of the defect output by the YOLO-WORLD multimodal model into coordinates relative to the original image, and ensure the accurate positioning of the defect position through the geometric mapping relationship between the sub-image and the original image; Step S32: Filtering algorithm: Defects within the Mask outline of the road sign are considered valid defects, and defects beyond the Mask outline are filtered out.

5. The road sign defect detection method based on Transformer according to any one of claims 1 to 4, characterized in that: The method also includes step S4: visually displaying the final detected defect results and generating a defect report including the defect type, location and detection confidence.

6. The road sign defect detection method based on Transformer according to claim 5 is characterized in that: In step S4, the defect report also includes the confidence score of the detection result and the defect type statistics, showing the number and distribution of various types of defects; when visually displayed, different types of defects are marked with different colors in the output detection image, and their specific pixel positions are displayed.

7. A Transformer-based road sign defect detection system, based on the Transformer-based road sign defect detection method according to any one of claims 1 to 6, characterized in that: It includes an image acquisition module, an image preprocessing module, a Mask contour generation module, a defect detection module and a visualization display module; the Mask contour generation module is used to segment the image into road sign areas based on the Mask2Former algorithm to generate the Mask contour of the road sign; The defect detection module is used to perform defect detection on the sub-graph based on the YOLO-WORLD multimodal model; the visualization display module is used to display the defect detection results and generate a defect report.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the Transformer-based road sign defect detection method described in any one of claims 1 to 6 is implemented.

Images