SAM and Transformer-based traffic sign defect detection method and system, and medium

Abstract

The invention discloses a traffic sign defect detection method and system based on SAM and Transform and a medium, and the method comprises the steps: collecting an image of a traffic sign, carrying out the preprocessing, recognizing the traffic sign in the image through a YOLO target detection model, generating a detection frame of the traffic sign, and framing a region where the traffic sign is located; inputting the detection frame and the corresponding image into an SAM model, and extracting the contour of the traffic sign; and simultaneously inputting the extracted contours of the traffic signs and contours of the traffic signs in a normal state in the database into a neural network classifier based on Transform for comparative analysis, finally outputting a defect detection result, and giving a specific defect type and a confidence score. According to the method, the YOLO target detection, the SAM model and the neural network classifier based on the Transform are combined, and the task of each stage can be optimized on the basis of the result of the previous stage, so that the final detection precision and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of road traffic sign defect detection, and in particular relates to a traffic sign defect detection method, system and medium based on SAM and Transformer. Background Art

[0002] Traditional daily inspections of urban roads mainly rely on inspectors patrolling on the road, or the public's voluntary supervision and reporting. For the defects of road traffic signs found, they are often reported manually by filling out paper forms or reporting manually by telephone feedback, which is inefficient. In addition, this method has strong subjectivity and lag. It is difficult for different inspectors or the public to maintain consistent cognition of the degree of the same traffic sign defect, and the public's consciousness is inconsistent, making it difficult to report in time. In addition, the length of the road inspected by inspectors every day is generally between 30-40km, which is a short mileage. Therefore, the use of computer technology to conduct intelligent analysis of road traffic sign defects is of great practical significance.

[0003] However, in the field of road traffic sign defect detection, traditional defect detection methods usually rely on small-scale data sets for training, lack the ability to adapt to large-scale and diverse data, and have poor generalization capabilities. Secondly, current defect detection methods usually rely on single image input for defect recognition, and there is no mechanism to compare and analyze the contour of the currently detected sign with the contour of the sign in the previous normal state. This method makes it difficult to accurately determine whether the sign has undergone shape changes or structural damage. Summary of the invention

[0004] The purpose of the present invention is to provide a traffic sign defect detection method, system and medium based on SAM and Transformer, aiming to solve the above-mentioned problems.

[0005] The present invention is mainly achieved through the following technical solutions: The traffic sign defect detection method based on SAM and Transformer includes the following steps: Step S1: First, collect and pre-process the image of the traffic sign, use the YOLO target detection model to identify the traffic sign in the image, and generate a detection frame of the traffic sign to frame the area where the traffic sign is located; Step S2: Then, the detection frame and the corresponding image are input into the SAM model to extract the outline of the traffic sign; Step S3: The outline of the traffic sign extracted in step S2 and the outline of the traffic sign in the normal state in the database are simultaneously input into the Transformer-based neural network classifier for comparative analysis, and finally the defect detection result is output, giving the specific defect type and confidence score.

[0006] In order to better implement the present invention, further, in step S1, the YOLO target detection model first divides the image into multiple grids, and predicts whether there is a traffic sign and the category of the traffic sign for each grid, and generates a detection frame including position coordinates and category labels.

[0007] In order to better implement the present invention, further, in step S2, the SAM model determines the preliminary identification area based on the detection frame output by the YOLO target detection model, and uses a multi-layer feature pyramid structure to refine the preliminary identification area layer by layer, and finally generates the outline of the traffic sign and captures the boundary shape and details of the traffic sign.

[0008] In order to better implement the present invention, further, in step S3, the Transformer-based neural network classifier extracts local and global features of the sign contour layer by layer, and analyzes the similarities and differences between the two; the Transformer-based neural network classifier calculates the differences in different areas in parallel through multiple attention heads to identify the defects of the traffic sign.

[0009] In order to better implement the present invention, further, in the step S3, after completing the contour comparison analysis, the defect detection and classification stage is entered; the Transformer-based neural network classifier extracts and analyzes the comparison results, and finally outputs whether there is a traffic sign defect, as well as the specific type of the defect and the corresponding confidence score.

[0010] In order to better implement the present invention, further, in the step S1, images of road traffic signs are collected based on vehicles traveling on the road.

[0011] The present invention is mainly achieved through the following technical solutions: The traffic sign defect detection system based on SAM and Transformer is based on the above-mentioned traffic sign defect detection method based on SAM and Transformer, and includes an image acquisition and preprocessing module, a target detection module, a contour detection module and a defect detection module; the target detection module is used to perform preliminary detection of traffic signs in the image and generate a detection frame of the traffic sign; the contour detection module is used to extract the contour of the traffic sign based on the SAM model; the defect detection module is used to output the defect detection result through a Transformer-based neural network classifier, and give a specific defect type and confidence score.

[0012] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned traffic sign defect detection method based on SAM and Transformer.

[0013] The beneficial effects of the present invention are as follows: (1) The present invention proposes an innovative method of combining the YOLO target detection model with the visual large model SAM (Segment Anything Model). The YOLO model is responsible for quickly and accurately detecting the location of traffic signs and generating a detection frame, while the SAM model further accurately extracts the outline of the traffic sign through the detection frame. By introducing the visual large model SAM, the present invention can further extract a more accurate sign outline from the sign frame detected by the YOLO model. Compared with traditional edge detection algorithms, the SAM model can better cope with situations such as complex backgrounds and generate more accurate sign outlines. This method significantly improves the accuracy of traffic sign outline extraction, especially in scenes with complex backgrounds or uneven lighting. Compared with traditional edge detection and target detection methods, the present invention has significantly improved the accuracy of sign outlines and detection efficiency.

[0014] (2) The present invention uses a Transformer-based neural network classifier (ViT) to detect defects in traffic signs. The model compares the precise outline of the current sign with the outline of the "normal state" sign in the database, and uses a multi-layer self-attention mechanism to analyze the differences in shape, geometric features, etc. between the two, and can accurately identify various types of defects, including contamination, deformation, occlusion, etc. This method can handle a variety of subtle defects in signs in complex scenes, and at the same time has strong robustness and environmental adaptability. The present invention adopts a dual input method to compare and classify the current sign outline with the sign outline in the historical normal state. This mechanism can accurately capture the defects of traffic signs, provide more reliable defect detection results, and avoid the errors and limitations caused by the traditional method of relying on a single image input.

[0015] (3) The present invention proposes an innovative method for applying a multi-stage feature extraction strategy to traffic sign defect detection. The present invention combines target detection (YOLO), precise contour extraction (SAM) and deep feature analysis (Transformer), which greatly improves the detection system's ability to identify different types of sign defects. The present invention uses the visual large model SAM to have the ability to process large-scale and diverse data, and can more accurately capture the sign contour and its changes in complex scenes, significantly improve the detection accuracy and environmental adaptability of the model, and solve the problem of insufficient robustness of existing methods in diverse scenes. By performing comparative analysis through a dual input method, the deformation and defects of the sign can be effectively captured. Combined with the deep feature analysis capability of the Transformer neural network, the system can more accurately identify the shape changes and local defects of the sign, thereby significantly improving the detection accuracy of the defect. The multi-stage processing flow ensures that the tasks of each stage can be optimized based on the results of the previous stage, thereby improving the final detection accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flow chart of the traffic sign defect detection method based on SAM and Transformer of the present invention. DETAILED DESCRIPTION

[0017] Embodiment 1: A traffic sign defect detection method based on SAM and Transformer is an efficient, accurate and robust solution for defect detection of road traffic signs. The present invention combines the YOLO target detection model, the visual large model SAM (Segment Anything Model) and the Transformer-based neural network classifier, and realizes efficient detection and defect analysis of traffic signs through multi-stage detection. First, the YOLO model is used to perform preliminary detection of the traffic sign and obtain the detection frame of the sign; then the detection frame and the corresponding image are input into the SAM model to extract the precise sign outline; then, the extracted sign outline is compared with the sign outline in the normal state, and input into the Transformer-based neural network classifier for analysis, and finally the defect detection result is output.

[0018] Preferably, the method specifically comprises the following steps: Step 1: Target detection.

[0019] Step 1.1, image acquisition.

[0020] First, in a highway environment, the vehicle is equipped with high-resolution cameras that capture traffic signs on both sides of the road as the vehicle is driving. The cameras are installed on the front and top of the vehicle to ensure that clear images of traffic signs can be captured during driving. These cameras have autofocus and image stabilization functions to adapt to jitter and lighting changes when the vehicle is driving at high speeds. The collected image and video data are stored in real time on the on-board storage device and uploaded to the cloud server for subsequent processing.

[0021] Step 1.2: Input the image into the target detection model.

[0022] After a series of preprocessing, the collected image data is input into the YOLO (You Only Look Once) target detection model. The YOLO target detection model is used to detect road traffic signs, which can quickly identify traffic signs and accurately frame the area where the signs are located.

[0023] Step 2: contour extraction.

[0024] The logo box and the corresponding image detected by the YOLO target detection model are input into the visual large model SAM, and the precise outline of the logo is obtained through the SAM model. The SAM model can generate detailed logo outlines, further improving the accuracy of detection.

[0025] Step 3: Contour comparison analysis.

[0026] After the SAM model extracts the precise outline of the traffic sign, the system will compare and analyze the currently detected sign outline with the "normal state" sign outline stored in the database. Using a dual-input method, the currently detected sign outline and the previously recorded sign normal state outline are input into a neural network classifier based on Transformer (such as ViT, visual Transformer). The system enters the defect detection and classification stage, and the classifier based on the Transformer neural network further analyzes the differences in the outline, outputs whether there is a defect, whether there is deformation, etc., and gives a specific defect type and confidence score.

[0027] Embodiment 2: A traffic sign defect detection method based on SAM and Transformer, such as Figure 1 As shown, the specific steps include: Step 1: Target detection.

[0028] Vehicles traveling on the road collect image data of road traffic signs. After the images are preprocessed, the system inputs them into the YOLO (You Only Look Once) target detection model, and uses the target detection model to detect road traffic signs. The YOLO target detection model is an efficient model that can complete target detection in a single forward pass. It can globally process the input image and identify the traffic signs in a very short time. Specifically, when the system receives the preprocessed image input, the YOLO target detection model will first divide the image into multiple grids, and each grid predicts whether the sign exists and its category. By considering both the global and local features of the image, the YOLO target detection model can quickly identify traffic signs and generate a detection box (Bounding Box) including location coordinates and category labels, laying the foundation for subsequent contour extraction and defect analysis.

[0029] The YOLO target detection model was selected for preliminary target detection because it can achieve efficient and real-time target detection at a low computational cost, and has strong generalization ability to handle different types of traffic signs in complex road environments.

[0030] Step 2: contour extraction.

[0031] After completing the target detection, the sign detection box detected by the YOLO target detection model and its corresponding image are passed as input to the visual large model SAM (Segment Anything Model). The SAM model is an advanced segmentation model that can automatically segment the precise contours of the target of interest in the image. Specifically, SAM determines the preliminary sign area based on the detection box output by the YOLO target detection model, and uses its multi-layer feature pyramid structure to refine the area layer by layer. Finally, SAM can generate the precise contour of the traffic sign, capture the boundary shape and details of the sign, and further improve the detection accuracy.

[0032] Compared with traditional edge detection algorithms (such as Canny or Sobel), the SAM model can effectively reduce the impact of complex backgrounds, changing lighting and other environmental factors. Even when the outline of a traffic sign is complex or partially blocked, it can still accurately extract the complete outline. This method achieves relatively accurate outline extraction.

[0033] Step 3: Contour comparison analysis.

[0034] After obtaining the precise outline of the traffic sign, the system needs to perform defect analysis on the current state of the sign. To this end, the system compares the outline of the traffic sign extracted by the SAM model with the outline of the sign in a normal state stored in the database. The specific approach is to use the currently detected sign outline and the "standard" sign outline as dual inputs and send them to the Transformer-based neural network classifier (Visual Transformer, ViT). ViT is a deep learning model designed specifically for image alignment and feature extraction, which can perform global analysis of input image information through a multi-layer self-attention mechanism.

[0035] During the comparison process, the ViT model extracts local and global features of the sign outline layer by layer and analyzes the similarities and differences between the two. The model calculates the differences in different regions in parallel through multiple attention heads, so that it can identify whether the traffic sign has defects such as deformation, contamination, and occlusion in extremely subtle feature changes.

[0036] Step 4: Defect detection and classification.

[0037] After completing the contour comparison analysis, the system enters the defect detection and classification stage. The multi-head classifier based on the Transformer neural network will further extract and analyze the comparison results, and finally output whether there is a traffic sign defect and the specific type of defect. The classifier can detect many types of defects, such as: Deformation defect: The shape or structure of the logo has obvious geometric deformation, which may be caused by impact or natural damage; Defects: Dirt, dust, fading, etc. on the surface of the logo affect its readability; Occlusion defect: Traffic signs are partially or completely obscured by other objects such as tree branches and billboards.

[0038] The classifier will not only output the defect type, but also give a corresponding confidence score to indicate the accuracy of the model in detecting the defect. These results will be presented to the platform in a visual form and saved in the platform system log in text form for subsequent analysis and maintenance decisions.

[0039] Compared with the prior art, the present invention utilizes a combination of a large visual model and a Transformer classifier to greatly improve the accuracy and efficiency of traffic sign defect detection.

[0040] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A traffic sign defect detection method based on SAM and Transformer, characterized in that: The following steps are involved: Step S1: First, collect and pre-process the image of the traffic sign, use the YOLO target detection model to identify the traffic sign in the image, and generate a detection frame of the traffic sign to frame the area where the traffic sign is located; Step S2: Then, the detection frame and the corresponding image are input into the SAM model to extract the outline of the traffic sign; Step S3: The outline of the traffic sign extracted in step S2 and the outline of the traffic sign in the normal state in the database are simultaneously input into the Transformer-based neural network classifier for comparative analysis, and finally the defect detection result is output, giving the specific defect type and confidence score.

2. The traffic sign defect detection method based on SAM and Transformer according to claim 1 is characterized in that: In step S1, the YOLO target detection model first divides the image into multiple grids, and predicts whether there is a traffic sign and the category of the traffic sign for each grid, and generates a detection frame including position coordinates and category labels.

3. The traffic sign defect detection method based on SAM and Transformer according to claim 2 is characterized in that: In step S2, the SAM model determines the preliminary identification area based on the detection frame output by the YOLO target detection model, and uses a multi-layer feature pyramid structure to refine the preliminary identification area layer by layer, and finally generates the outline of the traffic sign and captures the boundary shape and details of the traffic sign.

4. The traffic sign defect detection method based on SAM and Transformer according to claim 3 is characterized in that: In step S3, the Transformer-based neural network classifier extracts local and global features of the sign contour layer by layer, and analyzes the similarities and differences between the two; the Transformer-based neural network classifier calculates the differences in different areas in parallel through multiple attention heads to identify the defects of the traffic sign.

5. The traffic sign defect detection method based on SAM and Transformer according to claim 4 is characterized in that: In step S3, after completing the contour comparison analysis, the defect detection and classification stage is entered; the Transformer-based neural network classifier extracts and analyzes the comparison results, and finally outputs whether there is a traffic sign defect, the specific type of the defect and the corresponding confidence score.

6. The traffic sign defect detection method based on SAM and Transformer according to any one of claims 1 to 5, characterized in that: In the step S1, images of road traffic signs are collected based on vehicles traveling on the road.

7. A traffic sign defect detection system based on SAM and Transformer, based on the traffic sign defect detection method based on SAM and Transformer according to any one of claims 1 to 6, characterized in that: It includes an image acquisition and preprocessing module, a target detection module, a contour detection module and a defect detection module; the target detection module is used to perform preliminary detection of traffic signs in the image and generate a detection frame of the traffic sign; the contour detection module is used to extract the contour of the traffic sign based on the SAM model; the defect detection module is used to output the defect detection result through a Transformer-based neural network classifier, and give a specific defect type and confidence score.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the traffic sign defect detection method based on SAM and Transformer described in any one of claims 1 to 6 is implemented.

Images