Method for detecting traffic sign damage based on deep learning

Abstract

This invention discloses a deep learning-based method for detecting missing traffic signs, comprising the following steps: S1: preprocessing the traffic sign detection dataset; S2: supervising the preprocessed dataset for training; S3: obtaining the coordinates of a video capture vehicle based on GPS positioning information; S4: identifying the traffic sign and obtaining its relative position to the video capture vehicle using a deep learning neural network; S5: superimposing the coordinates of the video capture vehicle and the relative position of the traffic sign to obtain the absolute position of the traffic sign; S6: constructing a map using visual SLAM; S7: adding the absolute position information of the traffic sign as a landmark to the visual SLAM map; S8: determining whether the traffic sign is missing based on the similarity of comparative feature points combined with a preset threshold. This invention, by promptly detecting and repairing missing traffic signs, can prevent safety facility problems caused by rain, strong winds, and traffic accidents, thereby improving road safety.

Description

Technical Field

[0001] This invention relates to the field of road inspection technology, and in particular to a method for detecting missing traffic signs based on deep learning. Background Technology

[0002] Currently, the main methods for detecting damage to road traffic signs are divided into two types: routine inspection and periodic inspection.

[0003] Routine inspections are primarily conducted manually, involving visual inspection, manual measurement and form filling, and subsequent manual statistical summarization. This method has the following drawbacks:

[0004] (1) The detection effect and accuracy depend heavily on the sense of responsibility and experience of the inspection personnel, and cannot be effectively guaranteed.

[0005] (2) Manual inspection is inefficient, affects road traffic, and poses safety risks.

[0006] (3) It is impossible to conduct a comprehensive road inspection (long cycle and large workload), which can easily lead to missed detection of defects.

[0007] Periodic inspections: Primarily using specialized inspection vehicles equipped with complex instruments, these inspections collect road data and analyze road conditions while maintaining a constant speed. This is currently the most widely adopted mandatory periodic inspection method, but it has the following drawbacks:

[0008] (1) It requires professional inspection vehicles and teams. It cannot meet the needs of maintenance units for freedom and flexibility.

[0009] (2) High cost and long testing cycle, making it unsuitable for routine testing.

[0010] Routine inspections are currently mainly conducted manually within the industry, with common methods including purely manual reporting and a combination of manual reporting and software assistance. Disease detection primarily relies on visual inspection, photographing, and reporting from reports.

[0011] The manual-assisted software system model has gradually developed in recent years. Its main approach is to digitize the photo-taking and data entry process using mobile apps, saving the time spent on manual statistics and data entry later. This model changes the traditional method of reporting using handwritten notes. While software systems are involved, manual identification of damaged traffic signs, photographing, and categorizing are still necessary.

[0012] In summary, both models rely heavily on manual operation, and are particularly constrained by factors such as staffing levels and personnel experience, making it impossible to guarantee the timeliness and accuracy of detecting missing traffic signs. This is especially true given the current situation, where highway infrastructure construction is still in a growth phase. As the road network continues to expand and labor costs rise, the pressure and challenges regarding personnel will become increasingly urgent. Summary of the Invention

[0013] To address the shortcomings of the existing technologies, this invention provides a deep learning-based method for detecting missing traffic signs, the specific technical solution of which is as follows:

[0014] A deep learning-based method for detecting missing traffic signs is characterized by the following steps:

[0015] S1: Preprocess the traffic sign detection dataset;

[0016] S2: Perform supervised training on the preprocessed dataset;

[0017] S3: Obtain the coordinates of the video capture vehicle based on the positioning information obtained from GPS;

[0018] S4: Identify traffic signs and obtain their relative positions to the video capture vehicle using a deep learning neural network;

[0019] S5: Overlay the coordinates of the video capture vehicle and the traffic sign with the relative position of the video capture vehicle to obtain the absolute position of the traffic sign;

[0020] S6: Building maps using visual SLAM;

[0021] S7: Add the absolute position information of traffic signs as road signs to the visual SLAM map;

[0022] S8: Based on the similarity of the comparative feature points and the preset threshold, determine whether the traffic sign is damaged.

[0023] Specifically, the traffic sign detection dataset mentioned in step S1 includes the CCTSDB dataset and the tt100k_2021 dataset;

[0024] The supervised training described in step S2 uses a pyramid structure for encoding and decoding.

[0025] Specifically, step S4, which involves recognizing traffic signs, also includes using convolutional neural network layers of different scales to extract image features at the corresponding scales. After performing five convolutions in sequence, five deconvolutions are then performed in sequence to obtain a small generative adversarial network, thereby enhancing the recognition accuracy.

[0026] Specifically, the deep learning neural network described in step S4 further includes a step of training the ability to estimate the depth of the environment, which includes the following sub-steps:

[0027] The video image frame at time t is input into the depth estimation network to obtain the depth image;

[0028] The two images at times t-1 and t+1 are input into the pose estimation network to estimate the position and pose of the camera when the image frame at time t is captured.

[0029] Reconstruct the image at time t using depth and pose information;

[0030] The real image and the reconstructed image at time t are sent to the discriminator for discrimination. If the discriminator can distinguish that the reconstructed image is fake, the training continues. When the discriminator cannot distinguish between the real and fake reconstructed images, the training ends.

[0031] Specifically, the visual SLAM described in step S6 includes pose tracking, local map construction, and loop closure detection.

[0032] Specifically, the pose tracking locates the camera in each frame and determines when to insert a new keyframe, including the following sub-steps:

[0033] Perform feature matching between the current frame and the previous frame, and use camera movement BA to optimize the pose;

[0034] If tracking is lost, perform a global relocation;

[0035] When feature matching and initial estimation of camera pose are both satisfied, local retrieval is performed using the co-view of keyframes maintained by the system.

[0036] The camera pose is optimized by reprojecting the current frame to search for map points that match the local map and using all matching points.

[0037] Decide whether to insert a keyframe.

[0038] Specifically, the local map construction process processes new keyframes and performs local BA to optimize the reconstruction of the surrounding camera poses, including the following sub-steps:

[0039] Search for matching points of ORB features in the new keyframe in the connected keyframes of the common view to perform triangulation on the new points;

[0040] Remove redundant keyframes.

[0041] Specifically, the loop closure detection uses the Levenberg-Marquardt algorithm to perform loop closure search on each new keyframe, achieving global consistency in pose graph optimization.

[0042] Specifically, the preset threshold is 70%.

[0043] The beneficial effects of this invention are:

[0044] This invention aims to enable highways to promptly identify missing traffic signs and perform timely minor repairs, thus preventing safety facility problems caused by rain, strong winds, and traffic accidents. This reduces traffic accidents, improves road quality, extends the overall lifespan of the road surface, and lowers overall maintenance costs. This patent allows for the rapid and timely identification of traffic signs, reducing repetitive work by maintenance departments and minimizing waste of manpower, resources, and funds.

[0045] This invention facilitates intelligent detection technology for traffic safety facilities, which is of crucial significance in improving road service levels. Traffic signs, as an indispensable part of road traffic, aim to optimize traffic flow, ensuring better road conditions and providing safety for drivers. Safety facilities not only improve traffic capacity but also enhance the aesthetics of safe operation. Furthermore, prioritizing traffic sign detection can improve the overall safety of the transportation system. Attached Figure Description

[0046] Figure 1 The diagram shown illustrates the traffic sign recognition process of this invention.

[0047] Figure 2 The diagram shown is a schematic representation of the traffic sign positioning process of the present invention.

[0048] Figure 3 The diagram shown is a visual SLAM schematic of the present invention;

[0049] Figure 4 The diagram shown is a technical roadmap of the present invention. Detailed Implementation

[0050] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0051] like Figure 4 As shown, this invention proposes a traffic sign damage detection method based on deep learning, characterized by the following steps:

[0052] S1: Preprocess the traffic sign detection dataset;

[0053] S2: Perform supervised training on the preprocessed dataset;

[0054] S3: Obtain the coordinates of the video capture vehicle based on the positioning information obtained from GPS;

[0055] S4: Identify traffic signs and obtain their relative positions to the video capture vehicle using a deep learning neural network;

[0056] S5: Overlay the coordinates of the video capture vehicle and the traffic sign with the relative position of the video capture vehicle to obtain the absolute position of the traffic sign;

[0057] S6: Building maps using visual SLAM;

[0058] S7: Add the absolute position information of traffic signs as road signs to the visual SLAM map;

[0059] S8: Based on the similarity of the comparative feature points and the preset threshold, determine whether the traffic sign is damaged.

[0060] Specifically, the traffic sign detection dataset mentioned in step S1 includes the CCTSDB dataset and the tt100k_2021 dataset;

[0061] The supervised training described in step S2 uses a pyramid structure for encoding and decoding.

[0062] Specifically, step S4, which involves recognizing traffic signs, also includes using convolutional neural network layers of different scales to extract image features at corresponding scales, such as... Figure 1 As shown, conv4 to conv7 are convolutional neural network layers of different scales used to extract image features at their respective scales, while dconv4 to dconv7 are deconvolutional neural network layers of different scales used for feature image decoding. The 3x3 boxes in the figure represent the convolutional kernels. The convolution stride of conv4 and conv5 in this network is 2, and the feature image size of conv8 generated after conv7 and conv6 is 4x4. After 5 convolutions, the system performs 5 deconvolutions in sequence. The deconvolutional structure helps to obtain functionality similar to a small GAN ​​(Generative Adversarial Network), thereby enhancing its accuracy in recognizing small-sized traffic signs in images, especially traffic signs photographed from a distance.

[0063] Specifically, such as Figure 2 As shown, the deep learning neural network described in step S4 further includes a step of training the ability to estimate the depth of the environment, which includes the following sub-steps:

[0064] The video image frame at time t is input into the depth estimation network to obtain the depth image;

[0065] The two images at times t-1 and t+1 are input into the pose estimation network to estimate the position and pose of the camera when the image frame at time t is captured.

[0066] Reconstruct the image at time t using depth and pose information;

[0067] The real image and the reconstructed image at time t are sent to the discriminator for discrimination. If the discriminator can distinguish that the reconstructed image is fake, the training continues. When the discriminator cannot distinguish between the real and fake reconstructed images, the training ends.

[0068] Specifically, such as Figure 3 As shown, the visual SLAM described in step S6 includes pose tracking, local map construction, and loop closure detection.

[0069] Specifically, the pose tracking locates the camera in each frame and determines when to insert a new keyframe, including the following sub-steps:

[0070] Perform feature matching between the current frame and the previous frame, and use camera movement BA to optimize the pose;

[0071] If tracking is lost, perform a global relocation;

[0072] When feature matching and initial estimation of camera pose are both satisfied, local retrieval is performed using the co-view of keyframes maintained by the system.

[0073] The camera pose is optimized by reprojecting the current frame to search for map points that match the local map and using all matching points.

[0074] Decide whether to insert a keyframe.

[0075] Specifically, the local map construction process processes new keyframes and performs local BA to optimize the reconstruction of the surrounding camera poses, including the following sub-steps:

[0076] Search for matching points of ORB features in the new keyframe in the connected keyframes of the common view to perform triangulation on the new points;

[0077] Remove redundant keyframes.

[0078] Specifically, the loop closure detection uses the Levenberg-Marquardt algorithm to perform loop closure search on each new keyframe, achieving global consistency in pose graph optimization.

[0079] Specifically, the preset threshold is 70%.

[0080] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A traffic sign damage detection method based on deep learning, characterized in that, Includes the following steps: S1: Preprocess the traffic sign detection dataset; S2: Perform supervised training on the preprocessed dataset; S3: Obtain the coordinates of the video capture vehicle based on the positioning information obtained from GPS; S4: Identify traffic signs and obtain their relative positions to the video capture vehicle using a deep learning neural network; S5: Overlay the coordinates of the video capture vehicle and the traffic sign with the relative position of the video capture vehicle to obtain the absolute position of the traffic sign; S6: Building maps using visual SLAM; S7: Add the absolute position information of traffic signs as road signs to the visual SLAM map; S8: Based on the similarity of feature points between the identified traffic signs and the road signs in visual SLAM, combined with a preset threshold, determine whether the traffic signs are damaged.

2. The traffic sign damage detection method based on deep learning according to claim 1, characterized in that, The traffic sign detection dataset mentioned in step S1 includes the CCTSDB dataset and the tt100k_2021 dataset; The supervised training described in step S2 uses a pyramid structure for encoding and decoding.

3. The traffic sign damage detection method based on deep learning according to claim 1, characterized in that, The traffic sign recognition step S4 also includes using convolutional neural network layers of different scales to extract image features at the corresponding scales. After performing five convolutions in sequence, five deconvolutions are then performed in sequence to obtain a small generative adversarial network to enhance recognition accuracy.

4. The traffic sign damage detection method based on deep learning according to claim 3, characterized in that, The deep learning neural network described in step S4 further includes a step of training the ability to estimate the depth of the environment, which includes the following sub-steps: The video image frame at time t is input into the depth estimation network to obtain the depth image; The two images at times t-1 and t+1 are input into the pose estimation network to estimate the position and pose of the camera when the image frame at time t is captured. Reconstruct the image at time t using depth and pose information; The real image and the reconstructed image at time t are sent to the discriminator for discrimination. If the discriminator can distinguish that the reconstructed image is fake, the training continues. When the discriminator cannot distinguish between the real and fake reconstructed images, the training ends.

5. The traffic sign damage detection method based on deep learning according to claim 1, characterized in that, The visual SLAM described in step S6 includes pose tracking, local map construction, and loop closure detection.

6. The traffic sign damage detection method based on deep learning according to claim 5, characterized in that, The pose tracking locates the camera in each frame and determines when to insert a new keyframe, including the following sub-steps: Perform feature matching between the current frame and the previous frame, and use camera movement BA to optimize the pose; If tracking is lost, a global relocation is performed; When feature matching and initial estimation of camera pose are both satisfied, local retrieval is performed using the co-view of keyframes maintained by the system. The camera pose is optimized by reprojecting the current frame to search for map points that match the local map and using all matching points. Decide whether to insert a keyframe.

7. The traffic sign damage detection method based on deep learning according to claim 5, characterized in that, The local map construction process handles new keyframes and performs local BA to optimize the reconstruction of the surrounding camera poses, including the following sub-steps: Search for matching points of ORB features in new keyframes within the already connected keyframes of the common view to perform triangulation on the new points; discard redundant keyframes.

8. The traffic sign damage detection method based on deep learning according to claim 5, characterized in that, The loop closure detection uses the Levenberg-Marquardt algorithm to perform loop closure search on each new keyframe, achieving global consistency in pose graph optimization.

9. The traffic sign damage detection method based on deep learning according to claim 1, characterized in that, The preset threshold is 70%.

Images