A method and system for all-weather autonomous road defect detection

Abstract

This invention discloses an all-weather autonomous road defect detection method and system, relating to the field of computer vision. It includes a hybrid vision sensor for all-weather autonomous road defect detection mounted on an unmanned vehicle (UAV). The hybrid vision sensor is equipped with a computing unit, which runs a hybrid vision road defect recognition and localization algorithm. The hybrid vision in the UAV sensor includes traditional RGB vision and DVS pulse vision. The traditional RGB vision is used to detect the texture details of the road, while the DVS pulse vision is used to detect road anomalies at high speeds. By utilizing the rich texture details of RGB and combining the sensitive response of DVS vision to low light and dynamic lighting changes, the accuracy of road defect detection is improved. This invention features real-time asynchronous response, high dynamic range, information capture capability in low-light scenarios, and reduced motion blur.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to an all-weather autonomous road defect detection method and system. Background Technology

[0002] With the rapid development of infrastructure and the increase in vehicles, road conditions gradually deteriorate due to repeated vehicle traffic and changes in environmental conditions, leading to road defects. Therefore, the detection and maintenance of road defects are becoming increasingly important. Currently, there are many solutions for detecting road defects, such as ground-embedded detection methods, which suffer from time-consuming and labor-intensive installation, as well as high maintenance and construction costs; fixed camera monitoring methods, which have limited detection range and are expensive to deploy over large areas; and manual patrol methods, currently the mainstream approach, which require significant manpower and resources and are also costly. None of these are the optimal choices.

[0003] In existing technologies, RGB cameras are typically installed on vehicles for road defect detection. These cameras employ embedded AI chips and deep learning neural network algorithms. Under daytime visibility conditions, visual algorithms perform real-time defect detection and identification on the road surface. When a significant defect is confirmed, the vehicle's GPS signal is triggered to report the defect, which is then identified and confirmed by professionals. However, when faced with dynamic changes in lighting, the response speed is slow due to frame rate constraints, making it impossible to capture the complete trajectory of high-speed moving targets. The detection dynamic range is also low, as it is easily affected by strong changes in lighting, such as when facing backlight. The information capture capability is also poor in low-light scenarios such as at night. Furthermore, it is difficult to capture extremely high-speed moving targets, such as when passing through trees, where the moving image is blurred when the vehicle is traveling at high speed. Overall, the accuracy of road defect detection and perception is low. Summary of the Invention

[0004] To overcome the technical problem of low accuracy in detecting and perceiving road defects in existing technologies, this invention provides a road defect detection method and system based on computer vision. By utilizing the rich texture detail characteristics of RGB and combining the sensitivity characteristics of DVS vision in low light and dynamic light changes, the accuracy of road defect detection and perception is improved.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] An all-weather autonomous road defect detection method includes the following steps:

[0007] S1. Using the frame-based pattern of dynamic vision, spatiotemporal registration and correction are performed with traditional visual images to obtain hybrid vision;

[0008] S2. The hybrid vision encodes and represents discrete asynchronous pulse data to obtain grid data, and then fuses it with data from traditional vision as network input.

[0009] S3. The network input uses a lane line recognition algorithm to determine the type of road defect and the specific location where the road defect occurs. The network outputs labels and coordinates.

[0010] S4. Pre-detect the dynamic visual image, extract the area with obvious contour changes, and calculate the high-frequency information content of the image in this area.

[0011] S5. Compare the high-frequency information content with the set threshold. If the high-frequency information content is greater than the set threshold, proceed to S6; otherwise, return to S4.

[0012] S6. Determine if iOU is greater than 0.5. If yes, confirm the result; otherwise, return to S2.

[0013] Preferably, the traditional vision is traditional RGB vision, and the dynamic vision is DVS pulse vision. The frame-based mode of the DVS pulse vision is used to perform spatiotemporal registration and correction with the image of the traditional RGB vision to obtain the hybrid vision.

[0014] Preferably, in step S2, the registered hybrid vision encodes and represents the discrete asynchronous pulse (DVS) pulse visual data, and ( , , , Pulse trains in the format of ) are frequency-encoded into 3D voxel grid data. , , ), and compared with traditional RGB visual three-dimensional data ( , , ) merge into ( , , ) is used as the network input; wherein, This indicates the location where the pulse event occurred, corresponding to the pixel coordinates in the image. Represents the polarity of the event. Represents the time when the current event occurred; after representation , These represent the image's length and width, respectively. This refers to the number of channels in DVS; the default is Conquer Dual Channel = 2. This refers to the number of RGB channels; the default is RGB color = 3. Data acquired by the DVS camera: ( , , , ), by statistically analyzing a time window The number of internal pulses is used to represent information: , .

[0015] Preferably, in step S3, the lane line recognition algorithm uses voxel grid data fused with RGB visual data after DVS pulse visual representation as the perception input to define common road defect types; after the fusion processing of RGB and DVS images, it is input into the detection and recognition network to determine whether it is a common road defect type and the specific location of the defect type; if so, the GPS signal corresponding to this location is uploaded to complete the reporting process.

[0016] Preferably, in step S4, the input shape is [w,h,c1+c2]; the output is the label with the highest recognition confidence and the corresponding location; at the same time, in order to reduce false alarms, the DVS is used to detect the DVS image by taking into account the sensitivity of the contour edge response, extracting the region with obvious contour changes, and calculating the high-frequency information content V1 of the image in this region.

[0017] Preferably, in step S5, if the high-frequency information content V1 of the image in the region is greater than a set threshold T, it is determined that the region has rich texture details and has the characteristics of road defects. If the V1 content is less than the threshold T, it is determined that the region is relatively smooth and does not have the characteristics of road defects. This determination is combined with the network recognition result as the final secondary confirmation.

[0018] Preferably, in step S6, iOU is calculated and determined to be greater than 0.5. If it is, the result is confirmed; otherwise, the process returns to step S2.

[0019] Preferably, in S3, the road defects include subsidence, cracks, potholes, and bulges.

[0020] An all-weather autonomous road defect detection system includes a hybrid vision sensor for all-weather autonomous road defect detection when mounted on an unmanned vehicle, and the hybrid vision sensor is equipped with a computing unit.

[0021] Preferably, the computing unit is used to run a hybrid vision road defect recognition and localization algorithm; the hybrid vision in the hybrid vision sensor of the unmanned vehicle includes traditional RGB vision and DVS pulse vision, the traditional RGB vision is used to detect the texture details of the road, and the DVS pulse vision is used to detect abnormal road events under high-speed driving conditions.

[0022] Compared with the prior art, the advantages of the present invention are:

[0023] By utilizing the rich texture detail characteristics of RGB and combining it with the DVS visual low-light and dynamic light change response sensitivity characteristics, the detection and perception accuracy of road defects has been improved.

[0024] Real-time asynchronous response, not constrained by frame rate, can capture the complete trajectory of high-speed moving targets, with high sensitivity and a frame rate of 1000 frames.

[0025] It has a high dynamic range, is unaffected by strong changes in light, and adapts to dynamic light with a range of light variation of 10 Lux-10 Klux;

[0026] It has the ability to capture information in low-light scenes. Traditional RGB has unclear texture features when the light intensity is less than 50 Lux, while DVS has the ability to display clear outlines at a minimum of 10 Lux.

[0027] Reduce motion blur: Pulse events are generated following changes in light intensity and have extremely high temporal resolution, thus enabling the capture of extremely high-speed moving targets, thereby reducing motion blur and solving the problem of blurry images when vehicles are moving at high speeds. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a detection method according to an embodiment of the present invention. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0031] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0032] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0033] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0034] See Figure 1 This is an embodiment of a road defect detection method and system according to the present invention. In this embodiment, an all-weather autonomous road defect detection method includes the following steps:

[0035] S1. Using the frame-based pattern of dynamic vision, spatiotemporal registration and correction are performed with traditional visual images to obtain hybrid vision;

[0036] S2. The hybrid vision encodes and represents discrete asynchronous pulse data to obtain grid data, and then fuses it with data from traditional vision as network input.

[0037] S3. The network input uses a lane line recognition algorithm to determine the type of road defect and the specific location where the road defect occurs. The network outputs labels and coordinates.

[0038] S4. Pre-detect the dynamic visual image, extract the area with obvious contour changes, and calculate the high-frequency information content of the image in this area.

[0039] S5. Compare the high-frequency information content with the set threshold. If the high-frequency information content is greater than the set threshold, proceed to S6; otherwise, return to S4.

[0040] S6. Determine if iOU is greater than 0.5. If yes, the result is confirmed; otherwise, return to S2. iOU represents the degree of overlap between the high-frequency information content and the set threshold.

[0041] In this embodiment, the traditional vision is traditional RGB vision, and the dynamic vision is DVS pulse vision. The frame-based mode of the DVS pulse vision is used to perform spatiotemporal registration and correction with the image of the traditional RGB vision to obtain the hybrid vision.

[0042] In this embodiment, in step S2, the registered hybrid vision encodes and represents the discrete asynchronous pulse (DVS) pulse visual data, and ( , , , Pulse trains in the format of ) are frequency-encoded into 3D voxel grid data. , , ), and compared with traditional RGB visual three-dimensional data ( , , ) merge into ( , , ) is used as the network input; wherein, This indicates the location where the pulse event occurred, corresponding to the pixel coordinates in the image. Represents the polarity of the event (outputs a positive polarity of 1 when the light intensity increases, and outputs a negative polarity of -1 when the light intensity decreases). Represents the time when the current event occurred; after representation , These represent the image's length and width, respectively. This refers to the number of channels in DVS; the default is Conquer Dual Channel = 2. This refers to the number of RGB channels; the default is RGB color = 3. Data acquired by the DVS camera: ( , , , ), by statistically analyzing a time window The number of internal pulses is used to represent information: , .

[0043] In this embodiment, in step S3, the lane line recognition algorithm uses voxel grid data after DVS pulse visual representation and RGB visual data as perception input to define common road defect types; after the fusion processing of RGB and DVS images, it is input to the detection and recognition network to determine whether it is a common road defect type and the specific location of the defect type; if so, the GPS signal corresponding to this location is uploaded to complete the reporting process; the detection and recognition network is the classic YOLOv5-s version.

[0044] In this embodiment, in step S4, the input shape is [w,h,c1+c2], which is [640,640,5] here; the output is the label with the highest recognition confidence and the corresponding location; at the same time, in order to reduce false alarms, the DVS is sensitive to the contour edge response, and the DVS image is pre-detected to extract the region with obvious contour changes, and the high frequency information content V1 of the image in this region is calculated.

[0045] In this embodiment, in step S5, if the high-frequency information content V1 of the image in the region is greater than a set threshold T, it is determined that the region has rich texture details and has the characteristics of road defects. If the V1 content is less than the threshold T, it is determined that the region is relatively smooth and does not have the characteristics of road defects. This determination is combined with the network recognition result as the final secondary confirmation.

[0046] In this embodiment, in step S6, it is calculated and determined whether iOU is greater than 0.5. If it is, the result is confirmed; otherwise, the process returns to step S2.

[0047] In this embodiment, the road defects in S3 include subsidence, cracks, potholes, and bulges.

[0048] An all-weather autonomous road defect detection system includes a hybrid vision sensor for all-weather autonomous road defect detection when mounted on an unmanned vehicle, and the hybrid vision sensor is equipped with a computing unit.

[0049] In this embodiment, the computing unit uses an NVIDIA Jetson Xavier NX chip to run a hybrid vision road defect recognition and localization algorithm. The hybrid vision in the hybrid vision sensor of the unmanned vehicle includes traditional RGB vision and DVS pulse vision. The traditional RGB vision is used to detect the texture details of the road, and the DVS pulse vision is used to detect abnormal road events at high speeds.

[0050] In this embodiment, by utilizing the rich texture detail characteristics of RGB and combining it with the DVS visual sensitivity characteristics in low light and dynamic light changes, the accuracy of road defect detection and perception is improved; real-time asynchronous response, not constrained by frame rate, can capture the complete trajectory of high-speed moving targets, with high sensitivity response speed, equivalent to 1000 frames per second; it has a high dynamic range, is not affected by strong light changes, and adapts to dynamic light changes ranging from 10 Lux to 10 Klux; it has the ability to capture information in low light scenes, while traditional RGB has unclear texture features when the light intensity is less than 50 Lux, while DVS has the ability to display clear outlines at a minimum of 10 Lux; it reduces motion blur: pulse events are generated following changes in light intensity, with extremely high temporal resolution, thus capturing extremely high-speed moving targets, thereby reducing motion blur and solving the problem of image blurring when vehicles are moving at high speeds.

[0051] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0052] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for all-weather autonomous road defect detection, characterized in that, Includes the following steps: S1. Using the frame-based pattern of dynamic vision, spatiotemporal registration and correction are performed with traditional visual images to obtain hybrid vision; S2. The hybrid vision encodes and represents discrete asynchronous pulse data to obtain grid data, and then fuses it with data from traditional vision as network input. S3. The network input uses a lane line recognition algorithm to determine the type of road defect and the specific location where the road defect occurs. The network outputs labels and coordinates. S4. Pre-detect the dynamic visual image, extract the area with obvious contour changes, and calculate the high-frequency information content of the image in this area. S5. Compare the high-frequency information content with the set threshold. If the high-frequency information content is greater than the set threshold, proceed to S6; otherwise, return to S4. S6. Determine if iOU is greater than 0.

5. If yes, confirm the result; otherwise, return to S2. In step S2, the registered hybrid vision encodes and represents the discrete asynchronous pulse (DVS) pulse visual data by encoding the pulse train in (x,y,p,t) format into 3D voxel grid data (w,h,t) through frequency encoding. ), and compared with traditional RGB visual three-dimensional data (w, h, ) merge into (w,h, ) as the network input; Where x and y represent the location of the pulse event, p corresponds to the pixel coordinates in the image, t represents the event polarity, and represents the time when the current event occurred; After characterization, w and h represent the image length and width, respectively. This refers to the number of channels in DVS; the default is Conquer Dual Channel = 2. The number of RGB channels, defaulting to RGB color = 3; Data acquired by the DVS camera: (x, y, p, t), calculated by statistically analyzing a time window. The number of internal pulses is used to represent information: , .

2. The all-weather autonomous road defect detection method according to claim 1, characterized in that, In step S1, the traditional vision is traditional RGB vision, and the dynamic vision is DVS pulse vision. The frame-based mode of the DVS pulse vision is used to perform spatiotemporal registration and correction with the image of the traditional RGB vision to obtain the hybrid vision.

3. The all-weather autonomous road defect detection method according to claim 1, characterized in that, In step S3, the lane line recognition algorithm uses voxel grid data after DVS pulse visual representation and RGB visual data as perception input to define common road defect types. After the fusion processing of RGB and DVS images, it is input into the detection and recognition network to determine whether it is a common road defect type and the specific location of the defect type. If so, the GPS signal corresponding to this location is uploaded to complete the reporting process.

4. The all-weather autonomous road defect detection method according to claim 1, characterized in that, In S4, the input shape is [w, h, Output the label with the highest confidence level and the corresponding location; at the same time, in order to reduce false alarms, the DVS is sensitive to the contour edge response, and the DVS image is pre-detected to extract the area with obvious contour changes and calculate the high frequency information content V1 of the image in this area.

5. The all-weather autonomous road defect detection method according to claim 1, characterized in that, In step S5, if the high-frequency information content V1 of the image in the region is greater than the set threshold T, it is determined that the region has rich texture details and has the characteristics of road defects. If the V1 content is less than the threshold T, it is determined that the region is relatively smooth and does not have the characteristics of road defects. This determination is combined with the network recognition result as the final secondary confirmation.

6. The all-weather autonomous road defect detection method according to claim 1, characterized in that, In step S6, calculate and determine whether iOU is greater than 0.

5. If it is, the result is confirmed; otherwise, return to step S2.

7. The all-weather autonomous road defect detection method according to claim 3, characterized in that, In S3, the road defects include subsidence, cracks, potholes, and bulges.

Images