Parallax-based pavement defect detection system, method and program product

Abstract

The invention discloses a pavement defect detection system and method based on parallax and a program product, and the system comprises at least one intelligent lamp post, a binocular camera is installed on the intelligent lamp post, the binocular camera is used for obtaining and transmitting pavement images, and the pavement images comprise a pavement left image and a pavement parallax map; the domain processor is configured to traverse parallax values in the received pavement disparity map column by column, determine a disparity sudden change area in the pavement disparity map, and send coordinates of the disparity sudden change area and a pavement left image; and the cloud server is configured to perform pavement defect detection based on the received coordinates of the parallax sudden change area and the pavement left image. According to the technical scheme, the pavement disparity map and the pavement left image are combined, the disparity sudden change area is found in the pavement disparity map, then positioning is carried out in the pavement left image, the accuracy of pavement defect detection is improved, and the possibility of false detection is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of smart cities, and in particular, to a parallax-based road surface defect detection system, a parallax-based road surface defect detection method, and a computer program product. Background Art

[0002] As an emerging urban management method, smart city can be widely used in urban planning and construction, public facilities maintenance and other aspects. In the process of realizing smart city, real-time perception is the foundation of building smart city. Therefore, smart light poles come into being. Through the sensing devices on smart light poles, such as binocular cameras, various types of information in the city can be obtained in real time.

[0003] In the maintenance of urban public facilities, road maintenance has always been an important task. Therefore, in smart cities, it is necessary to detect road conditions, such as damage, cracks, potholes, etc., especially for roads with various manhole covers. As vehicles continue to roll over, if they are not maintained in time, the manhole covers will sag and even affect the travel safety of citizens.

[0004] In the prior art, such problem roads are usually detected by image detection, such as Chinese patent CN117649633A. Such methods usually require that the road image captured must have obvious texture features, and then go through feature enhancement, model recognition and other processes to achieve road defect detection. However, for road problems with unclear texture features, such as depressions formed by long-term vehicle rolling and depressions at manhole covers, Figure 1 As shown, this type of detection based only on images is not very effective.

[0005] In addition, some repaired road cracks will leave obvious marks on the road surface, such as Figure 2 As shown in the figure, such traces may also cause false detection of road defects based on images. Summary of the invention

[0006] The purpose of this application is to improve the accuracy of road surface defect detection.

[0007] The technical solution of the first aspect of the present application is: a parallax-based road surface defect detection system is provided, the system comprising: at least one smart lamp pole, a binocular camera is installed on the smart lamp pole, the binocular camera is used to acquire and send road surface images, wherein the road surface images include a left road surface image and a road surface disparity map; at least one domain processor, the domain processor is configured to traverse the disparity values ​​in the received road surface disparity map column by column, determine the parallax sudden change area in the road surface disparity map, and send the coordinates of the parallax sudden change area and the left road surface image; a cloud server, the cloud server is configured to perform road surface defect detection based on the coordinates of the received parallax sudden change area and the left road surface image.

[0008] In any of the above technical solutions, further, the disparity values ​​in the received road disparity map are traversed column by column to determine the disparity sudden change area in the road disparity map, specifically including: traversing the disparity values ​​in the road disparity map column by column from left to right and from bottom to top; when it is determined that the disparity value changes from a positive value to 0, the current position coordinates in the road disparity map are marked as the starting point, and the coordinates of the adjacent disparity points with a disparity value of 0 are determined in an outward growth manner, recorded as regional coordinates; the minimum circumscribed rectangle of the area where the starting point and the regional coordinates are located is determined as the disparity sudden change area.

[0009] In any of the above technical solutions, further, the disparity values ​​in the received road disparity map are traversed column by column to determine the disparity sudden change area in the road disparity map, and also include: in the process of outward growth, when it is determined that the number of disparity points with a disparity value of 0 determined along any column is greater than a number threshold, the outward growth is stopped, and the area where the corresponding disparity points are located is determined as a disparity hole area; the disparity values ​​in the road disparity map are traversed again, and the disparity hole area is skipped.

[0010] In any of the above technical solutions, further, the disparity values ​​in the received road disparity map are traversed column by column to determine the disparity sudden change area in the road disparity map, specifically including: traversing the disparity values ​​in the road disparity map column by column from left to right and from bottom to top; when it is determined that the difference between the disparity values ​​within a preset column width is greater than or equal to the sudden change threshold, the current position coordinates in the road disparity map are marked as the starting point, and the coordinates of the disparity points in the adjacent area of ​​the starting point with the same disparity value and / or uniformly increasing disparity value are determined, which are recorded as the area coordinates; the minimum circumscribed rectangle of the area where the starting point and the area coordinates are located is determined as the disparity sudden change area.

[0011] In any of the above technical solutions, further, the disparity values ​​in the received road disparity map are traversed column by column to determine the disparity sudden change area in the road disparity map, and also include: when it is determined that the difference between the row coordinates of the starting point and the corresponding disparity point within the regional coordinate range is greater than or equal to the coordinate threshold, the minimum circumscribed rectangle of the area where the starting point and the regional coordinates are located is determined as the disparity sudden change area.

[0012] In any of the above technical solutions, further, when the number of smart street lamps is greater than or equal to two, the domain processor is also configured to: calculate the overlapping area in two adjacent road images based on the world coordinate system, the position coordinates of the smart lamp pole and the internal parameters of the binocular camera; perform image stitching on two adjacent road images containing the overlapping area to generate a road stitching image; traverse the road stitching image to determine the parallax sudden change area.

[0013] In any of the above technical solutions, further, the cloud server is also configured to: store the received left image of the road surface in time sequence; when it is determined that there is a defect in the road surface, determine the defect type and the image area of ​​the defect in the left image of the road surface; according to a preset time span, retrieve the stored left image of the road surface, and identify and count the number and type of vehicles passing through the image area; based on the defect type, the number of vehicles and the type of vehicles, construct a vehicle-road surface defect prediction model.

[0014] In any of the above technical solutions, further, a computing unit is provided in the binocular camera, and the computing unit is configured to calculate a road disparity map based on a left road image and a right road image, wherein the process of calculating the road disparity map specifically includes: obtaining corresponding left road image features and right road image features based on the left road image and the right road image; taking the left road image features as a reference, calculating the correlation between any row of pixel features in the left road image features and the corresponding row of pixel features in the right road image features, and generating a correlation matrix; extracting the last dimension in the correlation matrix, and performing a normalization operation to generate a matching probability distribution array; based on any pixel coordinate value in the left road image, the corresponding row coordinate in the right road image and the corresponding matching probability distribution array, calculating the disparity value of the corresponding pixel point, and obtaining the road disparity map.

[0015] The technical solution of the second aspect of the present application is: a parallax-based road surface defect detection method is provided, the method comprising: step 1: obtaining a road surface image, wherein the road surface image at least includes a road surface left image and a road surface disparity map; step 2: traversing the disparity values ​​in the road surface disparity map column by column to determine the parallax sudden change area in the road surface disparity map; step 3: performing road surface defect detection based on the parallax sudden change area and the road surface left image.

[0016] The technical solution of the third aspect of the present application is: providing a computer program product, including a computer program, which, when executed by a processor, implements the parallax-based road surface defect detection method as described in the technical solution of the second aspect of the present application.

[0017] The beneficial effects of this application are:

[0018] The technical solution in the present application combines the road surface disparity map with the left road surface image, finds the area of ​​sudden disparity change in the road surface disparity map, and then uses the correspondence between the disparity map and the left image to locate it in the left road surface image, uses the located image area as the input of image detection, and uses conventional image detection methods to perform final defect identification, which can improve the accuracy of road surface defect detection and reduce the possibility of false detection, while also reducing the computational complexity of image recognition and hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0020] Figure 1 It is a schematic diagram of the existing road surface and manhole cover depression;

[0021] Figure 2 It is a schematic diagram of the existing road surface repair traces;

[0022] Figure 3 is a schematic block diagram of a parallax-based road surface defect detection system according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a single-frame disparity map according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a sudden change process of the parallax value of a manhole cover depression area according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of epipolar matching according to an embodiment of the present application;

[0026] Figure 7 It is a schematic flow chart of a parallax-based road surface defect detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0028] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0029] Embodiment 1:

[0030] like Figure 3 As shown, this embodiment provides a parallax-based road surface defect detection system 100, which includes:

[0031] At least one smart lamp pole 101, on which a binocular camera is installed, and the binocular camera is used to acquire and send a road surface image, wherein the road surface image includes a road surface left image and a road surface disparity map;

[0032] At least one domain processor 102, the domain processor 102 is configured to traverse the disparity values ​​in the received road disparity map column by column, determine the disparity sudden change area in the road disparity map, and send the disparity sudden change area and the road left image, wherein the disparity sudden change area includes the upper left corner coordinates, the area length, the area width and the corresponding disparity value data;

[0033] The cloud server 103 is configured to perform road surface defect detection based on the received parallax sudden change area and the road surface left image.

[0034] Specifically, a smart city system 100 can be established, and a binocular camera can be installed on a lamp pole to form a smart lamp pole 101. After the binocular camera is calibrated, the image within the field of view can be obtained, including the (road surface) left image and the (road surface) disparity map. Then, the acquired image is sent to the cloud server 103 via the domain processor 102. Finally, the cloud server 103 constructs the smart city and performs related data analysis to realize various functions. Among them, one domain processor 102 can transmit data with multiple smart lamp poles 101, and one cloud server 103 can transmit data with multiple domain processors 102.

[0035] Those skilled in the art will appreciate that the coordinates between the road surface left image and the road surface parallax image are in a one-to-one correspondence. Figure 4 As shown in the figure, different targets in the disparity map are not adjacent in space, but for a single target, its outer contour is continuous. Therefore, based on this characteristic of the target in the disparity map, a segmentation algorithm (such as a clustering algorithm and a connected domain algorithm) can be used to segment different targets in the disparity map, thereby obtaining the regions of different targets in the disparity map, and then determining the position of the target in the left image, such as the coordinates, length, width and other information of the target frame.

[0036] Therefore, after the binocular camera in the smart lamp pole 101 sends the collected left image of the road surface and the calculated road surface disparity map to the domain processor 102, the domain processor 102 can perform target segmentation based on the road surface disparity map, and the specific process will not be repeated here.

[0037] It should be noted that the domain processor 102 is usually set to receive road images sent by multiple smart lamp poles 101 within a certain area, and the position of each smart lamp pole 101 and the internal parameters of the binocular camera (baseline distance B, focal length f, optical center coordinates, etc.) are known. Therefore, the domain processor 102 can stitch the multiple road images received at the same time by means of coordinate system transformation. Among them, for the road disparity map, the size and position of the same target are consistent, so the disparity data of the same target can be fused based on this information.

[0038] In this embodiment, since the binocular camera is fixedly mounted on the smart lamp pole 101, and the smart lamp pole 101 is fixedly mounted on both sides of the road, the shooting area of ​​any binocular camera is a fixed area. However, those skilled in the art can understand that the real-time requirements for road defect detection are not high, and the binocular camera can upload the collected images in real time. Therefore, the domain processor 102 can remove the moving targets (such as vehicles and pedestrians) in the road disparity map during the image stitching process, and only retain the disparity information of the road itself, and then fuse the road disparity maps obtained over a period of time to improve the density of the road disparity map.

[0039] For the road disparity map, it represents the distance between the road surface and the binocular camera (optical center). Therefore, this distance should change evenly. Therefore, when there is a sudden change in disparity in the road disparity map, it indicates that there are defects in the road surface. When the disparity value suddenly increases, it indicates that there are depressions and potholes on the road surface; when the disparity value suddenly decreases, it indicates that there are bulges and wrinkles on the road surface.

[0040] It should be noted that, considering that the binocular camera installed on the smart lamp pole 101 needs to shoot a short field of view and a small range, under the condition of a certain resolution, a smaller field of view angle (such as 20°-40°) and a larger baseline length (such as 40cm-80cm) can be set to ensure the accuracy of the binocular camera.

[0041] Therefore, when the domain processor 102 determines that there is an abnormal change in the disparity value of a certain area by traversing the disparity data in the road surface disparity map, the area can be marked as a disparity sudden change area, and the disparity sudden change area in the road surface disparity map can correspond to the left road surface image. Therefore, the cloud server 103 can determine its corresponding position in the left road surface image based on the received disparity sudden change area, and then perform image detection on the image at that location, combine the disparity data with the image detection results, and realize road surface defect detection.

[0042] For example, when the disparity value at a certain point in the road disparity map becomes larger and is determined to be a disparity sudden change area, if it is detected in the left road map that the area contains a manhole cover, it is considered that the manhole cover is sunken; similarly, if the disparity value of the area suddenly becomes 0, it means that the effective disparity value cannot be calculated in the area due to light reasons, then the left road map is used to detect whether the area contains a gap.

[0043] For another example, if the cloud finds potential gaps in an image of a certain area during image detection, but if the disparity value in the corresponding road disparity map is a non-sudden change area, then the potential gaps in the area can be considered to be traces generated during road repair.

[0044] In summary, this embodiment combines a binocular camera with the smart lamp pole 101, uses the road disparity map output by the binocular camera as the basis for road defect detection, and then combines it with conventional image detection methods, which helps to improve the accuracy of road defect detection and avoid false detection of road defects.

[0045] In any of the above embodiments, further, for a road surface defect of the gap type, due to the factor of illumination, there is no effective parallax at the gap, that is, the parallax value is 0, and the gap should have a certain length in the horizontal direction. Therefore, when traversing the road disparity map, in order to improve the efficiency and accuracy of the disparity traversal, the disparity values ​​in the received road disparity map are traversed column by column to determine the disparity sudden change area in the road disparity map, specifically including:

[0046] Traverse the disparity values ​​in the road disparity map column by column from left to right and from bottom to top;

[0047] When it is determined that the disparity value changes from a positive value to 0, the coordinates of the current position in the road disparity map are marked as the starting point, and the coordinates of the adjacent disparity points with a disparity value of 0 are determined in an outward growth manner and recorded as the regional coordinates;

[0048] The minimum bounding rectangle of the area where the starting point and the area coordinates are located is determined as the parallax sudden change area.

[0049] Specifically, since the distance of the road surface in the disparity map is continuous, this embodiment takes the lower left corner of the road surface disparity map as the starting position, traverses the disparity values ​​column by column, and when a disparity point with a disparity value of 0 is traversed, it is used as the starting point, and the coordinates of the disparity points with adjacent disparity values ​​of 0 are determined by growing outward. The area where these disparity points with a disparity value of 0 are located is the potential area where the road surface gaps are located.

[0050] At the same time, considering the irregularity of the gap, the target detection in the image is usually performed in the form of a target frame. Therefore, after determining the above potential gap area, its minimum circumscribed rectangle is determined, and this area is used as the parallax sudden change area.

[0051] It should be noted that, after the parallax sudden change area is determined, when the parallax of other columns in the road disparity map is traversed again, the position of the column located in the parallax sudden change area is skipped to improve the traversal efficiency.

[0052] In any of the above embodiments, further, although the density of the disparity map can be improved by performing disparity fusion on multiple frames of road disparity maps, it is inevitable that there will still be disparity holes in some areas (such as disparity holes caused by shadows). Therefore, when determining the area with sudden disparity changes, these areas need to be filtered out. At the same time, considering that the gap area with a disparity value of 0 is due to the lack of effective illumination, that is, the width of the gap is limited, so if the width of the area with a disparity value of 0 along the longitudinal direction is large, it can be determined that the area is not caused by the gap. Therefore, in this embodiment, the disparity values ​​in the received road disparity map are traversed column by column to determine the area with sudden disparity changes in the road disparity map, and it also includes:

[0053] In the process of outward growth, when it is determined that the number of disparity points with disparity values ​​of 0 determined along any column is greater than the number threshold, the outward growth is stopped, and the area where the corresponding disparity point is located is determined as a disparity hole area;

[0054] Re-traverse the disparity values ​​in the road disparity map and skip the disparity hole area.

[0055] In any of the above embodiments, further, traversing the disparity values ​​in the received road disparity map column by column to determine the disparity sudden change area in the road disparity map specifically includes:

[0056] Traverse the disparity values ​​in the road disparity map column by column from left to right and from bottom to top;

[0057] When it is determined that the difference between the disparity values ​​within the preset column width range is greater than or equal to the sudden change threshold, the coordinates of the current position in the road disparity map are marked as the starting point, and the coordinates of the disparity points in the adjacent area with the same disparity value and / or uniformly increasing disparity values ​​are determined and recorded as the area coordinates;

[0058] The minimum bounding rectangle of the area where the starting point and the area coordinates are located is determined as the parallax sudden change area.

[0059] Specifically, Figure 5 As shown in the figure, a theoretical analysis is carried out by taking the manhole cover depression as an example. In the road disparity map, the disparity value between point A and point O increases uniformly; due to the depression at point O, there is no effective illumination at the horizontal direction OB, so the disparity value at point B suddenly increases, that is, a sudden change occurs; and the disparity between point B and point C increases uniformly again; but between point C and point D, the disparity value decreases uniformly; after passing point D, between point D and point E, the disparity decreases uniformly again.

[0060] Therefore, based on the above-mentioned law of parallax change, this embodiment sets a column width range (such as the width of 3-10 parallax points) during column scanning, takes the current parallax point as the starting point, obtains the parallax values ​​within the column width range upward, and calculates the difference between these parallax values. When the calculated difference is greater than or equal to the sudden change threshold, it is considered that a sudden parallax change has occurred.

[0061] This disparity point is used as the starting point (point B) of the disparity sudden change area, and the coordinates of the disparity points with the same disparity value and / or uniformly increasing disparity value adjacent to it are determined by outward growth, and recorded as regional coordinates. Then, the minimum circumscribed rectangle of the area where these regional coordinates are located is determined as the disparity sudden change area.

[0062] In any of the above embodiments, further, considering that road surface defects usually have a certain length along the road surface width direction, traversing the disparity values ​​in the received road surface disparity map column by column to determine the disparity sudden change area in the road surface disparity map also includes:

[0063] When the difference between the starting point and the row coordinates of the corresponding disparity point within the regional coordinate range is greater than or equal to the coordinate threshold,

[0064] The minimum circumscribed rectangle of the area where the starting point and the area coordinates are located is determined as the disparity sudden change area. That is, when the width along the horizontal direction of the disparity map is greater than or equal to a certain value, this area is considered to be a potential road defect area.

[0065] In any of the above embodiments, further, when the number of smart street lamps is greater than or equal to two, the fields of view of the binocular cameras of two adjacent smart street lamps will have overlapping areas. In order to avoid multiple detections of the same target, the domain processor 102 also splices the multiple road images received at the same time through coordinate system transformation, and fuses the parallax data of the same target. At this time, the domain processor 102 is also configured as follows:

[0066] Based on the world coordinate system, the position coordinates of the smart lamp pole 101 and the internal parameters of the binocular camera, the overlapping area between two adjacent road images is calculated;

[0067] Perform image stitching on two adjacent road surface images containing overlapping areas to generate a road surface stitching image;

[0068] Traverse the road mosaic image and determine the area with sudden parallax change.

[0069] Specifically, since the internal parameters, installation location, location and height of the binocular camera are known, combined with the corresponding road disparity map, for any pixel point in the left image of the road surface, its position coordinate (xw ,y w ,z w ), the corresponding calculation formula is:

[0070]

[0071] Where (u0, v0) is the optical center coordinate of the binocular camera, disp is the disparity value, B is the baseline distance, and f is the focal length.

[0072] Then set an origin of the world coordinate system to construct the world coordinate system. Based on the position coordinates of any smart lamp pole 101 in the world coordinate system, the position coordinates of the above pixel points can be uniformly transformed to the world coordinate system, and then by comparing whether the transformed coordinates are the same, the overlapping area in the two adjacent road images can be obtained.

[0073] Considering that "the accuracy of near parallax is higher than that of far parallax," for two adjacent road images (road left image, road parallax map), the overlapping area below the image is retained, the overlapping area above the image is filtered out, and smoothing is performed to achieve image stitching of two adjacent road images containing overlapping areas, generate a road stitching image, and then determine the parallax sudden change area by traversal.

[0074] It should be noted that different weight values ​​may be set to weight the overlapping area and perform smoothing to achieve image splicing of two adjacent road surface images containing the overlapping area.

[0075] In any of the above embodiments, further, relying on the data processing capability of the cloud server 103, a road surface defect prediction model with prediction function can be constructed by using big data processing. The cloud server 103 is further configured as follows:

[0076] The received left road surface image is stored in time sequence;

[0077] When it is determined that there is a defect in the road surface, the defect type and the image area of ​​the defect in the left image of the road surface are determined;

[0078] According to the preset time span, the stored left image of the road surface is retrieved to identify and count the number and type of vehicles passing through the image area;

[0079] A vehicle-pavement defect prediction model is constructed based on defect type, vehicle quantity and vehicle type.

[0080] In any of the above embodiments, further, the image captured by the binocular camera includes the road left image I l And the road right image I rThe binocular camera is provided with a computing unit. In order to improve the accuracy of disparity calculation, the computing unit is configured to calculate a road disparity map based on a left road image and a right road image. The process of calculating the road disparity map specifically includes:

[0081] Based on the left road image and the right road image, corresponding left road image features and right road image features are obtained; in this process, a feature extraction network (such as ResNet, ViT, etc.) can be used to extract features to extract the left road image feature F l , road right image feature F r , where F l 、F r ∈R H×W×C Any feature in the left road image features corresponds to a pixel point at a corresponding position in the left road image, and any feature in the right road image features also corresponds to a pixel point at a corresponding position in the right road image.

[0082] Taking the left road image features as the reference, the correlation between any row of pixel features in the left road image features and the corresponding row of pixel features in the right road image features is calculated to generate a correlation matrix. In this process, because the epipolar line of the binocular camera is a horizontal line, such as Figure 6 As shown, taking the horizontal direction as the reference, the road surface left image feature F is calculated l The pixel features of the i-th row and the road right image feature F r Features of pixels in the same row The correlation of The corresponding calculation formula is:

[0083]

[0084] In the formula, C i is the correlation of the i-th row in the correlation matrix. Therefore, the correlation matrix of the left image features of the entire road surface is Wherein, W1 is the width of the left road surface image, W2 is the width of the right road surface image, and W1=W2.

[0085] Extract the last dimension in the correlation matrix and perform normalization to generate the matching probability distribution array M. The corresponding calculation formula is:

[0086]

[0087] Where softmax() is a normalization operation, and M(i,j) is the matching probability between the feature at (i,j) in the left road image feature and the W2-1 features in the i-th row in the right road image feature.

[0088] Based on the coordinate value of any pixel in the left road image, the corresponding row coordinate in the right road image and the corresponding matching probability distribution array, the disparity value of the corresponding pixel point is calculated to obtain the road disparity map.

[0089] In this process, we use the difference in coordinates as the disparity value of the corresponding pixel point, where the coordinates for calculating the disparity value include the pixel coordinate value and the matching coordinate value. In the process of calculating the matching coordinate value, the corresponding calculation formula is:

[0090]

[0091] In the formula, is the matching coordinate value of the (i, j)th position in the left image of the road surface, that is, the matching coordinate value corresponding to the pixel point at (i, j) in the left image of the road surface is the horizontal coordinate P of the i-th row in the right image feature of the road surface r (i) The sum of the products of the corresponding matching probabilities M(i,j).

[0092] Therefore, the matching coordinate values ​​corresponding to the entire left image of the road are:

[0093]

[0094] Correspondingly, the disparity map is:

[0095]

[0096] In the formula, is the disparity value in the disparity map, and G is the horizontal pixel coordinate of the left road image.

[0097] Embodiment 2:

[0098] like Figure 7 As shown, this embodiment provides a road surface defect detection method based on parallax, the method comprising:

[0099] Step 1: Acquire a road surface image, wherein the road surface image at least includes a road surface left image and a road surface disparity map;

[0100] Step 2: Traverse the disparity values ​​in the road disparity map column by column to determine the disparity sudden change area in the road disparity map;

[0101] Step 3: Detect road surface defects based on the parallax sudden change area and the left road surface image.

[0102] In any of the above embodiments, further, for a road surface defect of the gap type, due to the factor of illumination, there is no effective parallax at the gap, that is, the parallax value is 0, and the gap should have a certain length in the horizontal direction. Therefore, when traversing the road disparity map, in order to improve the efficiency and accuracy of the disparity traversal, the disparity values ​​in the received road disparity map are traversed column by column to determine the disparity sudden change area in the road disparity map, specifically including:

[0103] Traverse the disparity values ​​in the road disparity map column by column from left to right and from bottom to top;

[0104] When it is determined that the disparity value changes from a positive value to 0, the coordinates of the current position in the road disparity map are marked as the starting point, and the coordinates of the adjacent disparity points with a disparity value of 0 are determined in an outward growth manner and recorded as the regional coordinates;

[0105] The minimum bounding rectangle of the area where the starting point and the area coordinates are located is determined as the parallax sudden change area.

[0106] In any of the above embodiments, further, although the density of the disparity map can be improved by performing disparity fusion on multiple frames of road disparity maps, it is inevitable that there will still be disparity holes in some areas (such as disparity holes caused by shadows). Therefore, when determining the area with sudden disparity changes, these areas need to be filtered out. At the same time, considering that the gap area with a disparity value of 0 is due to the lack of effective illumination, that is, the width of the gap is limited, so if the width of the area with a disparity value of 0 along the longitudinal direction is large, it can be determined that the area is not caused by the gap. Therefore, in this embodiment, the disparity values ​​in the received road disparity map are traversed column by column to determine the area with sudden disparity changes in the road disparity map, and it also includes:

[0107] In the process of outward growth, when it is determined that the number of disparity points with disparity values ​​of 0 determined along any column is greater than the number threshold, the outward growth is stopped, and the area where the corresponding disparity point is located is determined as a disparity hole area;

[0108] Re-traverse the disparity values ​​in the road disparity map and skip the disparity hole area.

[0109] In any of the above embodiments, further, traversing the disparity values ​​in the received road disparity map column by column to determine the disparity sudden change area in the road disparity map specifically includes:

[0110] Traverse the disparity values ​​in the road disparity map column by column from left to right and from bottom to top;

[0111] When it is determined that the difference between the disparity values ​​within the preset column width range is greater than or equal to the sudden change threshold, the coordinates of the current position in the road disparity map are marked as the starting point, and the coordinates of the disparity points in the adjacent area with the same disparity value and / or uniformly increasing disparity values ​​are determined and recorded as the area coordinates;

[0112] The minimum bounding rectangle of the area where the starting point and the area coordinates are located is determined as the parallax sudden change area.

[0113] In any of the above embodiments, further, considering that road surface defects usually have a certain length along the road surface width direction, traversing the disparity values ​​in the received road surface disparity map column by column to determine the disparity sudden change area in the road surface disparity map also includes:

[0114] When the difference between the starting point and the row coordinates of the corresponding disparity point within the regional coordinate range is greater than or equal to the coordinate threshold,

[0115] The minimum circumscribed rectangle of the area where the starting point and the area coordinates are located is determined as the disparity sudden change area. That is, when the width along the horizontal direction of the disparity map is greater than or equal to a certain value, this area is considered to be a potential road defect area.

[0116] In any of the above embodiments, further, when the number of smart street lamps is greater than or equal to two, the fields of view of the binocular cameras of two adjacent smart street lamps will have overlapping areas. In order to avoid multiple detections of the same target, multiple road images received at the same time can be spliced ​​and processed by means of coordinate system transformation to fuse the parallax data of the same target. At this time, the method also includes:

[0117] Based on the world coordinate system, the position coordinates of the smart light pole and the internal parameters of the binocular camera, the overlapping area between two adjacent road images is calculated;

[0118] Perform image stitching on two adjacent road surface images containing overlapping areas to generate a road surface stitching image;

[0119] Traverse the road mosaic image and determine the area with sudden parallax change.

[0120] In any of the above embodiments, further, the process of calculating the road disparity map specifically includes:

[0121] Based on the left road image and the right road image, corresponding left road image features and right road image features are obtained; in this process, a feature extraction network (such as ResNet, ViT, etc.) can be used to extract features to extract the left road image feature F l , road right image feature F r , where F l 、F r ∈R H×W×C Any feature in the left road image features corresponds to a pixel point at a corresponding position in the left road image, and any feature in the right road image features also corresponds to a pixel point at a corresponding position in the right road image.

[0122] Taking the left road image features as the reference, the correlation between any row of pixel features in the left road image features and the corresponding row of pixel features in the right road image features is calculated to generate a correlation matrix. In this process, because the epipolar line of the binocular camera is a horizontal line, such as Figure 6 As shown, therefore, taking the horizontal direction as the reference, the road surface left image feature F is calculated l The pixel features of the i-th row and the road right image feature F r Features of pixels in the same row The correlation of The corresponding calculation formula is:

[0123]

[0124] In the formula, C i is the correlation of the i-th row in the correlation matrix. Therefore, the correlation matrix of the left image features of the entire road surface is Wherein, W1 is the width of the left road surface image, W2 is the width of the right road surface image, and W1=W2.

[0125] Extract the last dimension in the correlation matrix and perform normalization to generate the matching probability distribution array M. The corresponding calculation formula is:

[0126]

[0127] Where softmax() is a normalization operation, and M(u,j) is the matching probability between the feature at (i,j) in the left road image feature and the W2-1 features in the i-th row in the right road image feature.

[0128] Based on the coordinate value of any pixel in the left road image, the corresponding row coordinate in the right road image and the corresponding matching probability distribution array, the disparity value of the corresponding pixel point is calculated to obtain the road disparity map.

[0129] In this process, we use the difference in coordinates as the disparity value of the corresponding pixel point, where the coordinates for calculating the disparity value include the pixel coordinate value and the matching coordinate value. In the process of calculating the matching coordinate value, the corresponding calculation formula is:

[0130]

[0131] In the formula, is the matching coordinate value of the (i, j)th position in the left image of the road surface, that is, the matching coordinate value corresponding to the pixel point at (i, j) in the left image of the road surface is the horizontal coordinate P of the i-th row in the right image feature of the road surface r (i) The sum of the products of the corresponding matching probabilities M(i,j).

[0132] Therefore, the matching coordinate values ​​corresponding to the entire left image of the road are:

[0133]

[0134] Correspondingly, the disparity map is:

[0135]

[0136] In the formula, is the disparity value in the disparity map, and G is the horizontal pixel coordinate of the left road image.

[0137] Embodiment 3:

[0138] Based on the above embodiment, this embodiment further provides a computer program product, the computer program product includes a processor, a storage medium for storing instructions executable by the processor;

[0139] Wherein, the processor is configured to implement the parallax-based road surface defect detection method described in any one of the above-mentioned embodiments 2 when executing executable instructions.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned control methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0141] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0142] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0143] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0144] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A parallax-based road defect detection system, characterized in that: The system comprises: At least one smart lamp pole, on which a binocular camera is installed, and the binocular camera is used to acquire and send a road surface image, wherein the road surface image includes a road surface left image and a road surface disparity map; at least one domain processor, the domain processor being configured to traverse the disparity values ​​in the received road disparity map column by column, determine a disparity sudden change area in the road disparity map, and send the coordinates of the disparity sudden change area and the road left image; A cloud server is configured to perform road surface defect detection based on the received coordinates of the parallax sudden change area and the left road surface image.

2. The parallax-based road surface defect detection system according to claim 1, characterized in that: The step of traversing the received disparity values ​​in the road disparity map column by column to determine a disparity sudden change area in the road disparity map specifically includes: Traversing the disparity values ​​in the road disparity map column by column from left to right and from bottom to top; When it is determined that the disparity value changes from a positive value to 0, the coordinates of the current position in the road disparity map are marked as the starting point, and the coordinates of the adjacent disparity points with a disparity value of 0 are determined in an outward growth manner and recorded as regional coordinates; The minimum circumscribed rectangle of the area where the starting point and the area coordinates are located is determined as the parallax sudden change area.

3. The parallax-based road surface defect detection system according to claim 2, characterized in that: The step of traversing the received disparity values ​​in the road disparity map column by column to determine a disparity sudden change area in the road disparity map further includes: In the process of outward growth, when it is determined that the number of disparity points with disparity values ​​of 0 determined along any column is greater than the number threshold, the outward growth is stopped, and the area where the corresponding disparity point is located is determined as a disparity hole area; The disparity values ​​in the road disparity map are traversed again, and the disparity hole area is skipped.

4. The parallax-based road surface defect detection system according to claim 1, characterized in that: The step of traversing the received disparity values ​​in the road disparity map column by column to determine a disparity sudden change area in the road disparity map specifically includes: Traversing the disparity values ​​in the road disparity map column by column from left to right and from bottom to top; When it is determined that the difference between the disparity values ​​within the preset column width range is greater than or equal to the sudden change threshold, the coordinates of the current position in the road disparity map are marked as the starting point, and the coordinates of the disparity points in the adjacent area of ​​the starting point with the same disparity value and / or uniformly increasing disparity value are determined and recorded as the area coordinates; The minimum circumscribed rectangle of the area where the starting point and the area coordinates are located is determined as the parallax sudden change area.

5. The parallax-based road surface defect detection system according to claim 2 or 4, characterized in that: The step of traversing the received disparity values ​​in the road disparity map column by column to determine a disparity sudden change area in the road disparity map further includes: When it is determined that the difference between the row coordinates of the starting point and the corresponding disparity point within the regional coordinate range is greater than or equal to the coordinate threshold, The minimum circumscribed rectangle of the area where the starting point and the area coordinates are located is determined as the parallax sudden change area.

6. The parallax-based road surface defect detection system according to claim 1, characterized in that: When the number of the smart street lamps is greater than or equal to two, the domain processor is further configured to: Based on the world coordinate system, the position coordinates of the smart lamp pole and the internal parameters of the binocular camera, calculating the overlapping area in two adjacent road images; Performing image stitching on two adjacent road surface images having overlapping areas to generate a road surface stitching image; The road surface stitching image is traversed to determine the parallax sudden change area.

7. The parallax-based road surface defect detection system according to claim 1, characterized in that: The cloud server is also configured to: storing the received left road surface image in a time sequence; When it is determined that there is a defect in the road surface, determining the defect type and the image area of ​​the defect in the left image of the road surface; According to a preset time span, the stored left image of the road surface is retrieved, and the number and type of vehicles passing through the image area are identified and counted; A vehicle-road surface defect prediction model is constructed based on the defect type, the number of vehicles and the vehicle type.

8. The parallax-based road surface defect detection system according to claim 1, characterized in that: The binocular camera is provided with a computing unit, and the computing unit is configured to compute the road disparity map based on the road left image and the road right image, wherein the process of computing the road disparity map specifically includes: Acquire corresponding left road surface image features and right road surface image features based on the left road surface image and the right road surface image; Taking the road surface left image feature as a reference, calculating the correlation between any row of pixel features in the road surface left image feature and the corresponding row of pixel features in the road surface right image feature, and generating a correlation matrix; Extracting the last dimension in the correlation matrix and performing a normalization operation to generate a matching probability distribution array; Based on any pixel coordinate value in the left road image, the corresponding row coordinate in the right road image and the corresponding matching probability distribution array, the disparity value of the corresponding pixel point is calculated to obtain the road disparity map.

9. A parallax-based road surface defect detection method, characterized in that: The method comprises: Step 1: Acquire a road surface image, wherein the road surface image at least includes a road surface left image and a road surface disparity map; Step 2: Traversing the disparity values ​​in the road disparity map column by column to determine a disparity sudden change area in the road disparity map; Step 3: Perform road surface defect detection based on the parallax sudden change area and the left road surface image.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the parallax-based road surface defect detection method according to claim 9 when being executed by a processor.

Images