Method, device and computer program product for determining the area of ​​pavement damage

Abstract

This disclosure provides a method, apparatus, electronic device, storage medium, and program product for determining the area of ​​pavement defects, relating to the field of computer technology, specifically to smart highways, computer vision, deep learning, and other technical fields, and applicable to pavement defect detection scenarios. The specific implementation scheme is as follows: pavement defect objects are identified from image data including the defect objects; based on the correspondence between point cloud data including the defect objects and image data, a target point cloud corresponding to the defect objects is determined from the point cloud data; and the area of ​​the defect objects is determined based on the target point cloud. This disclosure avoids the problem of inaccurate area determination caused by bounding box-based detection methods, where the bounding box includes non-defect areas outside the defect objects, thus improving the accuracy of the determined pavement defect area.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to technical fields such as smart highways, computer vision, and deep learning, and in particular to methods, devices, electronic devices, storage media, and computer program products for determining the area of ​​road defects, which can be applied to road defect detection scenarios. Background Art

[0002] In the field of smart highway maintenance, road images are generally captured by on-board cameras, and then a deep learning model is used to identify road defects based on the road images. After identifying the road defects, it is often necessary to calculate the size of the road defects to assess the severity of the road defects. In the prior art, the bounding box of the road defect is generally determined first, and then the severity of the road defect is estimated based on the size of the bounding box. The detection method based on the bounding box cannot directly calculate the area information of the defect itself, and the detection accuracy is low. Summary of the invention

[0003] The present disclosure provides a method, an apparatus, an electronic device, a storage medium and a computer program product for determining the area of ​​pavement damage.

[0004] According to a first aspect, a method for determining the area of ​​a pavement defect is provided, comprising: determining a pavement defect object from image data including the pavement defect object; determining a target point cloud corresponding to the pavement defect object from the point cloud data based on a correspondence between point cloud data including the pavement defect object and image data; and determining the area of ​​the pavement defect object based on the target point cloud.

[0005] According to a second aspect, a device for determining the area of ​​a pavement defect is provided, comprising: a first determination unit, configured to determine a pavement defect object from image data including the pavement defect object; a second determination unit, configured to determine a target point cloud corresponding to the pavement defect object from the point cloud data based on a correspondence between point cloud data including the pavement defect object and image data; and a third determination unit, configured to determine the area of ​​the pavement defect object based on the target point cloud.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0009] According to the technology disclosed in the present invention, a method for determining the area of ​​a pavement defect is provided. First, a pavement defect object is accurately determined from image data including the pavement defect object. Then, according to the correspondence between point cloud data and image data that also include the pavement defect object, a target point cloud corresponding to the pavement defect object is determined from the point cloud data based on the pavement defect object predetermined in the image data, thereby improving the accuracy of the determined target point cloud. Finally, based on the target point cloud, the actual area of ​​the pavement defect object is determined, thereby avoiding the problem of inaccurate area caused by the inclusion of non-disease areas outside the pavement defect object in the detection method based on the circumscribed rectangular frame, thereby improving the accuracy of the determined area of ​​the pavement defect.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0013] Figure 2 is a flow chart of an embodiment of a method for determining the area of ​​pavement damage according to the present disclosure;

[0014] Figure 3A is a schematic diagram of a pavement disease object according to the present disclosure;

[0015] Figure 3B is a schematic diagram of the result of detecting pavement damage objects on image data according to the present disclosure;

[0016] Figure 4 is a schematic diagram of a target point cloud corresponding to a pavement disease object according to the present disclosure;

[0017] Figure 5 is a schematic diagram of an application scenario of the method for determining the area of ​​road surface damage according to this embodiment;

[0018] Figure 6 is a schematic structural diagram of a sub-region in a unit region according to this embodiment;

[0019] Figure 7 is a flow chart of another embodiment of a method for determining the area of ​​pavement damage according to the present disclosure;

[0020] Figure 8 is a structural diagram of an embodiment of a device for determining the area of ​​pavement damage according to the present disclosure;

[0021] Fig. 9 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0024] Figure 1 An exemplary architecture 100 is shown to which the method and apparatus for determining the area of ​​pavement damage of the present disclosure may be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] Terminal devices 101, 102, 103 may be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, 103 are hardware, they may be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to inspection vehicles, smart phones, tablet computers, e-book readers, laptop computers and desktop computers equipped with image acquisition devices and / or point cloud acquisition devices, etc. When terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. It may be implemented as multiple software or software modules for providing distributed services, for example, or it may be implemented as a single software or software module. No specific limitation is made here.

[0027] The server 105 may be a server that provides various services, for example, a background processing server that determines the area of ​​a pavement damage object based on the image data and point cloud data of the pavement damage object acquired by the terminal devices 101, 102, 103. As an example, the server 105 may be a cloud server.

[0028] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0029] It should also be noted that the method for determining the area of ​​pavement damage provided by the embodiments of the present disclosure can be executed by a server, or by a terminal device, or by a server and a terminal device in cooperation with each other. Accordingly, the various parts (e.g., various units) included in the device for determining the area of ​​pavement damage can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0030] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements. When the electronic device on which the method for determining the area of ​​pavement damage is running does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (e.g., server or terminal device) on which the method for determining the area of ​​pavement damage is running.

[0031] Please refer to Figure 2 , Figure 2 A flow chart of a method for determining the area of ​​pavement damage provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0032] Step 201 : determining a pavement defect object from image data including the pavement defect object.

[0033] In this embodiment, the execution subject of the method for determining the area of ​​pavement damage (for example, Figure 1 The terminal device or server in the image data processing unit can obtain image data including the pavement damage object remotely or locally through a wired network connection or a wireless network connection, and determine the pavement damage object from the image data including the pavement damage object. The image data can include a picture of the pavement damage object or a video frame in a video.

[0034] Whether it is cement pavement or asphalt pavement, after a period of use, various road surface diseases such as damage, deformation and other defects will appear one after another. Specifically, road surface diseases include cracks, potholes, rutting, looseness, subsidence, bridge head culvert top jumping, surface damage and other types.

[0035] As an example, the execution subject can determine the pixel-level pavement damage objects from the image data including the pavement damage objects through the pre-trained semantic segmentation model. The semantic segmentation model is used to characterize the correspondence between the image data and the pavement damage objects. As an example, the semantic segmentation model can be, for example, a U-Net model, a FCN (Fully Convolutional Networks) model, a SegNet, a PSPNet (Pyramid Scene Parseing Network) model, or a DeepLab model.

[0036] As another example, the execution subject may determine the pavement damage object from the image data including the pavement damage object by using a pre-trained target detection model, wherein the target detection model is used to characterize the correspondence between the image data and the pavement damage object.

[0037] The target detection model can be either a single-stage target detection model or a two-stage target detection model. Taking the single-stage target detection model as an example, it can be a YOLO (You Only Look Once) model, an SSD (Single Shot MultiBox Detector) model, a RetinaNet model, a FCOS (Fully Convolutional One-Stage Object Detection) model, or a CornerNet model; taking the two-stage target detection model as an example, it can be a Faster RCNN (Region-based Convolutional Neural Networks) model, a Cascade RCNN model, or a Mask RCNN model.

[0038] The semantic segmentation model and the target detection model can be trained in the following way: first, a training sample set is obtained, wherein the training samples in the training sample set include sample images and pixel-level labels representing the pavement disease objects in the sample images; then, a machine learning method is used, with the sample images as the input of the initial semantic segmentation model and the initial target detection model, and the pixel-level labels corresponding to the input training samples as the expected output of the initial semantic segmentation model and the initial target detection model, and the initial target detection model is iteratively trained to obtain the trained semantic segmentation model and the target detection model.

[0039] By iteratively training the initial semantic segmentation model and the initial object detection model, in response to reaching a preset end condition, a trained semantic segmentation model and an object detection model are obtained. The preset end condition may be, for example, that the training time exceeds a preset time threshold, the number of training times exceeds a preset number threshold, and the training loss tends to converge.

[0040] Continue to refer Figure 3A , an image 300 representing pavement damage objects is shown, including pavement damage objects 301, 302, and 303. Figure 3B , which shows a schematic diagram of the results of pavement damage object detection on image data.

[0041] Step 202 : According to the correspondence between the point cloud data including the pavement defect object and the image data, a target point cloud corresponding to the pavement defect object is determined from the point cloud data.

[0042] In this embodiment, the above-mentioned execution entity can obtain point cloud data including pavement damage objects remotely or locally through a wired network connection or a wireless network connection, and determine the target point cloud corresponding to the pavement damage object from the point cloud data based on the correspondence between the point cloud data and the image data.

[0043] In this embodiment, the correspondence between the point cloud data and the image data is specifically manifested as the correspondence between the point cloud in the point cloud data and the pixel points in the image data. Based on the pixel-level pavement damage object determined in step 201, the point cloud corresponding to the pixel points corresponding to the pavement damage object in the point cloud data can be determined as the target point cloud corresponding to the pavement damage object. Figure 4 , showing Figure 3A , Figure 3B Schematic diagram of a target point cloud 400 of a pavement defect object 301 in FIG.

[0044] As an example, the point cloud data may be a three-dimensional environment model that represents the surrounding environment, including road disease objects; further, the above-mentioned execution entity may determine the calibration information corresponding to the image data, determine the projection matrix of the three-dimensional environment model onto the image data based on the calibration information, and further determine the correspondence between the point cloud data and the image data based on the projection matrix.

[0045] Step 203: Determine the area of ​​the pavement damage object according to the target point cloud.

[0046] In this embodiment, the above-mentioned execution subject can determine the area of ​​the road surface disease object according to the target point cloud.

[0047] As an example, the above execution subject can first determine the contour point cloud in the target point cloud, where the contour point cloud represents the contour of the pavement damage object; then, sort the contour point cloud to obtain an ordered contour point cloud; finally, determine the area of ​​the pavement damage object by the following formula:

[0048]

[0049] Among them, n represents the number of point clouds in the contour point cloud, x i y i+1 、x i+1 y i They represent the point clouds in the ordered contour point clouds respectively.

[0050] As another example, the execution subject may first divide the area corresponding to the target point cloud into grids of a preset size; then, count the number of grids involved in the area corresponding to the target point cloud; finally, according to the determined number and size of grids, determine the area of ​​the pavement defect object by the following formula:

[0051] AREA=nummber×L×L

[0052] Among them, nummber represents the number of grids, and L represents the side length of the grid.

[0053] Continue to see Figure 5 , Figure 5 FIG. 5 is a schematic diagram 500 of an application scenario of the method for determining the area of ​​road damage according to this embodiment. Figure 5 In the application scenario, first, the server 501 obtains the image data 503 and the point cloud data 504 including the pavement disease object through the inspection vehicle 502; then, the pavement disease object 5031 is determined from the image data 503; then, according to the correspondence between the point cloud data 504 including the pavement disease object and the image data 503, the target point cloud 505 corresponding to the pavement disease object is determined from the point cloud data 504; finally, according to the target point cloud 505, the area of ​​the pavement disease object is determined.

[0054] In this embodiment, a method for determining the area of ​​a pavement disease is provided. First, a pavement disease object is accurately determined from image data that includes the pavement disease object. Then, based on the correspondence between point cloud data and image data that also include the pavement disease object, a target point cloud corresponding to the pavement disease object is determined from the point cloud data based on the pavement disease object predetermined in the image data, thereby improving the accuracy of the determined target point cloud. Finally, based on the target point cloud, the actual area of ​​the pavement disease object is determined, thereby avoiding the problem of inaccurate area caused by the inclusion of non-disease areas outside the pavement disease object in the circumscribed rectangular frame-based detection method, thereby improving the accuracy of the determined pavement disease area.

[0055] In some optional implementations of the present embodiment, the execution entity may perform step 203 in the following manner: first, for each point in the target point cloud, determine the equivalent area of ​​the unit area corresponding to the point based on the distance between the point and its adjacent points; second, determine the area of ​​the pavement damage object based on the equivalent area of ​​the unit area corresponding to each point in the target point cloud.

[0056] As an example, the above-mentioned execution entity can first determine the distance between adjacent points in the target point cloud through the Euler formula; then, for each point in the target point cloud, the above-mentioned execution entity takes the point as the center, divides the boundary between the point and its adjacent points according to the distance between the point and its adjacent points, and determines the size information of the unit area corresponding to each point; then, the area of ​​the unit area is determined according to the size information of the unit area; finally, the equivalent area of ​​the unit area corresponding to each point in the target point cloud is accumulated to obtain the area of ​​the pavement disease object.

[0057] In this implementation, in the process of determining the area of ​​the pavement damage object, the area is subdivided into the area of ​​the unit area corresponding to each point, and the area of ​​the pavement damage object is finally determined by accumulating the equivalent area of ​​the unit area corresponding to each point in the target point cloud, thereby further improving the accuracy of the determined area of ​​the pavement damage object.

[0058] In some optional implementations of the present embodiment, the execution entity may perform the first step as follows: first, for each sub-region in the unit region corresponding to the point, determine the area of ​​the sub-region based on the distance between the target adjacent point corresponding to the sub-region among the adjacent points of the point and the point; then, based on the area of ​​each sub-region in the unit region corresponding to the point, determine the equivalent area of ​​the unit region corresponding to the point.

[0059] Continue to refer Figure 6 , a schematic diagram 600 of the structure of a sub-region in a unit region is shown. The adjacent points of point cloud 601 in the target point cloud are 602, 603, 604, and 605. The unit region of point cloud 601 includes sub-regions 6011, 6012, 6013, and 6014. Among them, sub-region 6011 corresponds to point clouds 602 and 603. The length of one side a of sub-region 6011 is determined by the distance between point clouds 601 and 602, and the length of another side b of sub-region 6011 is determined by the distance between point clouds 601 and 603.

[0060] As an example, half of the distance between point clouds 601 and 602 may be determined as the length of one side a of sub-region 6011 , and half of the distance between point clouds 601 and 603 may be determined as the length of another side b of sub-region 6011 .

[0061] After determining the lengths of the two sides of the sub-region 6011, the area of ​​the sub-region 6011 can be determined; similarly, the areas of the sub-regions 6012, 6013, and 6014 of the unit area of ​​the point cloud 601 can be determined. The area of ​​the unit area of ​​the point cloud 601 can be obtained by accumulating the areas of the sub-regions 6011, 6012, 6013, and 6014.

[0062] Based on the above method, the area of ​​the unit area of ​​each point can be obtained, and the area of ​​the pavement disease object can be determined by accumulating the area of ​​the unit area of ​​each point in the target point cloud.

[0063] In this implementation, a specific method for determining the area of ​​a unit region corresponding to a point cloud is provided, which improves the accuracy of the determined area of ​​the unit region, and further helps to improve the accuracy of the determined area of ​​the pavement defect object.

[0064] In some optional implementations of this embodiment, the above-mentioned execution entity can execute the above-mentioned step 202 in the following decentralized manner: first, according to the correspondence between the point cloud data and the image data, determine the initial target point cloud corresponding to the road surface disease object from the point cloud data; second, perform plane fitting on the initial target point cloud to obtain the target point cloud.

[0065] Point cloud data generally represents three-dimensional spatial information, and the point cloud corresponding to the pavement damage object determined from the point cloud data may be a point cloud in three-dimensional space. At this time, the point cloud corresponding to the pavement damage object determined from the point cloud data can be determined as the initial target point cloud, and the initial target point cloud is plane-fitted to obtain the target point cloud in the plane, so as to facilitate the calculation of the area of ​​the pavement damage object represented by the target point cloud.

[0066] The process of fitting a plane to discrete points in space is actually an optimization process, that is, finding the minimum sum of the distances from these points to a plane.

[0067] As an example, the above execution subject can perform plane fitting by the least square method. First, the general formula of the plane equation is as follows:

[0068] ax+by+cz+d=0

[0069] Assuming c≠0, the above formula can be converted to:

[0070]

[0071] make but:

[0072] z=a0x+a1y+a2

[0073] If there exists a series of point sets {(x,y,z)|(x,y,z)∈(x i ,y i ,z i ), i = 0, 1, 2..., n-1}, according to the least squares principle, the sum of squares of errors is minimized, that is:

[0074]

[0075] By solving the above formula, the coefficients of the fitted plane can be determined, and the fitted plane, that is, the target point cloud, can be obtained.

[0076] In this implementation, by fitting the point cloud of the pavement damage object in three-dimensional space into a planar target point cloud, it helps to improve the convenience and accuracy of the calculation process of the area of ​​the pavement damage object.

[0077] In some optional implementations of this embodiment, the above-mentioned execution entity can perform the above-mentioned second step in the following manner: first, plane fitting is performed on the initial target point cloud to obtain a plane point cloud; then, free points in the plane point cloud are filtered out to obtain a target point cloud.

[0078] As an example, the above execution entity can filter out free points in the plane point cloud based on statistical filtering or other methods to obtain the target point cloud.

[0079] In this implementation, by filtering out the free points in the plane point cloud, the accuracy of the target point cloud is improved, which helps to improve the calculation accuracy of the area of ​​the road surface disease object.

[0080] In some optional implementations of the present embodiment, the above-mentioned execution subject may also perform the following operations before executing the above-mentioned step 201: first, three-dimensional reconstruction is performed based on the multi-perspective image data acquired by the image acquisition device to obtain point cloud data; then, according to the three-dimensional reconstruction process, the correspondence between the point cloud in the point cloud data and the pixel points in the image data is determined.

[0081] As an example, the image acquisition device may be a binocular camera, and the multi-view image data may be left and right eye images taken by the binocular camera. After obtaining the left and right eye images, the above-mentioned execution subject may calibrate the binocular camera through a binocular calibration tool to determine the internal and external parameters of the binocular camera; then, perform binocular corrections such as distortion correction and stereo correction to obtain left and right eye correction images; then, based on the left and right eye correction images, stereo matching is performed on the left and right eye correction images through a stereo matching algorithm such as BM (Boyer-Moore) or SGM (Semi-Global Matching) to calculate a disparity map; based on the disparity map, three-dimensional reconstruction is performed to obtain point cloud data.

[0082] During the 3D reconstruction process, the correspondence between the point cloud in the point cloud data and the pixel points in the image data can be determined.

[0083] In this implementation, a scenario of calculating pavement damage objects under multi-view image data is provided, which expands the application scope of calculating pavement damage objects.

[0084] In some optional implementations of this embodiment, the above-mentioned execution entity may also perform the following operations before executing the above-mentioned step 201: collect single-view image data through an image acquisition device; collect point cloud data through a radar device; and determine the correspondence between the point cloud in the point cloud data and the pixel points in the image data based on a calibration algorithm.

[0085] In this implementation, a target-based joint calibration algorithm or a target-free joint calibration can be used. In the target-based joint calibration algorithm, the target refers to a man-made reference object, such as a chessboard, etc. The main algorithm is to obtain external parameters by detecting the corner points of the image and the lidar and designing the reprojection error function. The target-free joint calibration algorithm is generally suitable for dynamic calibration, and uses the environment around the vehicle body as a reference, such as matching lane lines, to complete the joint calibration between the radar and the image acquisition device.

[0086] Taking the target-based joint calibration algorithm as an example, first, a new calibration plate is designed as a target for the joint calibration task. Under this calibration plate, the joint calibration of the image acquisition device (e.g., camera) intrinsic parameters, distortion factors, and radar-camera extrinsic parameters can be completed. The calibration plate needs to be captured by the radar and the camera, and the calibration plate needs to have obvious features (e.g., edges, corners, etc.) for positioning. Specifically, the calibration plate contains a checkerboard for camera intrinsic parameter calibration and four circular holes for locating radar point clouds. The circular holes are more robust to the data of the positioning radar. Then, the calibration plate is detected to locate the corner points of the calibration plate and the center points of the circular holes. For images, the calibration plate can be detected using the opencv function; for point cloud data, a calibration plate mask is first established, and the approximate calibration plate area of ​​the mask is fitted by RANSAC (Random Sample Consensus), and then the grid search algorithm is used to accurately fit the center points of the four circular holes. Then, by minimizing the reprojection error of 3D-2D point pairs (point pairs originate from the center points of the circular holes of the calibration plate and the corner points of the chessboard), a nonlinear optimization function of the camera-radar is constructed. Then, based on the calibration information, image and point cloud calibration data are collected, and the reprojection error of the corner points of the chessboard and the reprojection error of the center points of the circular holes are optimized to determine the final joint calibration information, so as to determine the correspondence between the point cloud in the point cloud data and the pixel points in the image data based on the joint calibration information.

[0087] In this implementation, a scenario for calculating pavement damage objects based on single-view image data of an image acquisition device and point cloud data of a radar is provided, thereby expanding the scope of application of calculating pavement damage objects.

[0088] In some optional implementations of this embodiment, after determining the area of ​​the pavement damage object, the execution subject may further perform the following operations: determining degree information representing the severity of the pavement damage according to the area of ​​the pavement damage object.

[0089] As an example, the execution subject may establish a corresponding relationship table representing the corresponding relationship between the area range of the pavement disease object and the severity level. According to the area of ​​the pavement disease object, the area range to which it belongs is determined, and then the severity level corresponding to the determined area range is determined as the degree information representing the severity of the pavement disease.

[0090] In some optional implementations, in the process of determining the severity information, the execution subject may consider not only the area of ​​the pavement defect object, but also the defect type to which the pavement defect object belongs. It is understandable that cracks of the same area are generally less severe than potholes.

[0091] For each type of pavement damage object, the execution subject may establish a correspondence table that characterizes the correspondence between the area range and severity level of the pavement damage object of that type. While determining the area of ​​the pavement damage object, the execution subject determines the type of the pavement damage object through a pre-trained classification model; and then, through the correspondence table corresponding to the determined type, determines the severity information according to the area of ​​the pavement damage object.

[0092] In this implementation, the degree information representing the severity of the pavement disease is determined according to the area of ​​the pavement disease object, thereby improving the intuitiveness and effectiveness of the information.

[0093] Continue to refer Figure 7 , shows a schematic process 700 of another embodiment of a method for determining the area of ​​pavement damage according to the present disclosure, comprising the following steps:

[0094] Step 701: determine a pavement defect object from image data including the pavement defect object.

[0095] Step 702: According to the correspondence between the point cloud data and the image data, an initial target point cloud corresponding to the road surface disease object is determined from the point cloud data.

[0096] Step 703: perform plane fitting on the initial target point cloud to obtain a plane point cloud.

[0097] Step 704: filter out the free points in the plane point cloud to obtain the target point cloud.

[0098] Step 705: for each point in the target point cloud, perform the following operations:

[0099] Step 7051: for each sub-region in the unit region corresponding to the point, determine the area of ​​the sub-region according to the distance between the target adjacent point corresponding to the sub-region and the point among the adjacent points of the point.

[0100] Step 7052: Determine the equivalent area of ​​the unit area corresponding to the point based on the area of ​​each sub-area in the unit area corresponding to the point.

[0101] Step 706: Determine the area of ​​the pavement damage object based on the equivalent area of ​​the unit area corresponding to each point in the target point cloud.

[0102] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiments, process 700 of the method for determining the area of ​​pavement damage in this embodiment specifically illustrates the process of plane fitting, the process of filtering out discrete points, and the process of determining the area of ​​the pavement damage object based on the equivalent area of ​​the unit area of ​​each point, thereby further improving the accuracy of the determined area of ​​the pavement damage object.

[0103] Continue to refer Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for determining the area of ​​road damage. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0104] like Figure 8 As shown, the device 800 for determining the area of ​​pavement damage includes: a first determination unit 801, configured to determine the pavement damage object from the image data including the pavement damage object; a second determination unit 802, configured to determine the target point cloud corresponding to the pavement damage object from the point cloud data based on the correspondence between the point cloud data including the pavement damage object and the image data; a third determination unit 803, configured to determine the area of ​​the pavement damage object based on the target point cloud.

[0105] In some optional implementations of this embodiment, the third determination unit 803 is further configured to: for each point in the target point cloud, determine the equivalent area of ​​the unit area corresponding to the point based on the distance between the point and its adjacent points; and determine the area of ​​the pavement disease object based on the equivalent area of ​​the unit area corresponding to each point in the target point cloud.

[0106] In some optional implementations of the present embodiment, the third determination unit 803 is further configured to: for each sub-region in the unit region corresponding to the point, determine the area of ​​the sub-region based on the distance between the target adjacent point corresponding to the sub-region among the adjacent points of the point and the point; and determine the equivalent area of ​​the unit region corresponding to the point based on the area of ​​each sub-region in the unit region corresponding to the point.

[0107] In some optional implementations of this embodiment, the second determination unit 802 is further configured to: determine the initial target point cloud corresponding to the road surface disease object from the point cloud data based on the correspondence between the point cloud data and the image data; and perform plane fitting on the initial target point cloud to obtain the target point cloud.

[0108] In some optional implementations of this embodiment, the second determining unit 802 is further configured to: perform plane fitting on the initial target point cloud to obtain a plane point cloud; and filter out free points in the plane point cloud to obtain a target point cloud.

[0109] In some optional implementations of this embodiment, the above-mentioned device also includes: an obtaining unit (not shown in the figure), which is configured to perform three-dimensional reconstruction based on multi-perspective image data acquired by the image acquisition device to obtain point cloud data; a fourth determination unit (not shown in the figure), which is configured to determine the correspondence between the point cloud in the point cloud data and the pixel points in the image data according to the three-dimensional reconstruction process.

[0110] In some optional implementations of this embodiment, the above-mentioned device also includes: an acquisition unit (not shown in the figure), configured to acquire single-perspective image data through an image acquisition device; acquire point cloud data through a radar device; and a fifth determination unit (not shown in the figure), configured to determine the correspondence between the point cloud in the point cloud data and the pixel points in the image data based on a calibration algorithm.

[0111] In some optional implementations of this embodiment, the above-mentioned device also includes: a sixth determination unit (not shown in the figure), which is configured to determine the degree information representing the severity of the road surface disease according to the area of ​​the road surface disease object.

[0112] In the present embodiment, a device for determining the area of ​​a pavement defect is provided. First, a pavement defect object is accurately determined from image data including the pavement defect object. Then, according to the correspondence between point cloud data and image data which also include the pavement defect object, a target point cloud corresponding to the pavement defect object is determined from the point cloud data based on the pavement defect object predetermined in the image data, thereby improving the accuracy of the determined target point cloud. Finally, according to the target point cloud, the actual area of ​​the pavement defect object is determined, thereby avoiding the problem of inaccurate area caused by the inclusion of non-disease areas outside the pavement defect object in the detection method based on the circumscribed rectangular frame, thereby improving the accuracy of the determined area of ​​the pavement defect.

[0113] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the method for determining the area of ​​pavement damage described in any of the above embodiments when executing.

[0114] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method for determining the area of ​​pavement damage described in any of the above embodiments when executed.

[0115] An embodiment of the present disclosure provides a computer program product, which, when executed by a processor, can implement the method for determining the area of ​​pavement damage described in any of the above embodiments.

[0116] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0117] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0118] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0119] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a method for determining the area of ​​pavement damage. For example, in some embodiments, the method for determining the area of ​​pavement damage may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method for determining the area of ​​pavement damage described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the method for determining the area of ​​pavement damage in any other appropriate manner (eg, by means of firmware).

[0120] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0122] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services; it may also be a server for a distributed system, or a server combined with a blockchain.

[0126] According to the technical solution of the embodiment of the present disclosure, a method for determining the area of ​​pavement damage is provided. First, a pavement damage object is accurately determined from image data including the pavement damage object; then, according to the correspondence between point cloud data and image data that also include the pavement damage object, a target point cloud corresponding to the pavement damage object is determined from the point cloud data based on the pavement damage object predetermined in the image data, thereby improving the accuracy of the determined target point cloud; finally, based on the target point cloud, the actual area of ​​the pavement damage object is determined, thereby avoiding the problem of inaccurate area caused by the inclusion of non-damaged areas outside the pavement damage object in the detection method based on the circumscribed rectangular frame, thereby improving the accuracy of the determined area of ​​the pavement damage.

[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining the area of ​​pavement damage, include: Determining the pavement damage object from the image data including the pavement damage object; According to the correspondence between the point cloud data including the pavement damage object and the image data, determining a target point cloud corresponding to the pavement damage object from the point cloud data; For each sub-region in the unit region corresponding to each point in the target point cloud, divide the boundary line between the point and the multiple target adjacent points according to the distance between the point and the multiple target adjacent points, and determine the area of ​​the sub-region, wherein the multiple target adjacent points are multiple adjacent points corresponding to the sub-region among the adjacent points of the point; determine the equivalent area of ​​the unit region corresponding to the point according to the area of ​​each sub-region in the unit region corresponding to the point; The area of ​​the pavement damage object is determined based on the equivalent area of ​​the unit area corresponding to each point in the target point cloud.

2. The method according to claim 1, in, The step of determining a target point cloud corresponding to the pavement damage object from the point cloud data according to the correspondence between the point cloud data including the pavement damage object and the image data comprises: Determining an initial target point cloud corresponding to the pavement damage object from the point cloud data according to a corresponding relationship between the point cloud data and the image data; Performing plane fitting on the initial target point cloud to obtain the target point cloud.

3. The method according to claim 2, in, The performing plane fitting on the initial target point cloud to obtain the target point cloud includes: Performing plane fitting on the initial target point cloud to obtain a plane point cloud; The free points in the plane point cloud are filtered out to obtain the target point cloud.

4. The method according to claim 1, in, Also includes: Performing three-dimensional reconstruction based on the image data of multiple perspectives acquired by an image acquisition device to obtain the point cloud data; According to the three-dimensional reconstruction process, the correspondence between the point cloud in the point cloud data and the pixel points in the image data is determined.

5. The method according to claim 1, in, Also includes: Acquire the image data of a single perspective by an image acquisition device; Collecting the point cloud data by radar equipment; Based on a calibration algorithm, a correspondence between the point cloud in the point cloud data and the pixel points in the image data is determined.

6. The method according to any one of claims 1 to 5, in, Also includes: According to the area of ​​the pavement damage object, degree information representing the severity of the pavement damage is determined.

7. A device for determining the area of ​​road surface damage, include: A first determining unit is configured to determine the pavement damage object from the image data including the pavement damage object; A second determining unit is configured to determine a target point cloud corresponding to the pavement damage object from the point cloud data according to a correspondence between the point cloud data including the pavement damage object and the image data; The third determination unit is configured to, for each sub-region in the unit region corresponding to each point in the target point cloud, divide the boundary line between the point and the multiple target adjacent points according to the distance between the point and the multiple target adjacent points, and determine the area of ​​the sub-region, wherein the multiple target adjacent points are multiple adjacent points corresponding to the sub-region among the adjacent points of the point; determine the equivalent area of ​​the unit region corresponding to the point according to the area of ​​each sub-region in the unit region corresponding to the point; The area of ​​the pavement damage object is determined based on the equivalent area of ​​the unit area corresponding to each point in the target point cloud.

8. The device according to claim 7, in, The second determining unit is further configured to: According to the correspondence between the point cloud data and the image data, an initial target point cloud corresponding to the road surface disease object is determined from the point cloud data; and plane fitting is performed on the initial target point cloud to obtain the target point cloud.

9. The device according to claim 8, in, The second determining unit is further configured to: Performing plane fitting on the initial target point cloud to obtain a plane point cloud; filtering out free points in the plane point cloud to obtain the target point cloud.

10. The device according to claim 7, in, Also includes: an obtaining unit configured to perform three-dimensional reconstruction based on the image data of multiple perspectives acquired by an image acquisition device to obtain the point cloud data; The fourth determining unit is configured to determine the correspondence between the point cloud in the point cloud data and the pixel points in the image data according to the three-dimensional reconstruction process.

11. The device according to claim 7, in, Also includes: A collection unit is configured to collect the image data of a single perspective through an image collection device; and collect the point cloud data through a radar device; The fifth determining unit is configured to determine the correspondence between the point cloud in the point cloud data and the pixel points in the image data based on a calibration algorithm.

12. The device according to any one of claims 7 to 11, in, Also includes: The sixth determination unit is configured to determine the severity information representing the severity of the road surface disease according to the area of ​​the road surface disease object.

13. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, include: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images