Obstacle distance detection method and device, computer device and storage medium

Abstract

The application relates to an obstacle distance detection method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring original point cloud data and image data corresponding to the original point cloud data; performing ground point cloud detection based on the original point cloud data to obtain ground point cloud data, and generating a target ground point cloud region based on the ground point cloud data; performing contact line detection of an obstacle and the ground based on the image data to obtain an obstacle contact line; performing projection based on the target ground point cloud region to obtain a ground point cloud projection region, and determining an obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection region; determining obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud region, performing distance calculation based on horizontal plane coordinate information in the obstacle contact line point cloud data, and obtaining an obstacle distance. The method can improve the accuracy of obstacle distance detection.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an obstacle distance detection method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of autonomous driving technology, it is necessary to detect obstacles in the environment surrounding autonomous vehicles and obtain accurate distance information to plan driving routes and speeds. Existing obstacle ranging methods typically use image data for obstacle detection, but they cannot accurately obtain obstacle distance information. While LiDAR can obtain distance information for obstacle detection, the sparse point cloud of small and distant objects leads to low accuracy in obstacle distance detection. Summary of the Invention

[0003] Therefore, it is necessary to provide an obstacle distance detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of obstacle distance detection in response to the above-mentioned technical problems.

[0004] Firstly, this application provides an obstacle distance detection method. The method includes:

[0005] Acquire the raw point cloud data and the corresponding image data;

[0006] Ground point cloud detection is performed based on raw point cloud data to obtain ground point cloud data, and a target ground point cloud region is generated based on the ground point cloud data.

[0007] The contact line between the obstacle and the ground is detected based on image data to obtain the obstacle contact line;

[0008] Projecting is performed on the target ground point cloud region to obtain the ground point cloud projection region. The obstacle projection contact line corresponding to the obstacle contact line is then determined in the ground point cloud projection region.

[0009] In the target ground point cloud region, determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

[0010] Secondly, this application also provides an obstacle distance detection device. The device includes:

[0011] The acquisition module is used to acquire raw point cloud data and the corresponding image data.

[0012] The ground detection module is used to detect ground point clouds based on raw point cloud data, obtain ground point cloud data, and generate target ground point cloud regions based on ground point cloud data;

[0013] The contact line detection module is used to detect the contact line between an obstacle and the ground based on image data, and to obtain the contact line of the obstacle.

[0014] The projection module is used to project based on the target ground point cloud area to obtain the ground point cloud projection area, and to determine the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area.

[0015] The calculation module is used to determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud region, and to calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

[0016] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0017] Acquire the raw point cloud data and the corresponding image data;

[0018] Ground point cloud detection is performed based on raw point cloud data to obtain ground point cloud data, and a target ground point cloud region is generated based on the ground point cloud data.

[0019] The contact line between the obstacle and the ground is detected based on image data to obtain the obstacle contact line;

[0020] Projecting is performed on the target ground point cloud region to obtain the ground point cloud projection region. The obstacle projection contact line corresponding to the obstacle contact line is then determined in the ground point cloud projection region.

[0021] In the target ground point cloud region, determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

[0022] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0023] Acquire the raw point cloud data and the corresponding image data;

[0024] Ground point cloud detection is performed based on raw point cloud data to obtain ground point cloud data, and a target ground point cloud region is generated based on the ground point cloud data.

[0025] The contact line between the obstacle and the ground is detected based on image data to obtain the obstacle contact line;

[0026] Projecting is performed on the target ground point cloud region to obtain the ground point cloud projection region. The obstacle projection contact line corresponding to the obstacle contact line is then determined in the ground point cloud projection region.

[0027] In the target ground point cloud region, determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

[0028] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0029] Acquire the raw point cloud data and the corresponding image data;

[0030] Ground point cloud detection is performed based on raw point cloud data to obtain ground point cloud data, and a target ground point cloud region is generated based on the ground point cloud data.

[0031] The contact line between the obstacle and the ground is detected based on image data to obtain the obstacle contact line;

[0032] Projecting is performed on the target ground point cloud region to obtain the ground point cloud projection region. The obstacle projection contact line corresponding to the obstacle contact line is then determined in the ground point cloud projection region.

[0033] In the target ground point cloud region, determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

[0034] The aforementioned obstacle distance detection method, apparatus, computer equipment, storage medium, and computer program products detect ground point cloud data from raw point cloud data and generate a target ground point cloud region based on the ground point cloud data. Then, the contact line between the obstacle and the ground is detected using image data, and the target ground point cloud region is projected to obtain a ground point cloud projection region, ensuring that the projection data corresponding to the obstacle contact line exists within the ground point cloud projection region. Through the ground point cloud projection region, the obstacle projection contact line corresponding to the obstacle contact line can be accurately obtained, and thus accurate obstacle contact line point cloud data can be obtained from the obstacle projection contact line. Based on the obstacle contact line point cloud data, an accurate obstacle distance can be calculated, thereby improving the accuracy of obstacle distance detection. Attached Figure Description

[0035] Figure 1This is an application environment diagram of the obstacle distance detection method in one embodiment;

[0036] Figure 2 This is a flowchart illustrating an obstacle distance detection method in one embodiment;

[0037] Figure 3 This is a flowchart illustrating the process of calculating the distance to a target obstacle in one embodiment;

[0038] Figure 4 This is a schematic diagram of the division of ground point cloud regions in one embodiment;

[0039] Figure 5 This is a schematic diagram of the process for generating ground point cloud data in one embodiment;

[0040] Figure 6 This is a schematic diagram of the obstacle contact line in one embodiment;

[0041] Figure 7 This is a flowchart illustrating obstacle distance detection in one embodiment;

[0042] Figure 8 This is a structural block diagram of an obstacle distance detection device in one embodiment;

[0043] Figure 9 This is an internal structural diagram of a computer device in one embodiment;

[0044] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] The obstacle distance detection method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the motion carrier terminal 102 communicates with the server 104 via a network. A data storage system can store the data that the server 104 needs to process. The data storage system can be integrated onto the server 104 or placed on a cloud or other network server. The motion carrier terminal acquires raw point cloud data and corresponding image data; it performs ground point cloud detection based on the raw point cloud data to obtain ground point cloud data, and generates a target ground point cloud region based on the ground point cloud data; it performs contact line detection between the obstacle and the ground based on the image data to obtain the obstacle contact line; it projects the target ground point cloud region to obtain a ground point cloud projection region, and determines the obstacle projection contact line corresponding to the obstacle contact line within the ground point cloud projection region; it determines the obstacle contact line point cloud data corresponding to the obstacle projection contact line within the target ground point cloud region, and calculates the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance. The motion carrier terminal sends the obstacle distance to the server 104 and obtains the motion path, then moves according to the motion path. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0047] In one embodiment, such as Figure 2 As shown, an obstacle distance detection method is provided, which can be applied to... Figure 1 Taking a motion carrier terminal as an example for illustration, it can be understood that this method can also be applied to a server, and also to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0048] Step 202: Obtain the original point cloud data and the corresponding image data.

[0049] Here, raw point cloud data refers to the original point cloud data of the environment surrounding the moving vehicle, which may include obstacles. Image data refers to the original images of the environment surrounding the moving vehicle. The moving vehicle refers to a movable object, which can be an unmanned or manned moving vehicle.

[0050] Specifically, the moving vehicle is equipped with point cloud data acquisition equipment and image acquisition equipment. The point cloud data acquisition equipment and image acquisition equipment are pre-defined with a unified coordinate system and synchronously acquire data at the same frame rate; that is, one frame of image data corresponds to one frame of point cloud data in the original point cloud data. The moving vehicle terminal can acquire the original point cloud data around the moving vehicle through the point cloud data acquisition equipment and the image data around the moving vehicle through the image acquisition equipment. The moving vehicle can be a vehicle, aircraft, robot, etc.

[0051] Step 204: Perform ground point cloud detection based on the original point cloud data to obtain ground point cloud data, and generate the target ground point cloud region based on the ground point cloud data.

[0052] Ground point cloud data refers to the point cloud data representing the ground in the original point cloud data. The target ground point cloud region refers to the ground point cloud data obtained by filling in the ground region in the original point cloud data.

[0053] Specifically, the motion carrier terminal uses a preset ground point classification algorithm to detect the original point cloud data and obtain the ground point cloud data in the original point cloud data. Then, the motion carrier terminal scans the ground point cloud data, and when it detects blank areas in the ground point cloud data, it fills the blank areas with point cloud data based on the existing ground point cloud data to obtain the target ground point cloud area.

[0054] Step 206: Detect the contact line between the obstacle and the ground based on the image data to obtain the obstacle contact line.

[0055] The obstacle contact line refers to the contact area between the obstacle and the ground, which is displayed as a line in the image data.

[0056] Specifically, the motion carrier terminal uses a pre-set deep learning network model to detect the contact area between the image data and the ground, and obtains the obstacle contact line output by the model.

[0057] Step 208: Project the target ground point cloud region to obtain the ground point cloud projection region, and determine the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection region.

[0058] Here, projection refers to the process of projecting 3D point cloud data of a target ground point cloud region onto a 2D plane to obtain point cloud projection data. Point cloud projection data refers to the data of point cloud data projected onto a 2D plane, and it exists in the form of 2D data. In one embodiment, the point cloud data is projected onto a 2D coordinate system identical to the image data to obtain the corresponding point cloud projection data. The ground point cloud projection area refers to the point cloud projection area obtained after projecting the target ground point cloud region. The obstacle projection contact line refers to the point cloud projection data corresponding to the obstacle contact line in the ground point cloud projection area.

[0059] Specifically, the motion carrier terminal converts the three-dimensional coordinates in the target ground point cloud region into horizontal plane coordinates according to pre-set calibration parameters, obtaining the point cloud projection data after the target ground point cloud region is projected, i.e., the ground point cloud projection region. Then, the motion carrier terminal finds the corresponding obstacle projection contact line in the ground point cloud projection region based on the coordinate information of the obstacle contact line.

[0060] Step 210: Determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud region, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

[0061] Among them, obstacle contact line point cloud data refers to the point cloud data corresponding to the projected contact line of the obstacle in the target ground point cloud area. Horizontal plane coordinate information refers to the two-dimensional coordinates representing the horizontal plane in the three-dimensional coordinates of the point cloud data. Obstacle distance refers to the distance between the moving vehicle and the obstacle.

[0062] Specifically, the motion carrier terminal locates the obstacle contact line point cloud data corresponding to the obstacle's projected contact line in the target ground point cloud region based on the projection relationship between the target ground point cloud region and the ground point cloud projection region. Then, it obtains the horizontal plane coordinates from the obstacle contact line point cloud data to calculate the distance between the motion carrier and the obstacle.

[0063] In the aforementioned obstacle distance detection method, ground point cloud data is detected from the original point cloud data, and a target ground point cloud region is generated based on this data. Then, the contact line between the obstacle and the ground is detected using image data. The target ground point cloud region is projected to obtain a ground point cloud projection region, ensuring that the projection data corresponding to the obstacle contact line exists within this region. The ground point cloud projection region allows for the accurate determination of the obstacle's projected contact line, leading to accurate obstacle contact line point cloud data. Based on this point cloud data, the accurate obstacle distance is calculated, thus improving the accuracy of obstacle distance detection.

[0064] In one embodiment, such as Figure 3 The diagram illustrates a flowchart for calculating the distance to a target obstacle; the method further includes:

[0065] Step 302: Obtain the vertical coordinate information corresponding to the obstacle contact line point cloud data and the coordinate information corresponding to the obstacle contact line;

[0066] Step 304: Calculate the point cloud coordinates based on the vertical coordinate information and the coordinate information to obtain the target horizontal plane coordinate information corresponding to the obstacle contact line point cloud data;

[0067] Step 306: Calculate the distance based on the target horizontal plane coordinates to obtain the distance to the target obstacle.

[0068] Vertical coordinate information refers to the vertical coordinates in the three-dimensional coordinates of the point cloud data. Coordinate information refers to the two-dimensional planar coordinates corresponding to the image data. Target horizontal plane coordinate information refers to the horizontal plane coordinates corresponding to the obstacle contact line point cloud data calculated from the point cloud coordinates, including the horizontal and vertical coordinates of the point cloud data. Target obstacle distance refers to the distance between the moving vehicle and the obstacle calculated using the target horizontal plane coordinate information.

[0069] Specifically, the motion platform terminal scans the obstacle contact line point cloud data. When it detects ground point cloud data filling the obstacle contact line point cloud data, it acquires the vertical coordinate information corresponding to the obstacle contact line point cloud data. Then, the motion platform terminal acquires the coordinate information corresponding to the obstacle contact line, which can be obtained when the motion platform terminal uses a preset deep learning network model to detect the contact area between the image data and the ground. The motion platform terminal acquires a pre-stored point cloud coordinate calculation formula and uses the vertical coordinate information and the coordinate information of the obstacle contact line point cloud data to calculate the horizontal plane coordinate information corresponding to the obstacle contact line point cloud data according to the point cloud calculation formula. The motion platform terminal uses the calculated horizontal plane coordinate information of the obstacle contact line point cloud data as the target horizontal plane coordinate information. Then, the motion platform terminal uses the target horizontal plane coordinate information to calculate the distance to the target obstacle.

[0070] When the mobile carrier terminal detects that there is no ground point cloud data to fill in the obstacle contact line point cloud data, it uses the horizontal plane coordinate information corresponding to the existing ground point cloud data in the obstacle contact line point cloud data as the target horizontal plane coordinate information, and uses the target horizontal plane coordinate information to calculate the distance to the target obstacle.

[0071] In one specific embodiment, the point cloud data acquisition device can be a lidar, the image acquisition device can be a monocular camera, and the intrinsic and extrinsic parameter matrices can be obtained in advance through the calibration algorithms of the lidar and the monocular camera. The point cloud coordinate calculation formula is derived through the intrinsic and extrinsic parameter matrices, which are obtained in advance through the calibration algorithm. The calculation formula for projecting the three-dimensional coordinates in the camera coordinate system onto the image through the intrinsic parameter matrix is ​​shown in formula (1):

[0072]

[0073] Where fx and fy are the camera focal lengths, ox and oy are the origin offsets, Xc, Yc, and Zc are three-dimensional points in the camera coordinate system, and U and V are the two-dimensional coordinates of each point in the obstacle contact line.

[0074] The formula for transforming a 3D point in the lidar coordinate system to the camera coordinate system using the extrinsic parameter matrix is ​​shown in formula (2):

[0075]

[0076] Where M is the rotation matrix, tx, ty, tz are translation vectors, Xw, Yw, Zw are 3D points in the lidar coordinate system, and Xc, Yc, Zc are 3D points in the camera coordinate system.

[0077] The point cloud coordinate calculation formula is derived using formulas (1) and (2), as shown in formula (3).

[0078] Xw=((M11-(V-oy)*(M21) / fy)*((U-ox)*(M22*Zw+tz) / fx-M02*Zw-tx)-(M01-(U-ox)*(M21) / fx)*((V-oy)*(M22*Zw+tz) / f y-M12*Zw-ty)) / ((M11-(V-oy)*(M21) / fy)*(M00-(U-ox)*(M20) / fx)-(M01-(U-ox)*(M21) / fx)*(M10-(V-oy)*(M20) / fy));

[0079] Yw=((M10-(V-oy)*(M20) / fy)*((U-ox)*(M22*Zw+tz) / fx-M02*Zw-tx)-(M00-(U-ox)*(M20) / fx)*((V-oy)*(M22*Zw+tz) / fy-M 12*Zw-ty)) / ((M10-(V-oy)*(M20) / fy)*(M01-(U-ox)*(M21) / fx)-(M00-(U-ox)*(M20) / fx)*(M11-(V-oy)*(M21) / fy)) Formula (3),

[0080] Where Xw represents the horizontal coordinate information corresponding to the obstacle contact line point cloud data; Yw represents the vertical coordinate information corresponding to the obstacle contact line point cloud data. The moving vehicle obtains the vertical coordinate information Zw corresponding to the obstacle contact line point cloud data, and obtains the coordinate information (U,V) corresponding to the obstacle contact line. Using Zw and (U,V) according to formula (3), the template horizontal plane coordinate information (Xw, Yw) corresponding to the obstacle contact line point cloud data is obtained, and then (Xw, Yw) is used to calculate the distance to the target obstacle.

[0081] In this embodiment, when the obstacle contact line point cloud data contains filled ground point cloud data, the horizontal plane coordinate information corresponding to the obstacle contact line point cloud data is calculated using the point cloud coordinate calculation formula. This reduces the error of the filled ground point cloud data and improves the accuracy of obstacle distance detection.

[0082] In one embodiment, step 204, generating a target ground point cloud region based on ground point cloud data, includes:

[0083] The ground point cloud data is divided according to the preset division specifications to obtain the ground point cloud division region, which includes the ground point cloud region and the region to be filled.

[0084] Based on the regional ground point cloud data in the ground point cloud region, fill ground point cloud data corresponding to the area to be filled is generated. Based on the regional ground point cloud data and the fill ground point cloud data, the target ground point cloud region is generated.

[0085] The preset partitioning specifications refer to the pre-defined specifications for dividing ground point cloud data into regions. A ground point cloud partitioned region refers to the partitioned area obtained after dividing the ground point cloud data; this partitioned region includes ground point cloud data. A ground point cloud area refers to the area within a partitioned ground point cloud area that contains ground point cloud data. An area to be filled refers to the area within a partitioned ground point cloud area that needs to be filled with ground point cloud data. Filling ground point cloud data refers to filling the ground point cloud data into the area to be filled. Regional ground point cloud data refers to the existing ground point cloud data within a regional ground point cloud area.

[0086] Specifically, the motion carrier terminal divides the ground point cloud data into a fixed-size grid according to a pre-set division specification. For example, the ground point cloud data is divided into a 3*3 square grid according to a 3-meter division specification. Then, the motion carrier terminal searches for grids containing ground point cloud data and scans the distribution of ground point cloud data in the found grids. The areas in the scanned grids containing ground point cloud data are designated as ground point cloud regions, and the areas in the scanned grids without ground point cloud data are designated as regions to be filled.

[0087] Then, the motion carrier terminal estimates the ground point cloud data based on the ground point cloud data in the ground point cloud region, uses the estimated ground point cloud data as the filling ground point cloud data, and fills the area to be filled with the filling ground point cloud data. The motion carrier terminal can scan each grid with ground point cloud data again, and if it detects that there is no area to be filled, it takes each grid with ground point cloud data as the target ground point cloud region.

[0088] In one specific embodiment, such as Figure 4 As shown, this is a schematic diagram of ground point cloud region division; box A represents the divided ground point cloud region; box Aa represents the ground point cloud region; box Ab represents the region to be filled.

[0089] In this embodiment, the ground point cloud data is divided according to a preset division specification to obtain various ground point cloud division regions. The areas to be filled in the ground point cloud division regions are then filled with point cloud data to obtain complete ground point cloud data. This ensures that the target ground point cloud region contains the three-dimensional coordinates corresponding to the obstacle contact line, avoiding data loss when projecting the target ground point cloud region and thus improving the accuracy of obstacle distance detection.

[0090] In one embodiment, such as Figure 5 The diagram illustrates a process for generating filled ground point cloud data. The process involves generating filled ground point cloud data corresponding to the area to be filled based on regional ground point cloud data within the ground point cloud region, including:

[0091] Step 502: Establish regional point cloud coordinate relationships based on regional ground point cloud data;

[0092] Step 504: Obtain the horizontal plane coordinate information of the region corresponding to the regional ground point cloud data, and calculate the horizontal plane coordinate information of the region to be filled based on the regional horizontal plane coordinate information and the preset horizontal plane coordinate interval information.

[0093] Step 506: Based on the filling plane coordinate information, calculate according to the regional point cloud coordinate relationship to obtain the filling vertical coordinate information corresponding to the region to be filled;

[0094] Step 508: Generate the filled ground point cloud data corresponding to the area to be filled based on the filled horizontal plane coordinate information and the filled vertical coordinate information.

[0095] Among them, the regional point cloud coordinate relationship refers to the objective physical relationship between point clouds in the divided regions of the ground point cloud. Different regional point cloud regions correspond to different regional point cloud coordinate relationships. Regional horizontal plane coordinate information refers to the horizontal plane coordinate information corresponding to the ground point cloud data in the regional point cloud region. Filled ground point cloud data refers to the ground point cloud data estimated based on the regional ground point cloud data; this data is used to fill the region to be filled. Preset horizontal plane coordinate interval information refers to the pre-set interval values ​​used to estimate the horizontal plane coordinates between the filled ground point cloud data. Filled horizontal plane coordinate information refers to the horizontal coordinate information corresponding to the filled ground point cloud data. Filled vertical coordinate information refers to the vertical coordinate information corresponding to the filled ground point cloud data.

[0096] Specifically, the motion carrier terminal randomly acquires ground point cloud data from at least two regions, and establishes regional point cloud coordinate relationships based on the ground point cloud data from at least two regions. The motion carrier terminal can establish regional point cloud coordinate relationships using a plane equation: Ax + By + Cz + D = 0, where A, B, C, and D represent the coefficients of the plane equation. The coefficients of the plane equation can be estimated using algorithms such as least squares method and RANSAC. Then, the mobile platform terminal scans the regional ground point cloud data in the ground point cloud area to obtain the horizontal plane coordinate information of the region corresponding to the regional ground point cloud data closest to the region to be filled. Then, it accumulates the regional horizontal plane coordinate information according to the preset horizontal plane coordinate interval information to obtain multiple filling horizontal plane coordinate information corresponding to the region to be filled. For example, if the regional horizontal plane coordinate information corresponding to the regional ground point cloud data is (1, 1), and the preset horizontal plane coordinate interval information is 0.1, then the multiple filling horizontal plane coordinate information corresponding to the filling ground point cloud data are (1.1, 1.1), (1.2, 1.2), (1.3, 1.3), etc. Alternatively, the coordinates can be accumulated separately, such as (1.1, 1.1), (1.1, 1.2), (1.2, 1.3), (1.3, 1.4), etc., or (1.1, 1.1), (1.2, 1.1), (1.3, 1.2), (1.4, 1.3), etc. The motion carrier terminal uses the estimated horizontal plane coordinate information to calculate the vertical coordinate information according to the point cloud coordinate relationship. Then, the motion carrier terminal obtains the ground point cloud data corresponding to the area to be filled based on the horizontal plane coordinate information and the vertical coordinate information, and marks the ground point cloud data for subsequent processing.

[0097] In this embodiment, by estimating the existing ground point cloud data through preset horizontal plane coordinate interval information and regional point cloud coordinate relationships, the ground point cloud data can be accurately filled and complete. This ensures that the three-dimensional coordinates corresponding to the contact line of the obstacle exist in the target ground point cloud area, avoiding data loss when projecting the target ground point cloud area, thereby improving the accuracy of obstacle distance detection.

[0098] In one embodiment, step 208, determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area, includes:

[0099] Based on the contact line coordinate information corresponding to the contact line of the obstacle, search for the projection coordinate information that is the same as the contact line coordinate information in the ground point cloud projection area to obtain the target projection coordinate information;

[0100] Obstacle projection contact lines are obtained based on target projection coordinate information.

[0101] Contact line coordinate information refers to the two-dimensional coordinate information corresponding to the contact line of the obstacle in the acquired image data. Projection coordinate information refers to the two-dimensional coordinate information in the image data corresponding to the ground point cloud projection area.

[0102] Specifically, the motion carrier terminal scans the projection coordinate information in the ground point cloud projection area based on the contact line coordinate information corresponding to the obstacle contact line, and obtains the projection coordinate information in the ground point cloud projection area that is the same as the contact line coordinate information. This obtained projection coordinate information is then used as the target projection coordinate information. The motion carrier terminal then obtains the obstacle projection contact line based on the target projection coordinate information. Specifically, the number of two-dimensional points in the obstacle contact line is determined by the number of pixels corresponding to the image data. Therefore, the number of two-dimensional points in the obstacle projection contact line is determined by the number of two-dimensional points in the obstacle contact line. Furthermore, the number of three-dimensional points in the obstacle contact line point cloud data is determined by the number of two-dimensional points in the obstacle projection contact line.

[0103] In one specific embodiment, such as Figure 6 As shown, a schematic diagram of an obstacle contact line is provided; in the figure, A represents the obstacle in the acquired image data; B represents the ground area in the acquired image data; and line L represents the contact line between the obstacle and the ground.

[0104] In this embodiment, by finding the target projection coordinates that are the same as the contact line coordinates in the ground point cloud projection area, the obstacle projection contact line can be obtained, which can improve the coordinate accuracy of the obstacle projection contact line and thus improve the accuracy of obstacle distance detection.

[0105] In one embodiment, step 210, calculating the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance, includes:

[0106] Obtain the horizontal plane coordinate information corresponding to each three-dimensional point in the point cloud data of the obstacle contact line;

[0107] Based on the horizontal plane coordinate information corresponding to each three-dimensional point, the distance to the obstacle corresponding to each three-dimensional point is calculated.

[0108] The shortest obstacle distance among all the obstacle distances corresponding to each 3D point is taken as the obstacle distance.

[0109] In this context, 3D points represent the constituent units of point cloud data. Point obstacle distance refers to the distance between 3D points on the contact line between the moving vehicle and the obstacle.

[0110] Specifically, the motion carrier terminal acquires the horizontal coordinate information of each three-dimensional point in the obstacle contact line point cloud data, including both lateral and longitudinal coordinates. Distance calculations are then performed using the horizontal coordinate information of each three-dimensional point to obtain the distances to the obstacle at multiple points. The motion carrier terminal can use the formula for calculating the hypotenuse of a triangle to calculate the distance to the obstacle at each point. D represents the distance to the obstacle.

[0111] Then the mobile carrier terminal compares the distances to obstacles at each point and takes the shortest distance as the obstacle distance.

[0112] Then, with the front of the moving vehicle as 0° and counterclockwise as the positive direction, the obstacle's position is calculated based on the lateral and longitudinal coordinates corresponding to the obstacle's distance. The moving terminal vehicle can use the angle calculation formula atan2(y,x) to calculate the obstacle's position. For example, a calculated obstacle position of +45° indicates the obstacle is to the left front of the moving vehicle, and an obstacle position of -45° indicates the obstacle is to the right front of the moving vehicle.

[0113] In this embodiment, by calculating the obstacle distance of each three-dimensional point and taking the shortest obstacle distance as the obstacle distance, the accuracy of obstacle distance can be improved.

[0114] In one specific embodiment, such as Figure 7 As shown, a flowchart of obstacle distance detection is provided; during the driving process of an autonomous vehicle, the on-board terminal obtains the raw point cloud data of the current frame's 360-degree range from the LiDAR, obtains the 360-degree image data of the autonomous vehicle's surroundings from 6 cameras, and performs internal and external parameter calibration on the LiDAR and cameras.

[0115] The vehicle-mounted terminal uses ground point classification algorithms, such as deep learning and traditional geometric algorithms, to locate ground points in the raw point cloud data, obtaining ground point cloud data. Then, it divides the scene into grids with ground point cloud data at a coarse resolution, such as 3 meters per grid cell, and calculates the planar equation for each grid cell containing ground point cloud data. Next, it identifies the missing ground point cloud data areas within the grid, i.e., the areas to be filled. The vehicle-mounted terminal estimates the missing ground point cloud data based on the nearest available ground point cloud data for these areas. It generates horizontal plane coordinate information (x and y) at certain intervals (e.g., 0.1) on the x and y planes of that grid cell. Then, based on the horizontal plane coordinate information (x and y) of the grid cell, the vehicle-mounted terminal uses the corresponding planar equation of that grid cell to calculate the vertical coordinate information for filling, obtaining the filled ground point cloud data. This completes the ground point cloud estimation, thus yielding the target ground point cloud region.

[0116] The vehicle-mounted terminal uses target detection models such as YOLO, SSD, and CenterNet to detect obstacles in the image data collected by the image acquisition device, including traffic participants and obstacles such as vehicles, pedestrians, bicycles, and traffic cones, and outputs the contact line between the obstacle and the ground through the detection model.

[0117] The vehicle-mounted terminal projects the target ground point cloud area to obtain the ground point cloud projection area. Then, it searches within the ground point cloud projection area for the target projection coordinates that match the obstacle's contact line coordinates, thus obtaining the obstacle's projected contact line. Based on the projection relationship, it locates the corresponding obstacle contact line point cloud data within the target ground point cloud area. Using the horizontal plane coordinates from the obstacle contact line point cloud data, it calculates the obstacle distance. The vehicle-mounted terminal can then replan the driving route and speed based on the obstacle distance.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this application also provides an obstacle distance detection device for implementing the obstacle distance detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more obstacle distance detection device embodiments provided below can be found in the limitations of the obstacle distance detection method described above, and will not be repeated here.

[0120] In one embodiment, such as Figure 8 As shown, an obstacle distance detection device 800 is provided, including: an acquisition module 802, a ground detection module 804, a contact wire detection module 806, a projection module 808, and a calculation module 810, wherein:

[0121] The acquisition module 802 is used to acquire the original point cloud data and the image data corresponding to the original point cloud data;

[0122] The ground detection module 804 is used to perform ground point cloud detection based on the original point cloud data, obtain ground point cloud data, and generate a target ground point cloud region based on the ground point cloud data.

[0123] The contact line detection module 806 is used to detect the contact line between an obstacle and the ground based on image data, and to obtain the contact line of the obstacle.

[0124] The projection module 808 is used to project based on the target ground point cloud area to obtain the ground point cloud projection area, and to determine the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area.

[0125] The calculation module 810 is used to determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud region, and to calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

[0126] In one embodiment, the obstacle distance detection device 800 further includes:

[0127] The target obstacle distance calculation unit is used to acquire the vertical coordinate information and the coordinate information corresponding to the obstacle contact line point cloud data; calculate the point cloud coordinates based on the vertical coordinate information and the coordinate information to obtain the target horizontal plane coordinate information corresponding to the obstacle contact line point cloud data; and calculate the distance based on the target horizontal plane coordinate information to obtain the target obstacle distance.

[0128] In one embodiment, the ground detection module 804 includes:

[0129] The region division unit is used to divide the ground point cloud data according to a preset division specification to obtain ground point cloud division regions. The ground point cloud division regions include ground point cloud regions and regions to be filled. Based on the regional ground point cloud data in the ground point cloud regions, the corresponding filled ground point cloud data for the regions to be filled is generated. Based on the regional ground point cloud data and the filled ground point cloud data, the target ground point cloud region is generated.

[0130] In one embodiment, the ground detection module 804 includes:

[0131] The point cloud filling unit is used to establish the coordinate relationship of the regional point cloud based on the regional ground point cloud data; obtain the regional horizontal plane coordinate information corresponding to the regional ground point cloud data; calculate the filling horizontal plane coordinate information corresponding to the area to be filled based on the regional horizontal plane coordinate information and the preset horizontal plane coordinate interval information; calculate the filling vertical coordinate information corresponding to the area to be filled based on the filling plane coordinate information according to the regional point cloud coordinate relationship; and generate the filling ground point cloud data corresponding to the area to be filled based on the filling horizontal plane coordinate information and the filling vertical coordinate information.

[0132] In one embodiment, the projection module 808 includes:

[0133] The coordinate projection unit is used to search for projection coordinate information that is the same as the contact line coordinate information in the ground point cloud projection area based on the contact line coordinate information corresponding to the obstacle contact line, so as to obtain the target projection coordinate information; and to obtain the obstacle projection contact line based on the target projection coordinate information.

[0134] In one embodiment, the computing module 810 includes:

[0135] The distance calculation unit is used to acquire the horizontal plane coordinate information corresponding to each three-dimensional point in the obstacle contact line point cloud data; based on the horizontal plane coordinate information corresponding to each three-dimensional point, the distance is calculated to obtain the obstacle distance corresponding to each three-dimensional point; the shortest obstacle distance among the obstacle distances corresponding to each three-dimensional point is taken as the obstacle distance.

[0136] Each module in the aforementioned obstacle distance detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0137] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores raw point cloud data and image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements an obstacle distance detection method.

[0138] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an obstacle distance detection method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0139] Those skilled in the art will understand that Figure 9-10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0142] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting obstacle distance, characterized in that, The method includes: Acquire the raw point cloud data and the corresponding image data; Ground point cloud data is obtained by performing ground point cloud detection based on the original point cloud data; A target ground point cloud region is generated based on the ground point cloud data; wherein, the ground point cloud data is divided according to a preset division specification to obtain a ground point cloud division region, and the ground point cloud division region includes a ground point cloud region and a region to be filled. Based on the regional ground point cloud data in the ground point cloud region, generate the filled ground point cloud data corresponding to the area to be filled; Establish regional point cloud coordinate relationships based on the regional ground point cloud data; Obtain the horizontal plane coordinate information of the region corresponding to the ground point cloud data of the region, and calculate the horizontal plane coordinate information of the region to be filled based on the horizontal plane coordinate information of the region and the preset horizontal plane coordinate interval information. Based on the horizontal plane coordinate information to be filled, the vertical coordinate information to be filled is calculated according to the point cloud coordinate relationship of the region. Based on the horizontal plane coordinate information and the vertical coordinate information, generate the ground point cloud data corresponding to the area to be filled; The target ground point cloud region is generated based on the regional ground point cloud data and the filled ground point cloud data; Based on the image data, the contact line between the obstacle and the ground is detected to obtain the obstacle contact line; Projecting the target ground point cloud region onto the ground point cloud region yields a ground point cloud projection region. Within this ground point cloud projection region, the obstacle projection contact line corresponding to the obstacle contact line is determined. In the target ground point cloud region, the obstacle contact line point cloud data corresponding to the obstacle projection contact line is determined, and the distance is calculated based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the vertical coordinate information corresponding to the point cloud data of the obstacle contact line and the coordinate information corresponding to the obstacle contact line; Based on the vertical coordinate information and the coordinate information, the point cloud coordinates are calculated to obtain the target horizontal plane coordinate information corresponding to the obstacle contact line point cloud data; The distance to the target obstacle is calculated based on the target's horizontal plane coordinates.

3. The method according to claim 1, characterized in that, The process of generating a target ground point cloud region based on the ground point cloud data includes: Randomly acquire ground point cloud data from at least two regions, and establish regional point cloud coordinate relationships based on the ground point cloud data from at least two regions; the regional point cloud coordinate relationships include plane equations. Scan the regional ground point cloud data in the ground point cloud area to obtain the regional horizontal plane coordinate information corresponding to the regional ground point cloud data that is closest to the area to be filled. The horizontal plane coordinate information of the region is accumulated according to the preset horizontal plane coordinate interval information to obtain multiple horizontal plane coordinate information corresponding to the region to be filled; Based on the horizontal coordinate information of the area to be filled, the vertical coordinate information of the filling is calculated using the plane equation to obtain the filled ground point cloud data. The missing areas of the ground point cloud data are filled in based on the ground point cloud data.

4. The method according to claim 1, characterized in that, The process of generating a target ground point cloud region based on the ground point cloud data includes: The ground point cloud data is divided into grids of fixed specifications; Locate the grids containing ground point cloud data and scan the distribution of ground point cloud data within the found grids; The area in the scanned grid containing ground point cloud data is defined as the ground point cloud area, and the area in the scanned grid without ground point cloud data is defined as the area to be filled.

5. The method according to claim 1, characterized in that, Determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area includes: Based on the contact line coordinate information corresponding to the contact line of the obstacle, the target projection coordinate information is obtained by searching for the same projection coordinate information as the contact line coordinate information in the ground point cloud projection area. The obstacle projection contact line is obtained based on the target projection coordinate information.

6. The method according to claim 1, characterized in that, The distance calculation based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance includes: Obtain the horizontal plane coordinate information corresponding to each three-dimensional point in the point cloud data of the obstacle contact line; Based on the horizontal plane coordinate information corresponding to each of the three-dimensional points, the distance to the point obstacle corresponding to each of the three-dimensional points is calculated respectively. The shortest obstacle distance among the obstacle distances corresponding to each of the three-dimensional points is taken as the obstacle distance.

7. An obstacle distance detection device, characterized in that, The device includes: The acquisition module is used to acquire the original point cloud data and the image data corresponding to the original point cloud data; The ground detection module is used to perform ground point cloud detection based on the original point cloud data to obtain ground point cloud data, and generate a target ground point cloud region based on the ground point cloud data. The region division unit is used to divide the ground point cloud data based on a preset division specification to obtain a ground point cloud division region, which includes a ground point cloud region and a region to be filled. A point cloud filling unit is used to generate filled ground point cloud data corresponding to the area to be filled based on the regional ground point cloud data in the ground point cloud region; establish regional point cloud coordinate relationships based on the regional ground point cloud data; obtain regional horizontal plane coordinate information corresponding to the regional ground point cloud data; calculate the filling horizontal plane coordinate information corresponding to the area to be filled based on the regional horizontal plane coordinate information and preset horizontal plane coordinate interval information; calculate the filling vertical coordinate information corresponding to the area to be filled based on the filling horizontal plane coordinate information and the regional point cloud coordinate relationships; generate filled ground point cloud data corresponding to the area to be filled based on the filling horizontal plane coordinate information and the filling vertical coordinate information; and generate the target ground point cloud region based on the regional ground point cloud data and the filled ground point cloud data. A contact line detection module is used to detect the contact line between the obstacle and the ground based on the image data to obtain the obstacle contact line. The projection module is used to project based on the target ground point cloud region to obtain a ground point cloud projection region, and to determine the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection region. The calculation module is used to determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud region, and to perform distance calculation based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images