Millisecond multi-plane real-time extraction method and device in complex traffic scene

Abstract

The application provides a millisecond multi-plane real-time extraction method and device in a complex traffic scene, and relates to the technical field of three-dimensional reconstruction.The method comprises the following steps: for each pixel point of a depth image, searching for pixel points with non-empty depth values in four directions of up, down, left and right within a preset search range, as neighborhood points, and if neighborhood points can be searched in the four directions, the pixel point is a seed point; for each seed point, judging whether the seed point and the four neighborhood points satisfy the following conditions: the position points of the seed point and the two neighborhood points on the left and right in a camera coordinate system are on a straight line, and the position points of the seed point and the two neighborhood points on the top and bottom in the camera coordinate system are on a straight line; if yes, calculating the plane equation of the small plane where the spatial position point corresponding to the seed point is located; and clustering the small planes according to the parameters of the plane equation of the small planes to obtain the plane equation of each main plane in the depth image.The application improves the speed and accuracy of extracting multiple planes of a depth image.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method and apparatus for millisecond-level real-time multi-plane extraction in complex traffic scenarios. Background Technology

[0002] As the third type of advanced feature after feature points and feature lines, the plane can be used for state estimation and feature matching, and has the characteristics of robustness and universality; and is widely used in 3D reconstruction problems.

[0003] Early data acquisition typically used LiDAR, but its development was limited by its high cost and other drawbacks. The emergence of depth cameras improved this situation. This led to the development of various planar detection algorithms for point cloud data, such as RANSAC-based three-point planar detection and region growing algorithms. However, given the real-time requirements of autonomous driving, these algorithms struggle to meet the computational demands of CPU usage.

[0004] With the rapid development of GPUs, parallel computing can greatly optimize computing speed. However, some traditional planar detection algorithms are not suitable for parallel computing, making it difficult to reduce time overhead. Summary of the Invention

[0005] In view of this, this application provides a method and apparatus for millisecond-level multi-plane real-time extraction in complex traffic scenarios to solve the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a millisecond-level multi-plane real-time extraction method for complex traffic scenarios, including:

[0007] Acquire depth images captured by a depth camera;

[0008] For each pixel in the depth image, within a preset search range, search for pixels whose depth values ​​are not empty in the four directions of up, down, left, and right, and use them as neighboring points. If neighboring points can be found in all four directions, then the pixel is a seed point.

[0009] For each seed point, determine whether it and its four neighboring points satisfy the following conditions: the seed point and the positions of the two neighboring points on the left and right are on a straight line in the camera coordinate system, and the seed point and the positions of the two neighboring points above and below are on a straight line in the camera coordinate system; if so, calculate the plane equation of the small plane containing the seed point based on the spatial position coordinates of the seed point and its four neighboring points.

[0010] Based on the parameters of the plane equations of the small planes corresponding to various sub-points in space, the small planes are clustered to obtain the plane equations of each major plane in the depth image.

[0011] Furthermore, for each pixel in the depth image, within a preset search range, pixels with non-empty depth values ​​in the four directions (up, down, left, and right) are searched as neighboring pixels. If neighboring pixels can be found in all four directions, then the pixel is a seed pixel, including:

[0012] Using each pixel p in the depth image as the center point, search for pixels in the left direction within the minimum search distance search_min and the maximum search distance search_max. The first pixel p1 with a non-empty depth value is selected as the neighboring point to the left of the center point.

[0013] With pixel p as the center point, search for pixels in the upward direction within the minimum search distance search_min and the maximum search distance search_max. The first pixel p2 with a non-empty depth value is selected as the neighbor point in the upward direction of the center point.

[0014] With pixel p as the center point, search for pixels in the minimum search distance search_min and the maximum search distance search_max along the right direction. The first pixel p3 with a non-empty depth value is selected as the right-hand neighbor of the center point.

[0015] With pixel p as the center point, search for pixels in the minimum search distance search_min and the maximum search distance search_max along the downward direction. The first pixel p4 with a non-empty depth value is selected as the neighbor point of the center point in the downward direction.

[0016] If a pixel p can find four neighboring pixels p1, p2, p3, and p4, then the pixel is a seed pixel.

[0017] Furthermore, for each seed point, determine whether it and its four neighboring points satisfy the following conditions: the seed point and its two left and right neighboring points are on a straight line in the camera coordinate system, and the seed point and its two upper and lower neighboring points are on a straight line in the camera coordinate system, including:

[0018] For seed point p0 and its neighboring points p1 and p3 in the left and right directions, calculate the horizontal constraint horizaontal:

[0019] horizontal=left*left_depth*(d0-right_depth)+

[0020] right*right_depth*(d0-left_depth)

[0021] Where d0 is the depth value of seed point p0; left is the distance between neighboring point p1 and seed point p0; left_depth is the depth value of neighboring point p1; right is the distance between neighboring point p3 and seed point p0; right_depth is the depth value of neighboring point p3.

[0022] If horizaontal < α, then the seed point p0 and its neighboring points p1 and p3 in the left and right directions are on a straight line in the camera coordinate system; α is a preset threshold.

[0023] For seed point p0 and its neighboring points p2 and p4 in the upper and lower directions, the vertical constraint is calculated as follows:

[0024] vertical=up*up_depth*(d0-down_depth)+

[0025] down*down_depth*(d0-up_depth)

[0026] Where up is the distance between neighboring point p2 and seed point p0; up_depth is the depth value of neighboring point p2; down is the distance between neighboring point p4 and seed point p0; down_depth is the depth value of neighboring point p4.

[0027] If vertical < α, then the seed point p0 and its neighboring points p2 and p4 in the vertical and vertical directions are on a straight line in the camera coordinate system.

[0028] Furthermore, based on the spatial coordinates of the seed point and its four neighboring points, the plane equation of the small plane containing the spatial coordinates of the seed point is calculated; including:

[0029] Transform the coordinates of pixels p0, p1, p2, p3, and p4 from the camera coordinate system to the geocentric Cartesian coordinate system. The spatial coordinates of pixels p0, p1, p2, p3, and p4 are then obtained.

[0030] Calculate spatial location points The normal vector of the plane equation of the small plane.

[0031]

[0032] Where A, B, and C are normal vectors. The three elements;

[0033] Calculate the distance between the plane and the origin of the camera coordinate system to obtain the depth value d:

[0034]

[0035] Then the spatial location point The parameters of the plane equation of the small plane include A, B, C, and d.

[0036] Furthermore, based on the parameters of the plane equations of the small planes corresponding to the spatial locations of various sub-points, the small planes are clustered to obtain the plane equations of each major plane in the depth image; including:

[0037] The three elements of the normal vector of the small plane of each seed point are concatenated, and the concatenated value is used as the x-coordinate and the depth value as the y-coordinate to obtain the coordinate plane. All points are then drawn on the coordinate plane.

[0038] The coordinate plane is gridded according to a preset interval, dividing the coordinate plane into several grids;

[0039] Count the number of points in each grid, and put the grids with more than a preset threshold into the set to be clustered;

[0040] Randomly select a grid from the set to be clustered, and cluster the surrounding valid grids to generate a cluster. Valid grids are those with a number of points greater than a preset threshold. This process yields multiple clusters.

[0041] Get the number of points in all grids in each cluster: m1, m2…m n Where n is the number of grids, and the parameters A1, B1, C1, A2, B2, C2…A are used to obtain the plane equation of the small planes at the center points of all grids. n B n C n and depth values ​​d1, d2…d n ;

[0042] Calculate the normal vector (A0, B0, C0) and depth value d0 of the principal plane corresponding to the cluster:

[0043]

[0044]

[0045]

[0046]

[0047] This yields the plane equations for each principal plane in the depth camera data.

[0048] Secondly, embodiments of this application provide a millisecond-level multi-plane real-time extraction device for complex traffic scenarios, including:

[0049] The acquisition unit is used to acquire depth images captured by the depth camera;

[0050] The neighborhood point acquisition unit is used to search for pixels with non-empty depth values ​​in the four directions (up, down, left, and right) within a preset search range for each pixel in the depth image, and to use them as neighborhood points. If neighborhood points can be found in all four directions, then the pixel is a seed point.

[0051] The calculation unit is used to determine whether each seed point and its four neighboring points satisfy the following conditions: the seed point and the positions of the two neighboring points on the left and right are on a straight line in the camera coordinate system, and the seed point and the positions of the two neighboring points above and below are on a straight line in the camera coordinate system; if so, the plane equation of the small plane containing the seed point is calculated based on the spatial position coordinates of the seed point and the four neighboring points.

[0052] The clustering unit is used to cluster the small planes according to the parameters of the plane equations of the small planes corresponding to the spatial location points of various sub-points, so as to obtain the plane equations of each main plane in the depth image.

[0053] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0054] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method of embodiments of this application.

[0055] This application improves the speed and accuracy of extracting multiple planes from depth images. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 A flowchart of a millisecond-level multi-plane real-time extraction method for complex traffic scenarios provided in this application embodiment;

[0058] Figure 2 A schematic diagram of the seed point and four neighboring points provided in an embodiment of this application;

[0059] Figure 3 A schematic diagram of a small plane provided for an embodiment of this application;

[0060] Figure 4 A functional structure diagram of a millisecond-level multi-plane real-time extraction device for complex traffic scenarios provided in this application embodiment;

[0061] Figure 5 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0064] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0065] Currently, cameras are widely used in many scenarios of autonomous vehicles and robots, such as autonomous driving, object classification, target tracking, and SLAM. Camera-acquired images provide rich environmental representations but cannot express distance information. LiDAR is commonly used to express distance, but its development is limited by its high cost. The simultaneous use of LiDAR and cameras requires calibration; when the relative positions of the two sensors change, the perception results become unreliable. The emergence of depth cameras has improved these problems; currently, most depth cameras are designed as RGB-D cameras. They can simultaneously acquire RGB information and distance information, and their integrated design avoids the uncertainty of the positional relationship between sensors. Because the subprocesses of past planar extraction tasks were highly interdependent, the computation plane could not be parallelized, resulting in significant time consumption.

[0066] Existing methods for extracting planes from depth data, such as RANSAC, first extract the largest plane in the data and then search for other planes in the remaining data; these methods are based on the Manhattan hypothesis. However, many planes in the real environment do not satisfy the orthogonality property, thus failing to meet the requirements of real-time performance and accuracy.

[0067] To address the shortcomings of existing plane extraction techniques in terms of parallelization, this application provides a millisecond-level real-time multi-plane extraction method for complex traffic scenarios. This method can accurately and effectively extract all planes in a depth image, and each step of the implementation process can be computed in parallel. That is, a step is decomposed into a series of repetitive steps that can be executed concurrently, ensuring that each step can be computed using multiple threads simultaneously.

[0068] This application specifically optimizes facet detection using features from depth camera data, adding facet equation calculations and facet clustering to achieve facet detection. To further improve detection efficiency, a grid clustering approach is adopted, accelerating the clustering process and reducing the impact of noise on the results. Furthermore, based on the characteristic that the vertical and horizontal pixels of each pixel in a depth image are fixed, the GPU can perform parallel computation to extract planar information from the image in real time.

[0069] This application can detect all planes in a depth camera. It can use GPU acceleration to further improve the plane detection speed and realize real-time detection in autonomous driving scenarios. Compared with the traditional method of calculating the distance from all other points to the plane to be detected by random three points, small plane clustering reduces the amount of computation, increases the number of detected planes, and optimizes the detection speed while making the detection results more refined by using a parallel approach.

[0070] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0071] like Figure 1 As shown in the figure, this application provides a millisecond-level multi-plane real-time extraction method in complex traffic scenarios, including the following steps:

[0072] Step 101: Acquire the depth image captured by the depth camera;

[0073] Step 102: For each pixel in the depth image, search for pixels whose depth values ​​are not empty in the four directions (up, down, left, and right) within a preset search range. These pixels are then used as neighboring pixels. If neighboring pixels can be found in all four directions, then the pixel is a seed pixel.

[0074] Neighborhood search methods need to consider both search efficiency and accuracy. Searching for neighboring points prepares for mini-plane extraction; the selection of neighboring points should represent the planar variations around pixel p as much as possible. Similarly, the selection of the four nearest neighboring points should also represent the planar variations around p as much as possible. If the selected neighboring points are too close or too far, they cannot adequately represent the pixel variations around p. First, a valid search range is given. Then, valid neighboring points are searched within this range. Outliers and invalid points are removed during the search to ensure the effectiveness of mini-plane extraction. Outliers are considered points where the depth difference between the current point and its four surrounding points is significant. A point is considered invalid if more than one of its four surrounding points is missing.

[0075] Specifically, the steps include:

[0076] Using each pixel p in the depth image as the center point, search for pixels in the left direction within the minimum search distance search_min and the maximum search distance search_max. The first pixel p1 with a non-empty depth value is selected as the neighboring point to the left of the center point.

[0077] With pixel p as the center point, search for pixels in the upward direction within the minimum search distance search_min and the maximum search distance search_max. The first pixel p2 with a non-empty depth value is selected as the neighbor point in the upward direction of the center point.

[0078] With pixel p as the center point, search for pixels in the minimum search distance search_min and the maximum search distance search_max along the right direction. The first pixel p3 with a non-empty depth value is selected as the right-hand neighbor of the center point.

[0079] With pixel p as the center point, search for pixels in the minimum search distance search_min and the maximum search distance search_max along the downward direction. The first pixel p4 with a non-empty depth value is selected as the neighbor point of the center point in the downward direction.

[0080] If a pixel p can find four neighboring pixels p1, p2, p3, and p4, then the pixel is a seed pixel.

[0081] This process ensures that the selection of neighboring points is adaptively adjusted based on the density of points in the surrounding area. Furthermore, it eliminates most of the noise (outliers and invalid points).

[0082] The selection of the neighborhood points [p1, p2, p3, p4] of p is flexible. When there are significant differences in depth information, a suitable distance (empirical value) is explored along the width or height direction. If pixels with similar depth information exist within this distance, then this pixel is used first for facet calculation. This eliminates some of the noise influence and achieves relatively more accurate facet equation calculation. Meanwhile, noise point identification of p is also a necessary step. Only when a pixel has valid neighborhood points in all four directions (up, down, left, and right) is it considered a seed point.

[0083] For each pixel, its neighboring points in four directions can be searched in parallel.

[0084] Step 103: For each seed point, determine whether it and its four neighboring points satisfy the following conditions: the seed point and the positions of the two neighboring points on the left and right are on a straight line in the camera coordinate system, and the seed point and the positions of the two neighboring points above and below are on a straight line in the camera coordinate system; if so, calculate the plane equation of the small plane where the seed point is located based on the spatial position coordinates of the seed point and its four neighboring points.

[0085] like Figure 2 As shown, the coordinates of a pixel in the image plane are two-dimensional, and after adding the depth value, they become three-dimensional coordinates in the camera coordinate system.

[0086] For seed point p0 and its neighboring points p1 and p3 in the left and right directions, calculate the horizontal constraint horizaontal:

[0087] horizontal=left*left_depth*(d0-right_depth)+

[0088] right*right_depth*(d0-left_depth)

[0089] Where d0 is the depth value of seed point p0; left is the distance between neighboring point p1 and seed point p0; left_depth is the depth value of neighboring point p1; right is the distance between neighboring point p3 and seed point p0; right_depth is the depth value of neighboring point p3.

[0090] If horizaontal < α, then the seed point p0 and its neighboring points p1 and p3 in the left and right directions are on a straight line in the camera coordinate system; α is a preset threshold.

[0091] For seed point p0 and its neighboring points p2 and p4 in the upper and lower directions, the vertical constraint is calculated as follows:

[0092] vertical=up*up_depth*(d0-down_depth)+

[0093] down*down_depth*(d0-up_depth)

[0094] Where up is the distance between neighboring point p2 and seed point p0; up_depth is the depth value of neighboring point p2; down is the distance between neighboring point p4 and seed point p0; down_depth is the depth value of neighboring point p4.

[0095] If vertical < α, then the seed point p0 and its neighboring points p2 and p4 in the vertical and vertical directions are on a straight line in the camera coordinate system.

[0096] For each seed point, horizaontal and vertical can be calculated in parallel, and then a judgment can be made.

[0097] Specifically, based on the spatial coordinates of the seed point and its four neighboring points, the plane equation of the small plane containing the spatial coordinates of the seed point is calculated; including:

[0098] Transform the coordinates of pixels p0, p1, p2, p3, and p4 from the camera coordinate system to the geocentric Cartesian coordinate system. The spatial coordinates of pixels p0, p1, p2, p3, and p4 are then obtained.

[0099] Calculate spatial location points The normal vector of the plane equation of the small plane.

[0100]

[0101] Where A, B, and C are normal vectors. The three elements;

[0102] Calculate the distance between the plane and the origin of the camera coordinate system to obtain the depth value d:

[0103]

[0104] Then the spatial location point The parameters of the plane equation of the small plane include A, B, C, and d.

[0105] The plane equations of all small planes can be computed in parallel using multithreading, with the following effect: Figure 3 As shown.

[0106] Step 104: Based on the parameters of the plane equations of the small planes corresponding to the spatial locations of various sub-points, cluster the small planes to obtain the plane equations of each main plane in the depth image;

[0107] In common point clustering into planes, the clustering processes for different points are not independent, which increases the time required for the clustering process. Furthermore, the order in which points are selected for clustering can affect the reliability of the extracted planes. To improve computational speed and ensure the robustness of the extracted planes, this embodiment uses small planes for clustering, and the clustering steps can also be computed in parallel.

[0108] Specifically, the steps include:

[0109] The three elements of the normal vector of the small plane of each seed point are concatenated, and the concatenated value is used as the x-coordinate and the depth value as the y-coordinate to obtain the coordinate plane. All points are then drawn on the coordinate plane.

[0110] The coordinate plane is gridded according to a preset interval, dividing the coordinate plane into several grids;

[0111] Count the number of points in each grid, and put the grids with more than a preset threshold into the set to be clustered;

[0112] Randomly select a grid from the set to be clustered, and cluster the surrounding valid grids to generate a cluster. Valid grids are those with a number of points greater than a preset threshold. This process yields multiple clusters.

[0113] Get the number of points in all grids in each cluster: m1, m2…m n Where n is the number of grids, and the parameters A1, B1, C1, A2, B2, C2…A are used to obtain the plane equation of the small planes at the center points of all grids. n B n C n and depth values ​​d1, d2…d n ;

[0114] Calculate the normal vector (A0, B0, C0) and depth value d0 of the principal plane corresponding to the cluster:

[0115]

[0116]

[0117]

[0118]

[0119] This yields the plane equations for each principal plane in the depth camera data.

[0120] Based on the above embodiments, this application provides a millisecond-level multi-plane real-time extraction device for complex traffic scenarios, see below. Figure 4 As shown, the millisecond-level multi-plane real-time extraction device 200 for complex traffic scenarios provided in this application embodiment includes at least:

[0121] Acquisition unit 201 is used to acquire depth images captured by the depth camera;

[0122] The neighborhood point acquisition unit 202 is used to search for pixels with non-empty depth values ​​in the four directions (up, down, left, and right) within a preset search range for each pixel in the depth image, and to use them as neighborhood points. If neighborhood points can be found in all four directions, then the pixel is a seed point.

[0123] The calculation unit 203 is used to determine whether each seed point and its four neighboring points satisfy the following conditions: the seed point and the positions of the two left and right neighboring points in the camera coordinate system are on a straight line, and the seed point and the positions of the two up and down neighboring points in the camera coordinate system are on a straight line; if so, the plane equation of the small plane where the seed point is located is calculated based on the spatial position coordinates of the seed point and the four neighboring points.

[0124] Clustering unit 204 is used to cluster the small planes according to the parameters of the plane equations of the small planes corresponding to the spatial location points of various sub-points, so as to obtain the plane equations of each main plane in the depth image.

[0125] It should be noted that the principle of the millisecond-level multi-plane real-time extraction device 200 in complex traffic scenarios provided in this application embodiment to solve the technical problem is similar to the millisecond-level multi-plane real-time extraction method in complex traffic scenarios provided in this application embodiment. Therefore, the implementation of the millisecond-level multi-plane real-time extraction device 200 in complex traffic scenarios provided in this application embodiment can refer to the implementation of the millisecond-level multi-plane real-time extraction method in complex traffic scenarios provided in this application embodiment, and the repeated parts will not be described again.

[0126] Based on the above embodiments, this application also provides an electronic device, see below. Figure 5 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and capable of running on the processor 301. When the processor 301 executes the computer program, it implements the millisecond-level multi-plane real-time extraction method for complex traffic scenarios provided in this application embodiment.

[0127] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0128] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0129] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0130] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 5 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0131] It should be noted that, Figure 5 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0132] This application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, so that the electronic device 300 can implement the millisecond-level multi-plane real-time extraction method for complex traffic scenarios provided in this application by executing the built-in or installed executable program.

[0133] The method provided in this application embodiment can also be implemented as a program product, which includes program code. When the program product can run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the millisecond-level multi-plane real-time extraction method in complex traffic scenarios provided in this application embodiment.

[0134] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0135] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0136] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0137] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A millisecond-level multi-plane real-time extraction method for complex traffic scenarios, characterized in that, include: Acquire depth images captured by a depth camera; For each pixel in the depth image, within a preset search range, search for pixels whose depth values ​​are not empty in the four directions of up, down, left, and right, and use them as neighboring points. If neighboring points can be found in all four directions, then the pixel is a seed point. For each seed point, determine whether it and its four neighboring points satisfy the following conditions: the seed point and its two left and right neighboring points are on a straight line in the camera coordinate system, and the seed point and its two upper and lower neighboring points are on a straight line in the camera coordinate system. If so, calculate the plane equation of the small plane containing the seed point based on the spatial coordinates of the seed point and its four neighboring points. Based on the parameters of the plane equations of the small planes corresponding to various sub-points in space, the small planes are clustered to obtain the plane equations of each major plane in the depth image.

2. The millisecond-level multi-plane real-time extraction method for complex traffic scenarios according to claim 1, characterized in that, For each pixel in the depth image, within a preset search range, pixels with non-empty depth values ​​in the four directions (up, down, left, and right) are searched and designated as neighboring pixels. If neighboring pixels can be found in all four directions, the pixel is considered a seed pixel, including: Each pixel of the depth image Using the center point as the starting point, search for pixels within the minimum search distance (search_min) and maximum search distance (search_max) in a leftward direction. Record the first pixel with a non-empty depth value as the first pixel found. , to pixel Neighboring points to the left of the center point; In pixels Using the center point as the starting point, search for pixels within the minimum search distance (search_min) and maximum search distance (search_max) along the upward direction. Record the first pixel with a non-empty depth value as the first pixel found. , to pixel Neighboring points in the upward direction from the center point; In pixels Using the center point as the starting point, search for pixels within the minimum search distance (search_min) and maximum search distance (search_max) in a rightward direction. Record the first pixel with a non-empty depth value as the first pixel found. , to pixel Neighboring points to the right of the center point; In pixels Using the center point as the starting point, search for pixels within the minimum search distance (search_min) and maximum search distance (search_max) in a downward direction. Record the first pixel with a non-empty depth value as the first pixel found. , to pixel Neighboring points in the downward direction from the center point; If pixel Four neighboring points can be found Then the pixel is a seed point.

3. The millisecond-level multi-plane real-time extraction method for complex traffic scenarios according to claim 2, characterized in that, For each seed point, determine whether it and its four neighboring points satisfy the following conditions: the seed point and its two left and right neighboring points are on a straight line in the camera coordinate system, and the seed point and its two upper and lower neighboring points are on a straight line in the camera coordinate system, including: For seed point and neighboring points in the left and right directions and Calculate the approximate horizontal direction : in, Seed point The depth value; For neighborhood points With seed point The distance; For neighborhood points The depth value; For neighborhood points With seed point The distance; For neighborhood points The depth value; like Then the seed point and neighboring points in the left and right directions and The positions of the points in the camera coordinate system lie on a straight line; The preset threshold; For seed point and neighboring points in both the up and down directions and Calculate the approximate vertical direction for: in, For neighborhood points With seed point The distance; For neighborhood points The depth value; For neighborhood points With seed point The distance; For neighborhood points The depth value; like Then the seed point and neighboring points in both the up and down directions and The positions in the camera coordinate system lie on a straight line.

4. The millisecond-level multi-plane real-time extraction method for complex traffic scenarios according to claim 3, characterized in that, Based on the spatial coordinates of the seed point and its four neighboring points, calculate the plane equation of the small plane containing the corresponding spatial coordinate point of the seed point; including: pixels Transform the coordinates of a point in the camera coordinate system to the spatial coordinates of a point in the geocentric Cartesian coordinate system, pixel point The corresponding spatial location point is Calculate spatial location points The normal vector of the plane equation of the small plane. : in, Normal vector The three elements; Calculate the distance between the plane and the origin of the camera coordinate system to obtain the depth value. : Then the spatial location point The parameters of the plane equation of the small plane include and .

5. The millisecond-level multi-plane real-time extraction method for complex traffic scenarios according to claim 4, characterized in that, Based on the parameters of the plane equations of the small planes corresponding to various sub-points in spatial location, the small planes are clustered to obtain the plane equations of each main plane in the depth image; including: The three elements of the normal vector of the small plane of each point are concatenated, and the concatenated value is used as the x-coordinate and the depth value is used as the y-coordinate to obtain the coordinate plane. All points are then drawn on the coordinate plane. The coordinate plane is gridded according to a preset interval, dividing the coordinate plane into several grids; Count the number of points in each grid, and put the grids with more than a preset threshold into the set to be clustered; Randomly select a grid from the set to be clustered, and cluster the surrounding valid grids to generate a cluster. Valid grids are those with a number of points greater than a preset threshold. This process yields multiple clusters. Get the number of points in all grid cells within each cluster: Where n is the number of grids, and the parameters are used to obtain the plane equation of the small planes at the center points of all grids. and depth value ; Calculate the normal vector of the principal plane corresponding to the cluster. and depth value : This yields the plane equations for each principal plane in the depth camera data.

6. A millisecond-level multi-plane real-time extraction device for complex traffic scenarios, characterized in that, include: The acquisition unit is used to acquire depth images captured by the depth camera; The neighborhood point acquisition unit is used to search for pixels with non-empty depth values ​​in the four directions (up, down, left, and right) within a preset search range for each pixel in the depth image, and to use them as neighborhood points. If neighborhood points can be found in all four directions, then the pixel is a seed point. The calculation unit is used to determine whether each seed point and its four neighboring points satisfy the following conditions: the seed point and the positions of the two left and right neighboring points in the camera coordinate system are on a straight line, and the seed point and the positions of the two up and down neighboring points in the camera coordinate system are on a straight line. If so, calculate the plane equation of the small plane containing the seed point based on the spatial coordinates of the seed point and its four neighboring points. The clustering unit is used to cluster the small planes according to the parameters of the plane equations of the small planes corresponding to the spatial location points of various sub-points, so as to obtain the plane equations of each main plane in the depth image.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-5.

Images