Point cloud processing method and apparatus

Abstract

The application relates to a point cloud processing method and device. The method comprises the following steps: determining a first transformation matrix corresponding to adjacent first and second point cloud frames through a particle swarm optimization algorithm; rotating and / or translating an original coordinate system of the first point cloud frame through the first transformation matrix to obtain a first coordinate system corresponding to the first point cloud frame; and superimposing first point clouds in the first point cloud frame into the second point cloud frame according to the first coordinate system, a driving speed of a device, and a time difference between the first and second point cloud frames. The scheme provided by the application can increase the density of point clouds and improve the recognition ability of a laser radar for long-distance obstacles.

Description

Technical Field

[0001] This application relates to the field of image data processing technology, and in particular to point cloud processing methods and apparatus. Background Technology

[0002] LiDAR (Light Detection and Ranging) is a radar system that uses laser beams to detect the position, velocity, and other characteristics of targets. Its working principle involves emitting a detection signal (laser beam) towards the target, then comparing the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, information about the target can be obtained, such as its distance, azimuth, altitude, speed, attitude, and even shape. This allows it to be applied to obstacle detection in aircraft, vehicles, and other transportation vehicles.

[0003] Most current LiDAR-based algorithm development is based on processing single-frame point clouds. Due to the divergence characteristics of LiDAR, the density of the point cloud becomes sparse as the distance increases, resulting in poor recognition of obstacles at long distances.

[0004] In some solutions, high-beam LiDAR is used to identify obstacles at a distance. Compared with low-beam LiDAR, high-beam LiDAR can obtain more point clouds and has a stronger ability to identify obstacles at a distance. However, high-beam LiDAR is expensive. Summary of the Invention

[0005] To address or partially address the problems existing in related technologies, this application provides a point cloud processing method and apparatus that can increase the density of point clouds and improve the ability of lidar to identify obstacles at long distances.

[0006] The first aspect of this application provides a point cloud processing method, comprising: determining a first transformation matrix corresponding to an adjacent first point cloud frame and a second point cloud frame using a particle swarm optimization algorithm; rotating and / or translating the original coordinate system of the first point cloud frame using the first transformation matrix to obtain a first coordinate system corresponding to the first point cloud frame; and superimposing a first point cloud in the first point cloud frame onto the second point cloud frame based on the first coordinate system, the device's travel speed, and the time difference between the first point cloud frame and the second point cloud frame.

[0007] A second aspect of this application provides a point cloud processing apparatus, comprising: a determination module, configured to determine a first transformation matrix corresponding to adjacent first point cloud frames and second point cloud frames using a particle swarm optimization algorithm;

[0008] The transformation module is used to rotate and / or translate the original coordinate system of the first point cloud frame through a first transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame.

[0009] The overlay module is used to overlay the first point cloud in the first point cloud frame onto the second point cloud frame based on the first coordinate system, the device's driving speed, and the time difference between the first point cloud frame and the second point cloud frame.

[0010] A third aspect of this application provides an electronic device, comprising:

[0011] Processor; and

[0012] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0013] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0014] The technical solution provided in this application can determine the transformation matrix corresponding to two adjacent point cloud frames using a particle swarm optimization algorithm. The original coordinate system in the first point cloud frame is rotated and / or translated using this transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame. Based on the first coordinate system, the device's travel speed, and the time difference between the first and second point cloud frames, the first point cloud in the first frame is superimposed onto the second point cloud frame. In other words, this technical solution can register two point cloud frames, allowing the points in the two frames to be superimposed, thereby increasing the density of the point cloud. This means the density of the point cloud corresponding to distant obstacles will also increase, thus improving the scanning device's ability to detect distant obstacles.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0017] Figure 1 This is a schematic flowchart illustrating the point cloud processing method in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram illustrating the process of determining the first rotation and translation matrix using a particle swarm optimization algorithm, as shown in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of the original coordinate system of the first point cloud frame shown in the embodiments of this application;

[0020] Figure 4This is a schematic diagram of the first coordinate system corresponding to the first point cloud frame shown in the embodiments of this application;

[0021] Figure 5 This is a schematic diagram illustrating the display of the first and second point clouds in the original coordinate system of the second point cloud frame, as shown in an embodiment of this application.

[0022] Figure 6 This is a schematic diagram of the point cloud processing device shown in the embodiments of this application;

[0023] Figure 7 This is another schematic diagram of the point cloud processing device shown in the embodiments of this application;

[0024] Figure 8 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0025] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0026] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0028] To facilitate understanding of the embodiments of this application, some terms used in the embodiments of this application will be introduced below.

[0029] Point cloud: Generally refers to the information of points on the surface of an object measured by 3D scanning equipment, such as LiDAR, stereo cameras, and Time-of-Flight (TOF) cameras. Each point contains 3D coordinates and may contain color information or reflectivity information. These point clouds are collected by scanning equipment and output as relevant data, which can then be read by point cloud processing devices.

[0030] Point cloud frame: a frame of point cloud. The scanning device scans the surrounding environment at a certain rotation frequency. When it encounters an obstacle, it returns point cloud data. The point cloud data is then processed to obtain a frame of point cloud.

[0031] Particle Swarm Optimization (PSO): Also known as particle swarm optimization, it is a stochastic search algorithm based on group cooperation, developed by simulating the foraging behavior of flocks of birds. It is generally considered a type of swarm intelligence (SI). It can be incorporated into Multiagent Optimization Systems (MAOS).

[0032] Curvature: Used to reflect the geometric shape changes of point clouds and their neighborhoods.

[0033] Normal vector: also known as the normal unit vector, used to identify the direction of the principal normal to the surface formed by the point cloud and its neighborhood.

[0034] To facilitate understanding of the embodiments of this application, the applicable scenarios for the embodiments of this application are described below.

[0035] With the development of automotive electronics technology, related technologies such as sensing technology, image processing, and artificial intelligence are gradually being applied in the automotive field, making autonomous driving, assisted driving, and driverless driving technologies increasingly popular. Obstacle recognition is crucial in autonomous driving, assisted driving, and driverless driving technologies. Among related technologies, vehicles are typically equipped with LiDAR (Light Detection and Ranging) sensors, which allow vehicles to identify obstacles in their surrounding environment.

[0036] In related technologies, LiDAR (Light Detection and Ranging) scans the environment around a vehicle by rotating at a certain frequency. When it encounters an obstacle, it returns a frame of point cloud data. A point cloud processing device processes several frames of point cloud data obtained by the LiDAR scan through a predetermined recognition process and then outputs an image corresponding to the obstacle in the vehicle's surrounding environment. However, due to the divergent characteristics of LiDAR, the density of point cloud data in each frame becomes sparser as the distance increases. Therefore, the recognition ability is relatively poor for obstacles that are far away from the vehicle, meaning the accuracy of the output result is relatively low.

[0037] To address the aforementioned issues, this application provides a point cloud processing method that can increase the density of point clouds and improve the ability of lidar to identify obstacles at long distances.

[0038] It should be understood that the above scenarios are merely examples. The point cloud processing method in this embodiment can be applied not only to automobiles but also to other devices with driver assistance functions. This embodiment does not impose any specific limitations. The device for obtaining the point cloud in this embodiment can be not only a lidar system but also other scanning devices. This embodiment does not impose any specific limitations.

[0039] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0040] Figure 1 This is a schematic flowchart illustrating the point cloud processing method in an embodiment of this application.

[0041] See Figure 1 The point cloud processing method in this embodiment includes:

[0042] 101. The point cloud processing device determines the first transformation matrix corresponding to the adjacent first point cloud frame and second point cloud frame through the particle swarm algorithm.

[0043] The point cloud processing device acquires two adjacent point cloud frames through a scanning device, and then determines the first transformation matrix corresponding to these two point cloud frames according to the particle swarm optimization algorithm. For ease of description, this embodiment refers to these two adjacent point cloud frames as the first point cloud frame and the second point cloud frame.

[0044] Specifically, the point cloud processing device can obtain the first point cloud frame and the second point cloud frame in the following way: the scanning device scans the surrounding environment and returns the corresponding point cloud to the point cloud processing device when it encounters an obstacle. After obtaining the returned point cloud, the point cloud processing device uses a k-dimensional tree (kd-tree) to search for the k nearest neighbors of each point cloud to obtain the corresponding point cloud frame.

[0045] It should be understood that a kd-tree is a tree-like data structure for storing instance points in a k-dimensional space for fast retrieval, primarily used for searching key data in multi-dimensional space. The specific method of obtaining the corresponding point cloud frame by searching the k nearest neighbors of a point cloud using a kd-tree is common knowledge to those skilled in the art and will not be elaborated upon in this embodiment. It should also be understood that, in addition to searching point clouds using a kd-tree, the point cloud processing device can also search point clouds using other methods, which are not limited in this embodiment.

[0046] Specifically, the point cloud processing device can determine the first transformation matrix in the following manner, such as... Figure 2 As shown, the process of determining the first transformation matrix includes:

[0047] (1) Calculate the curvature of the first point cloud in the first point cloud frame and the second point cloud in the second point cloud frame;

[0048] After acquiring the first point cloud frame, the point cloud processing device divides the first point cloud frame into several first regions, each containing one or more adjacent first point clouds. Then, the point cloud processing device establishes a coordinate system for each first region, and for each first region, determines the first coordinate value corresponding to the first point cloud in the coordinate system of that first region. Based on the first coordinate value, the device determines the first surface corresponding to the first region using the least squares method, and determines the curvature of the first surface as the curvature corresponding to the first point cloud in that first region.

[0049] Optionally, the point cloud processing device can also determine the normal unit vector of the first surface as the normal vector corresponding to the first point cloud in the first region.

[0050] After acquiring the second point cloud frame, the point cloud processing device divides the second point cloud frame into several second regions, each containing one or more adjacent second point clouds. Then, the point cloud processing device establishes a coordinate system for each second region, and for each second region, determines the second coordinate value corresponding to the second point cloud in the coordinate system of that second region. Based on the second coordinate value, the device uses the least squares method to determine the second surface corresponding to that second region, and determines the curvature of the second surface as the curvature corresponding to the second point cloud in that second region.

[0051] Optionally, the point cloud processing device can also determine the normal unit vector of the second surface as the normal vector corresponding to the second point cloud in the second region.

[0052] (2) Pair the first point cloud in the first point cloud frame with the second point cloud in the second point cloud frame according to the curvature to obtain several pairs of point clouds.

[0053] After calculating the curvature, the point cloud processing device pairs the first point cloud in the first point cloud frame with the second point cloud in the second point cloud frame to obtain several pairs of point clouds. Each pair of point clouds contains point clouds with similar curvature. Specifically, in step (1) above, the curvature of several first regions and the curvature of second regions are calculated. The first regions and second regions with similar curvatures are grouped together. The first point cloud of the first region in each group and the second point cloud of the second region in this group constitute a pair of point clouds.

[0054] For example, the first region A contains two point clouds, A1 and A2, and the curvature of A is 25.25 cm. -1 The first region B contains three point clouds, B1, B2, and B3, with a curvature of 27.18 cm. -1 The second region C contains two point clouds, C1 and C2, with a curvature of 25.28 cm. -1The second region D contains three point clouds, D1, D2, and D3, with a curvature of 27.31 cm. -1 The point cloud processing device then groups the first region A and the second region C, which have similar curvatures, into one group, i.e., point cloud A (A1, A2) and point cloud C (C1 and C2) are a pair of point clouds. The first region B and the second region D are also grouped into one group, i.e., point cloud B (B1, B2 and B3) and point cloud D (D1, D2 and D3) are a pair of point clouds.

[0055] (3) Generate n target transformation matrices;

[0056] After the point cloud processing device obtains several pairs of point clouds, it generates n target transformation matrices. In this embodiment, the value of n (i.e., the number of target transformation matrices) corresponds to the number of particles in the particle swarm optimization algorithm, and is a user-preset integer greater than 0.

[0057] Specifically, in particle swarm optimization (PSO), particles are defined by their attributes (position and velocity). In this embodiment, one transformation matrix corresponds to one particle in the PSO algorithm. Different attribute values ​​correspond to different matrices. The point cloud processing device can generate initial n attribute values ​​using a random algorithm. The n matrices corresponding to these n attribute values ​​are the target transformation matrices. The point cloud processing device can also generate n target transformation matrices in other ways; this embodiment does not limit the specific method used.

[0058] (4) For each target transformation matrix, determine the optimization function value corresponding to the target transformation matrix;

[0059] After the point cloud processing device generates n target transformation matrices, or after each update of the target transformation matrix, it calculates the corresponding optimization function value for each target transformation matrix. This optimization function value corresponds to the sum of the cross products of the normal vectors of several point cloud pairs. It should be understood that the sum of the cross products of the normal vectors of several point cloud pairs refers to the sum of the cross products of the normal unit vectors of the several point cloud pairs. The optimization function value refers to the value calculated based on the target transformation matrix and the optimization function. The optimization function includes the sum of the cross products of the normal vectors of several point cloud pairs, where the normal vector of each point cloud in the point cloud pair corresponds to the target transformation matrix.

[0060] Specifically, the point cloud processing unit can use a graphics processing unit (GPU) to compute the optimization function values ​​corresponding to each target transformation matrix in parallel.

[0061] More specifically, for each target transformation matrix, the point cloud processing device can determine the optimization function value corresponding to the target transformation matrix in the following way:

[0062] S1. Rotate and / or translate the original coordinate system of the first point cloud frame in the first point cloud frame through the target transformation matrix to obtain the third coordinate system corresponding to the first point cloud frame;

[0063] Specifically, the target transformation matrix is ​​used to rotate the coordinate system clockwise / counterclockwise by an angle θ and then translate it by a vector T. The original coordinate system of the first point cloud frame is O-XY. The point cloud processing device rotates the coordinate system O-XYZ by an angle θ according to the target transformation matrix and then translates the coordinate system by a vector T to obtain the third coordinate system O`-X`Y`Z`.

[0064] S2. Calculate the first normal vector corresponding to the first point cloud in the first point cloud frame with the third coordinate system.

[0065] The point cloud processing device determines the normal vector (first normal vector) corresponding to each first point cloud based on the coordinate values ​​of each first point cloud in the third coordinate system O`-X`Y`Z`. Specifically, the point cloud processing device can determine the surface corresponding to the point cloud in the manner described in step (1) above, and determine the normal vector corresponding to the surface as the normal vector corresponding to the point cloud.

[0066] S3. For several pairs of point clouds, calculate the cross product of the first normal vector corresponding to the first point cloud in the pair and the second normal vector corresponding to the second point cloud in the pair, and add the cross products of the unit normal vectors of each pair of point clouds to obtain the optimization function value corresponding to the target transformation matrix.

[0067] After calculating the first normal vector corresponding to each first point cloud, the point cloud processing device performs a cross product of the first normal vector corresponding to the first point cloud in each pair of point cloud pairs and the normal vector of the second point cloud in the pair to obtain the cross product of the normal vectors of the pair. The value obtained by adding the cross products of the normal vectors of each pair of point cloud pairs is the optimization function value corresponding to the target transformation matrix.

[0068] For example, two pairs of point clouds are obtained by pairing: one pair consists of a first point cloud A and a second point cloud C, and the other pair consists of a first point cloud B and a second point cloud D. The normal vector of the first point cloud A is determined as F1, and the normal vector of the first point cloud C is determined as F2, based on the coordinate values ​​of the first point clouds A and C in the third coordinate system O`-X`Y`Z`. The normal vector of the second point cloud B is calculated as F3, and the normal vector of the second point cloud D is calculated as F4, obtained in step (2) above. The point cloud processing device calculates the optimized function value y = F1 × F2 + F3 × F4.

[0069] (5) Determine whether the minimum value in the optimization function is zero. If yes, execute step (6); otherwise, execute steps (7) and (8).

[0070] After determining the optimization function value corresponding to each target transformation matrix, the point cloud processing device needs to determine whether the function has converged. In this embodiment, the function convergence condition is that the minimum value is zero. After determining the optimization function value corresponding to each target transformation matrix, the point cloud processing device needs to determine whether the minimum value among these optimization function values ​​is zero. If it is, then step (6) is executed; if not, then steps (7) and (8) are executed.

[0071] (6) Determine the target transformation matrix corresponding to the minimum value as the first transformation matrix;

[0072] When the minimum value of the optimization function corresponding to each target transformation matrix is ​​determined to be zero, that is, the function converges, the point cloud processing device can stop the iterative calculation and output the target transformation matrix corresponding to the minimum value, that is, determine the target transformation matrix as the first transformation matrix.

[0073] (7) Determine the target transformation matrix corresponding to the minimum value as the transformation matrix to be optimized;

[0074] When it is determined that the minimum value of the optimization function corresponding to each target transformation matrix is ​​not zero, that is, the function has not converged, the point cloud processing device needs to continue to perform iterative calculations to determine the target transformation matrix corresponding to the minimum value as the transformation matrix to be optimized.

[0075] (8) Determine whether the number of iterations has reached the preset value. If yes, execute step (9); otherwise, execute step (10).

[0076] When it is determined that the minimum value of the optimization function value corresponding to each target transformation matrix is ​​not zero, the point cloud processing device determines whether the number of iterations has reached the preset value. If yes, then step (9) is executed; otherwise, step (10) is executed.

[0077] It should be understood that iteration is an activity involving repeated feedback processes, typically aimed at approximating a desired target or result. Each repetition of the process is called an "iteration," and the result of each iteration serves as the initial value for the next iteration. The number of iterations in iterative calculations refers to the number of loops performed during the iterative operation. In this embodiment, the number of iterations refers to the number of times the target transformation matrix is ​​updated and the optimization function value is calculated.

[0078] (9) Determine the first transformation matrix as the one with the smallest corresponding optimization function value among the transformation matrices obtained in each iteration.

[0079] At each iteration point, if the function does not converge, the point cloud processing device calculates a transformation matrix to be optimized. When the number of iterations has not reached a preset value, the point cloud processing device selects the transformation matrix with the smallest corresponding optimization function value among the currently calculated transformation matrices to be optimized as the first transformation matrix.

[0080] (10) Update the n target transformation matrices and execute steps (4) and (5).

[0081] When the number of iterations reaches the preset value, the point cloud processing device updates the n target transformation matrices and executes steps (4) to (5) and the corresponding subsequent processes. That is, the point cloud processing device will repeatedly execute steps (4) to (10) until the first transformation matrix is ​​determined (that is, the minimum value in the optimization function value is zero or the number of iterations reaches the preset value).

[0082] Specifically, in particle swarm optimization algorithms, particles are updated by updating their attributes (position and velocity). In this embodiment, the point cloud processing device can also set attributes for the target transformation matrix; different attribute values ​​correspond to different matrices. The point cloud processing device updates the target transformation matrix by updating the attribute values ​​of the target transformation matrix. The point cloud processing device can also update the target transformation matrix in other ways, which are not limited in this embodiment.

[0083] It should be understood that, in addition to determining the first transformation matrix through the methods corresponding to steps (1) to (10) above, the point cloud processing device can also determine the first transformation matrix through other methods, which are not limited in this embodiment.

[0084] 102. The point cloud processing device rotates and / or translates the original coordinate system of the first point cloud frame through a first transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame;

[0085] After determining the first transformation matrix, the point cloud processing device rotates and / or translates the original coordinate system of the first point cloud frame using the first transformation matrix to obtain the first coordinate system. It should be understood that the coordinate system in this embodiment refers to a spatial coordinate system. Transforming a spatial coordinate system using a transformation matrix is ​​common knowledge to those skilled in the art, and this embodiment does not limit the specific transformation.

[0086] 103. The point cloud processing device superimposes the first point cloud in the first point cloud frame onto the second point cloud frame based on the first coordinate system, the device's travel speed, and the time difference between the first point cloud frame and the second point cloud frame.

[0087] After the point cloud processing device determines the first coordinate system, it superimposes the point clouds in the two point cloud frames (the first point cloud frame and the second point cloud frame) based on the first coordinate system, the device's driving direction, driving speed, and the time difference between the two point cloud frames. Superimposing the point clouds in the two point cloud frames means displaying the point clouds in the two point cloud frames in the same coordinate system, which can also be understood as merging the point clouds in the two point cloud frames.

[0088] In some embodiments, the point cloud processing device may first calculate the target distance, which corresponds to the device's driving speed and the time difference between two point cloud frames (the first point cloud frame and the second point cloud frame). Specifically, the target distance is equal to the device's driving speed multiplied by the time difference between the two point cloud frames.

[0089] After calculating the target distance, the point cloud processing device can move the first coordinate system corresponding to the first point cloud frame by the target distance according to the direction of travel of the device to obtain the second coordinate system corresponding to the first point cloud frame, and then superimpose the first point cloud in the first point cloud frame onto the second point cloud frame according to the second coordinate system corresponding to the first point cloud frame.

[0090] Specifically, the process by which the point cloud processing device superimposes the first point cloud from the first point cloud frame onto the second point cloud frame includes:

[0091] (1) Determine the second correspondence between the first point cloud of the first point cloud frame and the second point cloud of the second point cloud frame based on the first correspondence between the second coordinate system and the original coordinate system of the second point cloud frame;

[0092] Specifically, determining the correspondence between points in a coordinate system based on the correspondence between coordinate systems is a common technique used by those skilled in the art, and will not be elaborated here.

[0093] (2) Determine the coordinate values ​​of the first point cloud in the first point cloud frame in the original coordinate system of the second point cloud frame according to the second correspondence relationship;

[0094] After determining the second correspondence, the point cloud processing device can transform the coordinate values ​​of the first point cloud in the second coordinate system through the second correspondence. The transformed coordinate values ​​are the coordinate values ​​of the first point cloud in the original coordinate system of the second point cloud frame.

[0095] (3) Based on the coordinate values ​​of the second point cloud in the original coordinate system of the second point cloud frame and the coordinate values ​​of the first point cloud in the original coordinate system of the second point cloud frame, display the first point cloud and the second point cloud in the original coordinate system of the second point cloud frame.

[0096] After determining the coordinate values ​​of the first point cloud and the second point cloud in the original coordinate system of the second point cloud frame, the point cloud processing device can display the first point cloud and the second point cloud in the original coordinate system of the second point cloud frame, thereby merging the first point cloud and the second point cloud and superimposing the first point cloud onto the second point cloud frame.

[0097] It should be understood that the device in this embodiment can be a vehicle or other equipment that needs to identify obstacles in the surrounding environment; this embodiment does not limit the specific device. It should also be understood that the time difference between two frames of point clouds is related to the scanning frequency of the scanning device; the specific calculation process is common knowledge in the art and will not be described in detail in this embodiment.

[0098] It should be understood that, in addition to the above methods, after determining the first coordinate system, the point cloud processing device can also superimpose the first point cloud in the first point cloud frame onto the second point cloud frame in the following way: the point cloud processing device determines the fourth relationship between the first point cloud in the first point cloud frame and the second point cloud in the second point cloud frame based on the third correspondence between the first coordinate system and the original coordinate system of the second point cloud frame; it determines the coordinate values ​​corresponding to the first point cloud in the first point cloud frame in the original coordinate system of the second point cloud frame based on the fourth correspondence; it displays the first point cloud and the second point cloud in the original coordinate system of the second point cloud frame based on the coordinate values ​​corresponding to the second point cloud in the original coordinate system of the second point cloud frame and the coordinate values ​​corresponding to the first point cloud in the original coordinate system of the second point cloud frame; it moves the first point cloud in the original coordinate system of the second point cloud by a target distance according to the travel direction of the device to obtain a target image, which is the image obtained after superimposing the first point cloud onto the second point cloud frame.

[0099] The point cloud processing device can also superimpose the first point cloud in the first point cloud frame onto the second point cloud frame in other ways, but this embodiment does not limit the specific method.

[0100] For example, the first point cloud frame contains a first point cloud A(x) A y A , z A ) and B(x B y B , z B The original coordinate system corresponding to the first point cloud frame is O-XYZ, and the second point cloud frame contains the second point cloud C(x). C y C , z C ) and D(x D y D , z D The original coordinate system corresponding to the second point cloud frame is O1-X1Y1Z1, as shown below. Figure 2 As shown.

[0101] The first transformation matrix is ​​used to rotate the coordinate system clockwise / counterclockwise by an angle θ1 and then translate it by a vector T1. The point cloud processing device rotates the coordinate system O-XYZ by an angle θ1 according to the first transformation matrix, and then translates the coordinate system by a vector T1 to obtain the first coordinate system O2-X2Y2Z2, such as... Figure 3 As shown.

[0102] When the device acquires the first and second point cloud frames, it travels forward at a speed v. The time difference between acquiring the first and second point cloud frames is t. After the point cloud processing device determines the first coordinate system, it calculates the target distance as S = vt. Then, the point cloud processing device moves the first coordinate system along the positive X-axis by s to obtain the second coordinate system O3-X3Y3Z3. Figure 4 As shown.

[0103] Based on the correspondence between coordinate system O1-X1Y1Z1 and coordinate system O3-X3Y3Z3, the first point cloud A(x A y A , z A ) and the second B(x) B y B , z B Perform a coordinate transformation to obtain the coordinate values ​​(x, y, z) of A in the coordinate system O1-X1Y1Z1. A ',y A ',z A The coordinates (x, y) of B in coordinate system O1-X1Y1Z1 are given by the coordinates of B. B ',y B ',z B Finally, the first point clouds A and B, and the second point clouds C and D are displayed in the coordinate system O1-X1Y1Z1, as shown below. Figure 5 As shown.

[0104] The technical solution provided in this application can determine the transformation matrix corresponding to two adjacent point cloud frames using a particle swarm optimization algorithm. The original coordinate system in the first point cloud frame is rotated and / or translated using this transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame. Based on the first coordinate system, the device's travel speed, and the time difference between the first and second point cloud frames, the first point cloud in the first frame is superimposed onto the second point cloud frame. In other words, this technical solution can register two point cloud frames, allowing the points in the two frames to be superimposed, thereby increasing the density of the point cloud. This means the density of the point cloud corresponding to distant obstacles will also increase, thus improving the scanning device's ability to detect distant obstacles.

[0105] Secondly, in this embodiment, when performing calculations using the particle swarm optimization algorithm, the optimization function values ​​corresponding to each particle (target transformation matrix) can be calculated in parallel using the GPU, thereby improving computational efficiency.

[0106] Furthermore, this embodiment can divide the point cloud frame into several regions, and then calculate the curvature of the point cloud using the least squares method, thereby improving the accuracy of the solution.

[0107] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a point cloud processing device, an electronic device, and corresponding embodiments.

[0108] Figure 6 This is a schematic diagram of the point cloud processing device shown in the embodiments of this application.

[0109] See Figure 6 The point cloud processing device in this embodiment includes:

[0110] The determination module 601 is used to determine the first transformation matrix corresponding to adjacent first point cloud frames and second point cloud frames through the particle swarm optimization algorithm.

[0111] Transformation module 602 is used to rotate and / or translate the original coordinate system of the first point cloud frame through a first transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame;

[0112] The overlay module 603 is used to overlay the first point cloud in the first point cloud frame onto the second point cloud frame based on the first coordinate system, the device's driving speed, and the time difference between the first point cloud frame and the second point cloud frame.

[0113] In the technical solution provided in this application, the determining module 601 can determine the transformation matrix corresponding to two adjacent point cloud frames using a particle swarm optimization algorithm. The transformation module 602 rotates and / or translates the original coordinate system in the first point cloud frame using the transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame. The overlay module 603 can overlay the first point cloud in the first point cloud frame onto the second point cloud frame based on the first coordinate system, the device's travel speed, and the time difference between the first and second point cloud frames. In other words, this technical solution can register two point cloud frames, allowing the points in the two point cloud frames to be overlaid, thereby increasing the density of the point cloud. That is, the density of the point cloud corresponding to distant obstacles will also increase, thus improving the scanning device's ability to detect distant obstacles.

[0114] For ease of understanding, the point cloud processing device in the embodiments of this application will be described in detail below. Please refer to [link / reference]. Figure 7 The point cloud processing apparatus in this application embodiment includes:

[0115] The determination module 701 is used to determine the first transformation matrix corresponding to the adjacent first point cloud frame and second point cloud frame through the particle swarm optimization algorithm.

[0116] Transformation module 702 is used to rotate and / or translate the original coordinate system of the first point cloud frame through a first transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame;

[0117] The overlay module 703 is used to overlay the first point cloud in the first point cloud frame onto the second point cloud frame based on the first coordinate system, the device's driving speed, and the time difference between the first point cloud frame and the second point cloud frame.

[0118] The overlay module 703 includes:

[0119] The moving unit 7031 is used to move the first coordinate system corresponding to the first point cloud frame by a target distance according to the driving direction of the device to obtain the second coordinate system corresponding to the first point cloud frame. The target distance corresponds to the driving speed of the device and the time difference between the first point cloud frame and the second point cloud frame.

[0120] The first determining unit 7032 is used to determine the second correspondence between the first point cloud of the first point cloud frame and the second point cloud of the second point cloud frame based on the first correspondence between the second coordinate system and the original coordinate system of the second point cloud frame.

[0121] The second determining unit 7033 is used to determine the coordinate values ​​of the first point cloud in the original coordinate system of the second point cloud frame according to the second correspondence relationship;

[0122] Display unit 7034 is used to display the first point cloud and the second point cloud in the original coordinate system of the second point cloud frame according to the coordinate values ​​of the second point cloud in the original coordinate system of the second point cloud frame and the coordinate values ​​of the first point cloud in the original coordinate system of the second point cloud frame.

[0123] The determination module 701 includes:

[0124] The calculation unit 7011 is used to calculate the curvature of the first point cloud in the first point cloud frame and the second point cloud in the second point cloud frame;

[0125] The pairing unit 7012 is used to pair the first point cloud in the first point cloud frame with the second point cloud in the second point cloud frame according to the curvature, so as to obtain several pairs of point clouds, each pair of point clouds containing point clouds with similar curvature.

[0126] The generation unit 7013 is used to generate n target transformation matrices, where n is an integer greater than or equal to 1;

[0127] The third determining unit 7014 is used to determine the optimization function value corresponding to each target transformation matrix, and the optimization function value corresponds to the sum of the normal difference products of several pairs of point cloud pairs;

[0128] The fourth determining unit 7015 is used to determine the minimum value among the optimization function values;

[0129] The fifth determining unit 7016 is used to determine the target transformation matrix corresponding to the minimum value as the first transformation matrix when the minimum value is zero;

[0130] The sixth determining unit 7017 is used to determine the target transformation matrix corresponding to the minimum value as the transformation matrix to be optimized when the minimum value is not zero;

[0131] The judgment unit 7018 is used to determine whether the number of iterations has reached a preset value;

[0132] The seventh determining unit 7019 is used to determine the first transformation matrix as the one with the smallest corresponding optimization function value among the transformation matrices to be optimized calculated in each iteration when the number of iterations reaches a preset value.

[0133] Update unit 70110 is used to update n target transformation matrices when the number of iterations has not reached a preset value.

[0134] Specifically, the third determining unit 7014 is also used to compute the optimization function value corresponding to each target transformation matrix in parallel using the GPU.

[0135] In some embodiments, the third determining unit includes:

[0136] The transformation sub-unit is used to rotate and / or translate the original coordinate system of the first point cloud frame through the target transformation matrix to obtain the third coordinate system corresponding to the first point cloud frame;

[0137] The first calculation subunit is used to calculate the first normal vector corresponding to the first point cloud in the first point cloud frame whose corresponding coordinate system is the third coordinate system.

[0138] The second computational subunit is used to calculate the cross product of the first normal vector corresponding to the first point cloud in a pair of point cloud pairs and the second normal vector corresponding to the second point cloud in a pair of point cloud pairs, and to add the cross products of the normal vectors of each pair of point cloud pairs to obtain the optimization function value corresponding to the target transformation matrix.

[0139] The computing unit includes:

[0140] The sub-unit is used to divide the first point cloud frame into several first regions and the second point cloud frame into several second regions. Each first region contains one or more adjacent first point clouds and each second region contains one or more adjacent second point clouds.

[0141] Establish sub-units to create a coordinate system for each of the first and second regions respectively;

[0142] The first determining sub-unit is used to determine, for each first region, the first coordinate value of the first point cloud in the first region in the coordinate system of the first region, determine the first surface corresponding to the first region by the least squares method based on the first coordinate value, and determine the curvature of the first surface as the curvature of the first point cloud in the first region.

[0143] The second determining sub-unit is used to determine, for each second region, the second coordinate value corresponding to the second point cloud in the coordinate system of the second region, determine the second surface corresponding to the second region by least squares method based on the second coordinate value, and determine the curvature of the first surface as the curvature corresponding to the second point cloud in the second region.

[0144] In the technical solution provided in this application, the determining module 701 can determine the transformation matrix corresponding to two adjacent point cloud frames using a particle swarm optimization algorithm. The transformation module 702 rotates and / or translates the original coordinate system in the first point cloud frame using the transformation matrix to obtain the first coordinate system corresponding to the first point cloud frame. The overlay module 703 can overlay the first point cloud in the first point cloud frame onto the second point cloud frame based on the first coordinate system, the device's travel speed, and the time difference between the first and second point cloud frames. In other words, this technical solution can register two point cloud frames, allowing the points in the two point cloud frames to be overlaid, thereby increasing the density of the point cloud. That is, the density of the point cloud corresponding to distant obstacles will also increase, thus improving the scanning device's ability to detect distant obstacles.

[0145] Secondly, in this embodiment, when performing calculations using the particle swarm optimization algorithm, the third determining unit 7014 can use the GPU to calculate the optimization function values ​​corresponding to each particle (target transformation matrix) in parallel, thereby improving computational efficiency.

[0146] Furthermore, in this embodiment, the sub-unit division can divide the point cloud frame into several regions. Then, the first and second determining sub-units can calculate the curvature of the point cloud using the least squares method, which improves the accuracy of the scheme.

[0147] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0148] Figure 8 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0149] See Figure 8 The electronic device 800 includes a memory 810 and a processor 820.

[0150] The processor 820 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0151] Memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 820 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 810 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, a high-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0152] The memory 810 stores executable code, which, when processed by the processor 820, can cause the processor 820 to execute part or all of the methods described above.

[0153] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0154] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0155] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of point cloud processing, the method comprising: The method comprises the following steps: When it is detected that there is an obstacle in the surrounding environment of the vehicle, a first point cloud frame is returned, and a second point cloud frame adjacent to the first point cloud frame is searched; A first transformation matrix corresponding to the adjacent first point cloud frame and the second point cloud frame is determined by using a particle swarm optimization algorithm; The original coordinate system of the first point cloud frame is rotated and / or translated by using the first transformation matrix to obtain a first coordinate system corresponding to the first point cloud frame; First point clouds in the first point cloud frame are superimposed into the second point cloud frame according to the first coordinate system, the driving speed of the device, and the time difference between the first point cloud frame and the second point cloud frame, wherein the following steps are included: the driving speed of the device is multiplied by the time difference to calculate a target distance; the first coordinate system is moved by the target distance in the driving direction of the device to obtain a second coordinate system corresponding to the first point cloud frame; and the first point clouds in the first point cloud frame are superimposed into the second point cloud frame according to the second coordinate system. The first transformation matrix corresponding to the adjacent first point cloud frame and the second point cloud frame is determined by using the particle swarm optimization algorithm, which comprises the following steps: (1) the curvatures of the first point clouds in the first point cloud frame and the second point clouds in the second point cloud frame are calculated, wherein the following steps are included: the first point cloud frame is divided into a plurality of first regions, and the second point cloud frame is divided into a plurality of second regions, each first region contains one or more adjacent first point clouds, and each second region contains one or more adjacent second point clouds; a coordinate system is established for each first region and second region, respectively; for each first region, a first coordinate value corresponding to the first point cloud in the first region in the coordinate system of the first region is determined, a first surface corresponding to the first region is determined by using the least square method according to the first coordinate value, and the curvature of the first surface is determined as the curvature corresponding to the first point cloud in the first region; for each second region, a second coordinate value corresponding to the second point cloud in the second region in the coordinate system of the second region is determined, a second surface corresponding to the second region is determined by using the least square method according to the second coordinate value, and the curvature of the first surface is determined as the curvature corresponding to the second point cloud in the second region; (2) the first point clouds in the first point cloud frame and the second point clouds in the second point cloud frame are paired according to the curvatures to obtain a plurality of groups of point cloud pairs, each group of point cloud pairs contains point clouds with similar curvatures; (3) n target transformation matrices are generated, wherein n is an integer greater than or equal to 1; (4) for each target transformation matrix, the original coordinate system of the first point cloud frame is rotated and / or translated by using the target transformation matrix to obtain a third coordinate system corresponding to the first point cloud frame; a first normal vector corresponding to the first point cloud in the first point cloud frame with the third coordinate system as the corresponding coordinate system is calculated; for the plurality of groups of point cloud pairs, a normal vector cross product of a first normal vector corresponding to the first point cloud in the group of point cloud pairs and a second normal vector corresponding to the second point cloud in the group of point cloud pairs is calculated, and the normal vector cross products of each group of point cloud pairs are added to obtain an optimization function value corresponding to the target transformation matrix, wherein the optimization function value corresponds to the normal vector cross products of the plurality of groups of point cloud pairs.

2. The method of claim 1, wherein, The superimposing the first point cloud in the first point cloud frame into the second point cloud frame according to the second coordinate system comprises: determining a second correspondence relationship between the first point cloud in the first point cloud frame and the second point cloud in the second point cloud frame according to a first correspondence relationship between the second coordinate system and an original coordinate system of the second point cloud frame; determining a coordinate value corresponding to the first point cloud in the original coordinate system of the second point cloud frame according to the second correspondence relationship; displaying the first point cloud and the second point cloud in the original coordinate system of the second point cloud frame according to the coordinate value corresponding to the second point cloud in the original coordinate system of the second point cloud frame and the coordinate value corresponding to the first point cloud in the original coordinate system of the second point cloud frame.

3. The method of claim 1, wherein, After the calculating the curvatures of the first point cloud in the first point cloud frame and the second point cloud in the second point cloud frame, further comprising: (5) determining the minimum value in the optimization function value; (6) if the minimum value is zero, determining that the target transformation matrix corresponding to the minimum value is the first transformation matrix; (7) if the minimum value is not zero, determining that the target transformation matrix corresponding to the minimum value is the to-be-optimized transformation matrix; (8) judging whether the number of iterations reaches a preset value; (9) if yes, determining that the to-be-optimized transformation matrix corresponding to the minimum optimization function value in each iteration is the first transformation matrix; (10) if no, updating the n target transformation matrices and performing steps (4) to (10).

4. A point cloud processing apparatus, characterized by comprising: comprising: a determining module configured to return a first point cloud frame when detecting that there is an obstacle in the surrounding environment of the vehicle, and search for a second point cloud frame adjacent to the first point cloud frame; and determine a first transformation matrix corresponding to the adjacent first point cloud frame and second point cloud frame through a particle swarm optimization algorithm; a transformation module configured to rotate and / or translate the original coordinate system of the first point cloud frame through the first transformation matrix to obtain a first coordinate system corresponding to the first point cloud frame; a superimposing module configured to superimpose a first point cloud in the first point cloud frame into a second point cloud frame according to the first coordinate system, the driving speed of the device, and the time difference between the first point cloud frame and the second point cloud frame, wherein the superimposing module comprises: multiplying the driving speed of the device by the time difference to calculate a target distance; moving the first coordinate system in the driving direction of the device by the target distance to obtain a second coordinate system corresponding to the first point cloud frame; and superimposing the first point cloud in the first point cloud frame into the second point cloud frame according to the second coordinate system. The determining module comprises a calculation unit configured to calculate the curvatures of the first point cloud in the first point cloud frame and the second point cloud in the second point cloud frame. The calculation unit comprises: a dividing sub-unit configured to divide the first point cloud frame into a plurality of first regions, and divide the second point cloud frame into a plurality of second regions, each first region containing one or more adjacent first point clouds, and each second region containing one or more adjacent second point clouds; an establishing sub-unit configured to establish a coordinate system for each first region and second region, respectively. The first determining subunit is configured to determine, for each first region, a first coordinate value corresponding to a first point cloud in the first region in a coordinate system of the first region, determine a first surface corresponding to the first region by least square method according to the first coordinate value, and determine a curvature of the first surface as a curvature corresponding to the first point cloud in the first region. The second determining subunit is configured to determine, for each second region, a second coordinate value corresponding to a second point cloud in the second region in a coordinate system of the second region, determine a second surface corresponding to the second region by least square method according to the second coordinate value, and determine a curvature of the first surface as a curvature corresponding to the second point cloud in the second region. The pairing unit is configured to pair the first point cloud in the first point cloud frame with the second point cloud in the second point cloud frame according to the curvatures to obtain a plurality of groups of point cloud pairs, each group of point cloud pairs including point clouds with similar curvatures. The generating unit is configured to generate n target transformation matrices, where n is an integer greater than or equal to 1. The third determining unit is configured to, for each target transformation matrix, rotate and / or translate an original coordinate system of the first point cloud frame by the target transformation matrix to obtain a third coordinate system corresponding to the first point cloud frame. The first normal vector corresponding to the first point cloud in the first point cloud frame with the third coordinate system is calculated. For the plurality of groups of point cloud pairs, the vector product of the first normal vector corresponding to the first point cloud in the group of point cloud pairs and the second normal vector corresponding to the second point cloud in the group of point cloud pairs is calculated, and the vector products of each group of point cloud pairs are added to obtain an optimization function value corresponding to the target transformation matrix.

5. The apparatus of claim 4, wherein, The superimposition module includes: The moving unit is configured to move the first coordinate system corresponding to the first point cloud frame by a target distance in the driving direction of the device to obtain a second coordinate system corresponding to the first point cloud frame, where the target distance corresponds to the driving speed of the device and the time difference between the first point cloud frame and the second point cloud frame. The first determining unit is configured to determine a second correspondence relationship between the first point cloud of the first point cloud frame and the second point cloud of the second point cloud frame according to a first correspondence relationship between the second coordinate system and an original coordinate system of the second point cloud frame. The second determining unit is configured to determine a coordinate value corresponding to the first point cloud in the original coordinate system of the second point cloud frame according to the second correspondence relationship. The display unit is configured to display the first point cloud and the second point cloud in the original coordinate system of the second point cloud frame according to the coordinate value corresponding to the second point cloud in the original coordinate system of the second point cloud frame and the coordinate value corresponding to the first point cloud in the original coordinate system of the second point cloud frame.

6. The apparatus of claim 4, wherein, The determination module further includes: The fourth determining unit is configured to determine a minimum value in the optimization function value. The fifth determining unit is configured to determine that the target transformation matrix corresponding to the minimum value is a first transformation matrix when the minimum value is zero. The sixth determining unit is configured to determine that the target transformation matrix corresponding to the minimum value is a to-be-optimized transformation matrix when the minimum value is not zero. The judging unit is configured to determine whether the number of iterations reaches a preset value. A seventh determination unit is configured to determine, when the iteration number reaches a preset value, that a to-be-optimized transformation matrix corresponding to the minimum optimization function value in each iteration calculation is a first transformation matrix. An updating unit is configured to update the n target transformation matrices when the iteration number does not reach the preset value.

7. An electronic device, comprising: Comprise: a processor; and a memory having stored executable codes, which, when executed by the processor, cause the processor to perform the method of any one of claims 1-3.

8. A computer-readable storage medium having stored executable codes, which, when executed by a processor of an electronic device, cause the processor to perform the method of any one of claims 1-3.

Images