Point cloud motion compensation method and device, electronic equipment and storage medium

Abstract

The invention discloses a point cloud motion compensation method and device, electronic equipment and a storage medium, and relates to the technical field of intelligent driving. The method comprises the following steps: receiving a point cloud information frame at a first frequency, and receiving an initial positioning information frame at a second frequency; performing timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence; partitioning the point cloud information frame to obtain a target number of point cloud information blocks; performing interpolation processing on the positioning information frame screening sequence to obtain a target number of target positioning information frames; performing timestamp alignment on each point cloud information block and each target positioning information frame to obtain a target matching positioning timestamp of each point cloud information block; according to each target matching positioning timestamp, determining a conversion relation between each self-vehicle and the world; and performing motion compensation on the point cloud information block according to the block self-vehicle-to-world conversion relation. According to the invention, motion compensation can be carried out on the point cloud information, and motion distortion removal is realized.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a point cloud motion compensation method and device, electronic equipment, and storage medium. Background Art

[0002] In the automotive field, LiDAR can be used to collect point cloud information to achieve functions such as target detection. The quality of the point cloud obtained by LiDAR can directly affect the safety and accuracy of assisted / autonomous driving. The LiDAR point cloud measures the distance between the object and the radar. Since the points in the same frame of the point cloud are not collected at the same time, in the real world, vehicles equipped with LiDAR sensors are often moving, so the LiDAR coordinate systems of different laser points are different, which will cause the collected point cloud information to often have certain motion distortion. In order to obtain high-quality laser perception data, it is necessary to perform motion compensation on the point cloud information to achieve motion distortion removal. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a point cloud motion compensation method and device, electronic device, and storage medium, which can perform motion compensation on point cloud information and achieve motion distortion removal.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a point cloud motion compensation method, the method comprising:

[0005] Receiving point cloud information frames at a first frequency, and receiving initial positioning information frames at a second frequency; wherein the second frequency is equal to N*the first frequency, and N is greater than 1;

[0006] Determining a point cloud timestamp interval according to the timestamp of the point cloud information frame;

[0007] Performing timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence;

[0008] Dividing the point cloud information frame into blocks to obtain a target number of point cloud information blocks;

[0009] Performing interpolation processing on the initial positioning information frames in the positioning information frame screening sequence to obtain the target number of target positioning information frames;

[0010] Performing timestamp alignment on each of the point cloud information blocks and each of the target positioning information frames to obtain a target matching positioning timestamp for each of the point cloud information blocks;

[0011] Determine each block of self-vehicle-to-world transformation relationship according to each target matching positioning timestamp, wherein the block of self-vehicle-to-world transformation relationship is used to characterize the transformation relationship from the self-vehicle coordinate system to the world coordinate system of the point cloud information block;

[0012] Motion compensation is performed on each of the point cloud information blocks according to a vehicle-to-world transformation relationship of each of the blocks.

[0013] Optionally, the point cloud timestamp interval includes a point cloud timestamp lower limit value and a point cloud timestamp upper limit value;

[0014] The step of performing timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence includes:

[0015] According to the lower limit value of the point cloud timestamp, a lower limit value of the positioning timestamp is determined from the timestamps of the multiple initial positioning information frames, and according to the upper limit value of the point cloud timestamp, an upper limit value of the positioning timestamp is determined from the timestamps of the multiple initial positioning information frames;

[0016] Determine a positioning timestamp interval according to the positioning timestamp lower limit value and the positioning timestamp upper limit value;

[0017] The positioning information frame screening sequence is obtained by merging all the initial positioning information frames located in the positioning timestamp interval.

[0018] Optionally, the determining a positioning timestamp lower limit value from timestamps of a plurality of the initial positioning information frames according to the point cloud timestamp lower limit value, and the determining a positioning timestamp upper limit value from timestamps of a plurality of the initial positioning information frames according to the point cloud timestamp upper limit value, comprises:

[0019] If the lower limit value of the point cloud timestamp is greater than or equal to the timestamp of each of the initial positioning information frames, a difference calculation is performed based on the lower limit value of the point cloud timestamp and the timestamp of the initial positioning information frame to obtain a lower limit difference value;

[0020] Determine the timestamp of the initial positioning information frame corresponding to the smallest lower bound difference as the positioning information timestamp lower bound value;

[0021] If the upper limit value of the point cloud timestamp is less than or equal to the timestamp of each of the initial positioning information frames, performing a difference calculation based on the timestamp of the initial positioning information frame and the upper limit value of the point cloud timestamp to obtain an upper limit difference value;

[0022] The timestamp of the initial positioning information frame corresponding to the smallest upper bound difference is determined as the upper bound value of the positioning information timestamp.

[0023] Optionally, after performing timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence, the method further includes:

[0024] Counting the number of initial positioning information frames in the positioning information frame screening sequence to obtain the number of frames;

[0025] If the number of frames is less than or equal to a preset number threshold, jump to the next frame of the point cloud information frame and enter the step of determining the point cloud timestamp interval based on the timestamp of the point cloud information frame; wherein the number threshold is less than or equal to N.

[0026] Optionally, the point cloud information block includes at least two point clouds;

[0027] The step of aligning the timestamps of each point cloud information block and each target positioning information frame to obtain the target matching positioning timestamp of each point cloud information block includes:

[0028] Determine a timestamp median according to the timestamps of all the point clouds of each point cloud information block;

[0029] According to the median of the timestamp, the timestamps of the target number of the positioning information frames are matched most closely to obtain the target matching positioning timestamp.

[0030] Optionally, performing motion compensation on each of the point cloud information blocks according to the vehicle-to-world transformation relationship of each of the blocks includes:

[0031] Using a preset laser radar to ego vehicle conversion relationship, coordinate conversion is performed on the initial laser radar coordinates of the point cloud information block to obtain a first ego vehicle coordinate;

[0032] The first self-vehicle coordinates are transformed by utilizing the block self-vehicle-to-world transformation relationship to obtain world coordinates; wherein the world coordinates of all the point cloud information blocks share the same world coordinate system.

[0033] Optionally, after performing coordinate transformation on the first ego-vehicle coordinates by using the block ego-vehicle-to-world transformation relationship to obtain world coordinates, performing motion compensation on each of the point cloud information blocks according to each of the block ego-vehicle-to-world transformation relationships further includes:

[0034] Determine a target point cloud information block from all the point cloud information blocks;

[0035] Using the block self-vehicle-to-world transformation relationship of the target point cloud information block, coordinate transformation is performed on the world coordinates of each of the point cloud information blocks to obtain a second self-vehicle coordinate;

[0036] The second self-vehicle coordinates are transformed by utilizing the laser radar to self-vehicle conversion relationship to obtain target laser radar coordinates; wherein the target laser radar coordinates of all the point cloud information blocks share the laser radar coordinate system corresponding to the target point cloud information block.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a point cloud motion compensation device, the device comprising:

[0038] An information receiving module, used to receive point cloud information frames at a first frequency and receive initial positioning information frames at a second frequency; wherein the second frequency is equal to N*the first frequency, and N is greater than 1;

[0039] An interval determination module, used to determine a point cloud timestamp interval according to the timestamp of the point cloud information frame;

[0040] A first alignment module, configured to perform timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence;

[0041] An information segmentation module, used to segment the point cloud information frame into blocks to obtain a target number of point cloud information blocks;

[0042] An information interpolation module, used for performing interpolation processing on the initial positioning information frames in the positioning information frame screening sequence to obtain the target number of target positioning information frames;

[0043] A second alignment module is used to align the timestamps of each point cloud information block and each target positioning information frame to obtain the target matching positioning timestamp of each point cloud information block;

[0044] A conversion determination module, used to determine each block of ego-vehicle-to-world conversion relationship according to each target matching positioning timestamp, wherein the block ego-vehicle-to-world conversion relationship is used to represent the conversion relationship from the ego-vehicle coordinate system to the world coordinate system of the point cloud information block;

[0045] The motion compensation module is used to perform motion compensation on each of the point cloud information blocks according to the vehicle-to-world transformation relationship of each of the blocks.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0048] The point cloud motion compensation method and device, electronic device, and storage medium proposed in the present application receive point cloud information frames at a first frequency and receive initial positioning information frames at a second frequency, respectively, and the second frequency is equal to N*first frequency, so that each time a point cloud information frame is received, a maximum of N initial positioning information frames will be received. Then the first timestamp alignment is performed, and the timestamp of the obtained positioning information frame screening sequence will include the timestamp of the point cloud information frame. Then the point cloud information frame is divided into blocks and the initial positioning information frame in the positioning information frame screening sequence is interpolated to obtain the target number of point cloud information blocks and the target number of target positioning information frames. Then the second timestamp alignment is performed to obtain the target matching positioning timestamp of each point cloud information block. The target matching timestamp indicates the timestamp of the target positioning information frame corresponding to the point cloud information block, so that the block self-vehicle to world conversion relationship can be determined based on the target matching positioning timestamp. Finally, motion compensation is performed on each point cloud information block according to each block self-vehicle to world conversion relationship. In summary, this embodiment can perform motion compensation on point cloud information and achieve motion distortion removal.

[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a point cloud motion compensation method provided in an embodiment of the present application;

[0051] Figure 2 It is a schematic diagram of a point cloud timestamp interval and a positioning timestamp interval provided in an embodiment of the present application;

[0052] Figure 3 yes Figure 1 Flow chart of step 103;

[0053] Figure 4 is a flow chart of a point cloud motion compensation method provided by another embodiment of the present application;

[0054] Figure 5 yes Figure 1 Flowchart of step 106;

[0055] Figure 6 yes Figure 1 Flowchart of step 108;

[0056] Figure 7yes Figure 1 Another flow chart of step 108;

[0057] Figure 8 It is a structural schematic diagram of a point cloud motion compensation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0062] LiDAR: It is a radar system that uses laser beams to detect the position, speed and other characteristic quantities of a target. Its working principle is to transmit a detection signal (laser beam) to the target, and then compare the received signal reflected from the target (called the target echo, hereinafter referred to as point cloud information) with the transmitted signal. After appropriate processing, relevant information about the target can be obtained, such as target distance, direction, height, speed, attitude, and even shape parameters, so as to detect, track and identify the target (such as an aircraft). LiDAR is generally composed of a laser transmitter, an optical receiver, a turntable and an information processing system. The laser converts electrical pulses into light pulses and transmits them. The optical receiver then converts the light pulses reflected from the target into electrical pulses and sends them to the display.

[0063] Point cloud information frame: contains point cloud information of multiple point clouds collected at different times.

[0064] Positioning information: refers to the position of the vehicle equipped with LiDAR. Since the vehicle is moving, the positioning information is changing.

[0065] The point cloud motion compensation method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the point cloud motion compensation method, etc., but is not limited to the above forms.

[0066] The point cloud motion compensation method, point cloud motion compensation device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the point cloud motion compensation method in the embodiments of the present application is described.

[0067] Please refer to Figure 1 , Figure 1 An optional flow chart of a point cloud motion compensation method is disclosed. Figure 1 The method may include but is not limited to steps 101 to 108.

[0068] Step 101, receiving a point cloud information frame at a first frequency, and receiving an initial positioning information frame at a second frequency; wherein the second frequency is equal to N*the first frequency, and N is greater than 1;

[0069] Step 102, determining the point cloud timestamp interval according to the timestamp of the point cloud information frame;

[0070] Step 103, performing timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence;

[0071] Step 104, dividing the point cloud information frame into blocks to obtain a target number of point cloud information blocks;

[0072] Step 105, performing interpolation processing on the initial positioning information frames in the positioning information frame screening sequence to obtain a target number of target positioning information frames;

[0073] Step 106, aligning the timestamps of each point cloud information block and each target positioning information frame to obtain the target matching positioning timestamp of each point cloud information block;

[0074] Step 107, determining each block of ego-vehicle-to-world transformation relationship according to each target matching positioning timestamp, where the block ego-vehicle-to-world transformation relationship is used to characterize the transformation relationship from the ego-vehicle coordinate system to the world coordinate system of the point cloud information block;

[0075] Step 108 , motion compensation is performed on each point cloud information block according to each vehicle-to-world transformation relationship.

[0076] In the steps 101 to 108 shown in the embodiment of the present application, each time a point cloud information frame is received, a maximum of N initial positioning information frames will be received. The first timestamp alignment is performed first, and the timestamp of the obtained positioning information frame screening sequence will include the timestamp of the point cloud information frame. Then the point cloud information frame is divided into blocks and the initial positioning information frame in the positioning information frame screening sequence is interpolated to obtain the target number of point cloud information blocks and the target number of target positioning information frames. Then the second timestamp alignment is performed to obtain the target matching positioning timestamp of each point cloud information block. The target matching positioning timestamp indicates the timestamp of the target positioning information frame corresponding to the point cloud information block. Since the target positioning information frame can indicate the position of the vehicle equipped with a laser radar, the position of the vehicle equipped with a laser radar of the target positioning information frame corresponding to the point cloud information block can be determined based on the target matching positioning timestamp, and then the self-vehicle-to-world conversion relationship of the block is determined. Finally, motion compensation is performed on each point cloud information block according to each self-vehicle-to-world conversion relationship. In summary, this embodiment can perform motion compensation on the point cloud information and achieve motion distortion removal.

[0077] In step 101, point cloud information frames are received at a first frequency, and initial positioning information frames are received at a second frequency. The first frequency refers to the laser radar frequency, also known as the frequency for receiving point cloud information. The second frequency refers to the GPS positioning information frequency, also known as the frequency for receiving positioning information. The second frequency is equal to N*first frequency, where N is greater than 1. For example, if the first frequency is 10 Hz (Hertz), then the second frequency is 100 Hz, and N is 10 at this time. That is to say, the laser radar can receive approximately 10 positioning information frames within one frame time. For another example, if the first frequency is 10 Hz, then the second frequency is 105 Hz, and N is 10.5 at this time. That is to say, the laser radar can receive approximately 10 positioning information frames within one frame time.

[0078] In one embodiment, step 101 includes: creating a point cloud information receiving queue and a positioning information receiving queue; receiving a point cloud information frame at a first frequency and placing the point cloud information frame into the point cloud information receiving queue; receiving an initial positioning information frame at a second frequency and placing the initial positioning information frame into the positioning information receiving queue. This embodiment stores point cloud information and positioning information in queues, and can receive point cloud information and positioning information in real time.

[0079] In step 102, a point cloud timestamp interval is determined based on the timestamp of the point cloud information frame. The point cloud timestamp interval refers to the timestamp distribution of a point cloud information frame. Figure 2 , the point cloud timestamp interval contains the point cloud timestamp lower limit value t d1and the upper limit of the point cloud timestamp t d2 . Figure 2 The lower limit of the positioning timestamp t is also shown. d3 and the upper limit of the positioning timestamp t d4 , which will be described in detail below. The timestamp corresponding to the point cloud information frame can be used as the upper limit of the point cloud timestamp, and the timestamp corresponding to the previous frame of the point cloud information frame can be used as the lower limit of the point cloud timestamp. For example, the first frequency is 10Hz, and the point cloud information receiving queue stores 3 point cloud information frames, and the point cloud timestamp intervals are [0,0.1], [0.1,0.2], [0.2,0.3] respectively.

[0080] It should be noted that the units of the timestamps involved in the entire application can be milliseconds, seconds, etc., and can be set according to actual conditions.

[0081] In step 103, timestamp alignment is performed based on the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence. In order to ensure that the point cloud information matches the positioning information, the point cloud information and the positioning information need to be aligned in terms of timestamps to obtain a positioning information frame screening sequence. This timestamp alignment can be called a coarse timestamp alignment.

[0082] In one embodiment, referring to Figure 3 , step 103 comprises:

[0083] Step 301, determining a positioning timestamp lower limit value from timestamps of a plurality of initial positioning information frames according to a point cloud timestamp lower limit value, and determining a positioning timestamp upper limit value from timestamps of a plurality of initial positioning information frames according to an point cloud timestamp upper limit value;

[0084] Step 302, determining a positioning timestamp interval according to a positioning timestamp lower limit value and a positioning timestamp upper limit value;

[0085] Step 303: All initial positioning information frames located in the positioning timestamp interval are merged to obtain a positioning information frame screening sequence.

[0086] For example, refer to Figure 2 , according to the lower limit value t of the point cloud timestamp d1 , determine the lower limit value t of the positioning timestamp from the timestamps of multiple initial positioning information frames in the positioning information receiving queue d3 , and according to the upper limit value t of the point cloud timestamp d2 , determine the upper limit value t of the positioning timestamp from the timestamps of multiple initial positioning information frames in the positioning information receiving queue d4 Then according to the lower limit value t of the positioning timestamp d3 and the upper limit of the positioning timestamp t d4 , determine the positioning timestamp interval [t d3,t d4 ], and finally according to the timestamp at [t d3 ,t d4 ] are merged to obtain a positioning information frame screening sequence. Among them, the lower limit value of the positioning timestamp is less than or equal to the lower limit value of the point cloud timestamp, and the upper limit value of the positioning timestamp is greater than or equal to the upper limit value of the point cloud timestamp, that is, t d3 ≤t d1 ,t d4 ≥t d2 .

[0087] In one embodiment, step 301 includes:

[0088] If the lower limit value of the point cloud timestamp is greater than or equal to the timestamp of each initial positioning information frame, a difference calculation is performed based on the lower limit value of the point cloud timestamp and the timestamp of the initial positioning information frame to obtain a lower limit difference;

[0089] The timestamp of the initial positioning information frame corresponding to the minimum lower bound difference is determined as the lower bound value of the positioning information timestamp;

[0090] If the upper limit value of the point cloud timestamp is less than or equal to the timestamp of each initial positioning information frame, the difference between the timestamp of the initial positioning information frame and the upper limit value of the point cloud timestamp is calculated to obtain the upper limit difference;

[0091] The timestamp of the initial positioning information frame corresponding to the smallest upper bound difference is determined as the upper bound value of the positioning information timestamp.

[0092] For example, the lower bound of the point cloud timestamp is t d1 =0.1, the upper limit of the point cloud timestamp t d2 =0.2, and the positioning information receiving queue has 15 positioning information frames, and the timestamps are t1=0.02, t2=0.04, t3=0.06, t4=0.08, t5=0.10, t6=0.12, t7=0.14, t8=0.16, t9=0.18, t 10 =0.20, t 11 =0.22, t 12 =0.24, t 13 =0.26, t 14 =0.28, t 15 =0.30. d1 No need to compare t6 to t 15 The difference calculation is performed, but t d1 Calculate the difference between t1 to t5 respectively to get t d1 The difference between the lower bound calculated with t5 is the smallest, and then t5 is determined as the lower bound value of the positioning information timestamp t d3 , that is, t d3=0.10. d2 No difference calculation is required with t1 to t9, but t d2 Respectively with t 10 to 15 Calculate the difference to get t d2 With t 10 The calculated upper bound difference is the minimum, and then t 10 Determine the upper limit value t of the positioning information timestamp d4 , that is, t d4 = 0.20. The units of the timestamps in this example are all seconds.

[0093] The benefit of the above embodiment is that it can save computing resources, thereby improving the efficiency of point cloud motion compensation.

[0094] In one embodiment, referring to Figure 4 After step 103, the point cloud motion compensation method may further include:

[0095] Step 401, counting the number of initial positioning information frames in the positioning information frame screening sequence to obtain the number of frames;

[0096] Step 402, if the number of frames is less than or equal to a preset number threshold, jump to the next frame of the point cloud information frame, and enter the step of determining the point cloud timestamp interval according to the timestamp of the point cloud information frame; wherein the number threshold is less than or equal to N.

[0097] For example, if the positioning information frame screening sequence includes 4 initial positioning information frames, the number of frames obtained after the number statistics is 4. Assume that N=10, and the number threshold is 5. At this time, if the number of frames is less than 5, it is considered that all the positioning information frames in the positioning information frame screening sequence are unavailable, and there is no need to perform motion compensation on the current point cloud information frame, but it is possible to jump to the next frame of the point cloud information frame and re-enter step 102 to enter the process of performing motion compensation on the next frame. This embodiment can ensure the accuracy of motion compensation.

[0098] It should be noted that if there is no abnormal situation (such as poor GPS signal), the number of frames corresponding to the positioning information frame screening sequence will be greater than the number threshold, and subsequent steps can be performed normally to ensure the accuracy of motion compensation.

[0099] In step 104, the point cloud information frame is divided into blocks (also called grouping) to obtain a target number of point cloud information blocks. In combination with the above, the point cloud information frame includes point cloud information of multiple point clouds. In one example, all point cloud information of the point cloud information frame is evenly divided in chronological order to obtain a target number of point cloud information blocks. For example, the LIDAR_POINT_NUMS point clouds contained in the point cloud information frame are divided into blocks to obtain LOC_NUMS point cloud information blocks, and the number of point clouds in a single point cloud information block is LIDAR_POINT_NUMS / LOC_NUMS. If there are n point cloud information blocks in total, it can be expressed as {block1, block2, …, blockn}.

[0100] In step 105, the initial positioning information frames in the positioning information frame screening sequence are interpolated (also called linear interpolation) to obtain a target number of target positioning information frames. For example, the positioning information frame screening sequence includes 10 initial positioning information frames LOC_A, and after interpolation, n target positioning information frames LOC_B are obtained.

[0101] It should be noted that the number of point cloud information blocks is consistent with the number of target positioning information frames.

[0102] In step 106, timestamp alignment is performed on each point cloud information block and each target positioning information frame to obtain the target matching positioning timestamp of each point cloud information block. The timestamp alignment here can be called fine timestamp alignment. In combination with the above, there are n point cloud information blocks and n target positioning information frames, and the target positioning information LOC_B can be matched one by one with the point cloud information block block, so that the timestamp of the target positioning information LOC_B is used as the target matching positioning timestamp of the point cloud information block block.

[0103] The point cloud information block includes at least two point clouds. Figure 5 , step 106 comprises:

[0104] Step 501, determining the median of the timestamp according to the timestamps of all point clouds of each point cloud information block;

[0105] Step 502: perform a closest match on the timestamps of the target number of positioning information frames according to the timestamp median to obtain the target matching positioning timestamp.

[0106] In step 501, the median of the timestamp refers to the value in the middle position after the timestamps of all the point clouds of the point cloud information block are arranged from small to large. If the number of point clouds is an odd number, the median is the middle number; if it is an even number, the median is the average of the two middle numbers. For example, if the point cloud information block has 3 point clouds, and the timestamps are 0.09 seconds, 0.1 seconds, and 0.11 seconds, the median is 0.1 seconds. Based on the median of 0.1 seconds and the closest match with the timestamps of 100 positioning information frames, the target positioning timestamp may be exactly equal to 0.1 seconds, or it may be less than or greater than 0.1 seconds.

[0107] In step 502, the target matching positioning timestamp is the timestamp of the positioning information frame with the smallest difference with the timestamp median. Step 502 may include: performing a difference calculation based on the timestamp median and the timestamp of each positioning information frame to obtain a timestamp difference; and determining the timestamp of the positioning information frame corresponding to the absolute value of the smallest timestamp difference as the target matching positioning timestamp.

[0108] The benefit of the above embodiment is that the accuracy of timestamp alignment can be further improved.

[0109] In step 107, the ego-to-world transformation relationship of each block is determined according to each target matching positioning timestamp. The block ego-to-world transformation relationship is used to characterize the transformation relationship from the ego coordinate system (ego) to the world coordinate system (global) of the point cloud information block. For example, if there are n point cloud information blocks in total, there are n target matching positioning timestamps corresponding to them, so that n block ego-to-world transformation relationships can be determined.

[0110] It should be noted that each target matching and positioning timestamp is the timestamp of the target positioning information frame corresponding to the point cloud information block, and under the timestamp of the target positioning information frame, there is a corresponding conversion relationship from the vehicle coordinate system to the world coordinate system. Therefore, according to the target matching and positioning timestamp, the conversion relationship from the vehicle coordinate system to the world coordinate system corresponding to each point cloud information block can be obtained.

[0111] In step 108, motion compensation is performed on each point cloud information block according to each block of self-car-to-world transformation relationship. Specifically, the point cloud is coordinate-transformed using the block self-car-to-world transformation relationship to achieve motion distortion removal.

[0112] In one embodiment, referring to Figure 6 , step 108 may include:

[0113] Step 601, using a preset laser radar to ego vehicle conversion relationship, coordinate conversion is performed on the initial laser radar coordinates of the point cloud information block to obtain a first ego vehicle coordinate;

[0114] Step 602, using the block self-vehicle to world transformation relationship, the first self-vehicle coordinates are transformed to obtain world coordinates; wherein the world coordinates of all point cloud information blocks share the same world coordinate system.

[0115] The lidar-to-ego-vehicle conversion relationship refers to the conversion relationship from the lidar coordinate system to the ego coordinate system. For example, assuming there are n point cloud information blocks, specifically {block1, block2, …, blockn}, all point cloud information blocks need to be converted to the same world coordinate system. First, the lidar-to-ego-vehicle conversion relationship is used to convert the lidar coordinate system to the ego coordinate system, and then the block ego-to-world conversion relationship is used to convert each point cloud in blocki (i∈[1,n]) to the world coordinate system. In the world coordinate system, different point clouds share the same world coordinate system.

[0116] The benefit of the above embodiment is that the coordinate transformation is performed by using the block-from-car-to-world transformation relationship to achieve motion distortion removal.

[0117] In one embodiment, referring to Figure 7 , after step 602, step 108 further includes:

[0118] Step 701, determining a target point cloud information block from all point cloud information blocks;

[0119] Step 702, using the block ego-vehicle to world transformation relationship of the target point cloud information block, coordinate transformation is performed on the world coordinates of each point cloud information block to obtain a second ego-vehicle coordinate;

[0120] Step 703, using the laser radar to vehicle conversion relationship, the second vehicle coordinates are converted to obtain the target laser radar coordinates; wherein the target laser radar coordinates of all point cloud information blocks share the laser radar coordinate system corresponding to the target point cloud information block.

[0121] In the above step 701, one point cloud information block can be randomly selected from all point cloud information blocks as the target point cloud information block. In one example, the first point cloud information block is selected from all point cloud information blocks in chronological order as the target point cloud information block, so that the coordinates of all point cloud information blocks can be converted to the coordinate system of the first point cloud information block, which can further improve the intuitiveness of the coordinate changes of different point cloud information blocks over time on the basis of achieving coordinate unification.

[0122] For example, assuming that point cloud information block block1 is determined as the target point cloud information block from n point cloud information blocks, point cloud information block blocki needs to be converted to point cloud information block block1. Combined with the above example, after obtaining the world coordinates, the world coordinates of point cloud information block blocki are converted using the self-vehicle-to-world conversion relationship of point cloud information block block1 to obtain the second self-vehicle coordinates of point cloud information block blocki. Then, using the laser radar to self-vehicle conversion relationship, the second self-vehicle coordinates of point cloud information block blocki are converted to obtain the target laser radar coordinates of point cloud information block blocki. The target laser radar coordinates of point cloud information blocks block1 to blockn share the laser radar coordinate system corresponding to block1.

[0123] It can be understood that when using the block self-vehicle-to-world transformation relationship of the target point cloud information block to transform the world coordinates of each point cloud information block, the block self-vehicle-to-world transformation relationship can be inverted first to obtain the block self-vehicle-to-world inverse transformation relationship, and then the world coordinates of the point cloud information block and the block self-vehicle-to-world inverse transformation relationship can be used for coordinate calculation to obtain the second self-vehicle coordinates.

[0124] The benefit of the above embodiment is that coordinate unification can be achieved on the basis of being able to implement motion compensation.

[0125] In summary of the above embodiments, the present application can use the positioning information (also called navigation positioning information) carried by the assisted driving or autonomous driving platform to perform motion compensation on the obtained point cloud information. The addition of navigation positioning information provides a world coordinate system for different point cloud information in the same frame of the laser radar, which effectively solves the distortion problem caused by the movement of the vehicle.

[0126] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0127] The present application also discloses a point cloud motion compensation device, referring to Figure 8The point cloud motion compensation device includes: an information receiving module 801, an interval determination module 802, a first alignment module 803, an information blocking module 804, an information interpolation module 805, a second alignment module 806, a conversion determination module 807 and a motion compensation module 808. The information receiving module 801 is used to receive the point cloud information frame at a first frequency and receive the initial positioning information frame at a second frequency; wherein the second frequency is equal to N*the first frequency, and N is greater than 1; the interval determination module 802 is used to determine the point cloud timestamp interval according to the timestamp of the point cloud information frame; the first alignment module 803 is used to perform timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence; the information blocking module 804 is used to block the point cloud information frame to obtain a target number of point cloud information blocks; the information interpolation module 805 is used to screen the positioning information frame The initial positioning information frames in the sequence are interpolated to obtain the target positioning information frames of the target number; the second alignment module 806 is used to align the timestamps of each point cloud information block and each target positioning information frame to obtain the target matching positioning timestamp of each point cloud information block; the conversion determination module 807 is used to determine the self-vehicle-to-world conversion relationship of each block according to each target matching positioning timestamp, and the self-vehicle-to-world conversion relationship of each block is used to characterize the conversion relationship from the self-vehicle coordinate system to the world coordinate system of the point cloud information block; the motion compensation module 808 is used to perform motion compensation on each point cloud information block according to each self-vehicle-to-world conversion relationship.

[0128] In one embodiment, the point cloud motion compensation device also includes a frame jumping module, which is used to: count the number of initial positioning information frames in the positioning information frame screening sequence to obtain the number of frames; if the number of frames is less than or equal to a preset number threshold, jump to the next frame of the point cloud information frame, and enter the step of determining the point cloud timestamp interval according to the timestamp of the point cloud information frame; wherein the number threshold is less than or equal to N.

[0129] It should be noted that the specific implementation of the point cloud motion compensation device is basically the same as the specific implementation of the above-mentioned point cloud motion compensation method, and will not be repeated here.

[0130] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned point cloud motion compensation method when executing the computer program. In one embodiment, the electronic device is a vehicle, and the laser radar is installed on the vehicle.

[0131] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the point cloud motion compensation method as described above is implemented.

[0132] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0134] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0135] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0137] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0138] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0143] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A point cloud motion compensation method, characterized in that: The method comprises: Receiving point cloud information frames at a first frequency, and receiving initial positioning information frames at a second frequency; wherein the second frequency is equal to N*the first frequency, and N is greater than 1; Determining a point cloud timestamp interval according to the timestamp of the point cloud information frame; Performing timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence; Dividing the point cloud information frame into blocks to obtain a target number of point cloud information blocks; Performing interpolation processing on the initial positioning information frames in the positioning information frame screening sequence to obtain the target number of target positioning information frames; Performing timestamp alignment on each of the point cloud information blocks and each of the target positioning information frames to obtain a target matching positioning timestamp for each of the point cloud information blocks; Determine each block of self-vehicle-to-world transformation relationship according to each target matching positioning timestamp, wherein the block of self-vehicle-to-world transformation relationship is used to characterize the transformation relationship from the self-vehicle coordinate system to the world coordinate system of the point cloud information block; Motion compensation is performed on each of the point cloud information blocks according to a vehicle-to-world transformation relationship of each of the blocks.

2. The method according to claim 1, characterized in that The point cloud timestamp interval includes a point cloud timestamp lower limit value and a point cloud timestamp upper limit value; The step of performing timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence includes: According to the lower limit value of the point cloud timestamp, a lower limit value of the positioning timestamp is determined from the timestamps of the multiple initial positioning information frames, and according to the upper limit value of the point cloud timestamp, an upper limit value of the positioning timestamp is determined from the timestamps of the multiple initial positioning information frames; Determine a positioning timestamp interval according to the positioning timestamp lower limit value and the positioning timestamp upper limit value; The positioning information frame screening sequence is obtained by merging all the initial positioning information frames located in the positioning timestamp interval.

3. The method according to claim 2, characterized in that The step of determining a positioning timestamp lower limit value from timestamps of a plurality of the initial positioning information frames according to the point cloud timestamp lower limit value, and determining a positioning timestamp upper limit value from timestamps of a plurality of the initial positioning information frames according to the point cloud timestamp upper limit value comprises: If the lower limit value of the point cloud timestamp is greater than or equal to the timestamp of each of the initial positioning information frames, a difference calculation is performed based on the lower limit value of the point cloud timestamp and the timestamp of the initial positioning information frame to obtain a lower limit difference value; Determine the timestamp of the initial positioning information frame corresponding to the smallest lower bound difference as the positioning information timestamp lower bound value; If the upper limit value of the point cloud timestamp is less than or equal to the timestamp of each of the initial positioning information frames, performing a difference calculation based on the timestamp of the initial positioning information frame and the upper limit value of the point cloud timestamp to obtain an upper limit difference value; The timestamp of the initial positioning information frame corresponding to the smallest upper bound difference is determined as the upper bound value of the positioning information timestamp.

4. The method according to any one of claims 1 to 3, characterized in that: After aligning the timestamps according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence, the method further includes: Counting the number of initial positioning information frames in the positioning information frame screening sequence to obtain the number of frames; If the number of frames is less than or equal to a preset number threshold, jump to the next frame of the point cloud information frame and enter the step of determining the point cloud timestamp interval based on the timestamp of the point cloud information frame; wherein the number threshold is less than or equal to N.

5. The method according to any one of claims 1 to 3, characterized in that: The point cloud information block includes at least two point clouds; The step of aligning the timestamps of each point cloud information block and each target positioning information frame to obtain the target matching positioning timestamp of each point cloud information block includes: Determine a timestamp median according to the timestamps of all the point clouds of each point cloud information block; According to the median of the timestamp, the timestamps of the target positioning information frames of the target number are matched most closely to obtain the target matching positioning timestamp.

6. The method according to any one of claims 1 to 3, characterized in that: The performing motion compensation on each of the point cloud information blocks according to the self-car-to-world transformation relationship of each of the blocks comprises: Using a preset laser radar to ego vehicle conversion relationship, coordinate conversion is performed on the initial laser radar coordinates of the point cloud information block to obtain a first ego vehicle coordinate; The first self-vehicle coordinates are transformed by utilizing the block self-vehicle-to-world transformation relationship to obtain world coordinates; wherein the world coordinates of all the point cloud information blocks share the same world coordinate system.

7. The method according to claim 6, characterized in that After the first self-vehicle coordinates are transformed by using the block self-vehicle-to-world transformation relationship to obtain the world coordinates, the motion compensation is performed on each of the point cloud information blocks according to each of the block self-vehicle-to-world transformation relationships, further comprising: Determine a target point cloud information block from all the point cloud information blocks; Using the block self-vehicle-to-world transformation relationship of the target point cloud information block, coordinate transformation is performed on the world coordinates of each of the point cloud information blocks to obtain a second self-vehicle coordinate; The second self-vehicle coordinates are transformed by utilizing the laser radar to self-vehicle conversion relationship to obtain target laser radar coordinates; wherein the target laser radar coordinates of all the point cloud information blocks share the laser radar coordinate system corresponding to the target point cloud information block.

8. A point cloud motion compensation device, characterized in that: The device comprises: An information receiving module, used to receive point cloud information frames at a first frequency and receive initial positioning information frames at a second frequency; wherein the second frequency is equal to N*the first frequency, and N is greater than 1; An interval determination module, used to determine a point cloud timestamp interval according to the timestamp of the point cloud information frame; A first alignment module, configured to perform timestamp alignment according to the point cloud timestamp interval and the timestamp of the initial positioning information frame to obtain a positioning information frame screening sequence; An information segmentation module, used to segment the point cloud information frame into blocks to obtain a target number of point cloud information blocks; An information interpolation module, used for performing interpolation processing on the initial positioning information frames in the positioning information frame screening sequence to obtain the target number of target positioning information frames; A second alignment module is used to align the timestamps of each point cloud information block and each target positioning information frame to obtain the target matching positioning timestamp of each point cloud information block; A conversion determination module, used to determine each block of ego-vehicle-to-world conversion relationship according to each target matching positioning timestamp, wherein the block ego-vehicle-to-world conversion relationship is used to represent the conversion relationship from the ego-vehicle coordinate system to the world coordinate system of the point cloud information block; The motion compensation module is used to perform motion compensation on each of the point cloud information blocks according to the vehicle-to-world transformation relationship of each of the blocks.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images