Motion compensation method and device for three-dimensional point cloud data, medium and electronic equipment

Abstract

The application relates to the technical field of laser radar, in particular to a motion compensation method and device of three-dimensional point cloud data, a medium and an electronic device. The method comprises the following steps: acquiring a first image, a second image, first point cloud data and second point cloud data of a target; the first image and the first point cloud data respectively contain first position data and first depth data of a target point on the target at a first moment; the second image and the second point cloud data respectively contain second position data and second depth data of the target point at a second moment; acquiring target frame point cloud data of the target in a target time period; the target time period is associated with the first moment and / or the second moment; determining an image position deviation of the target point according to the first image and the second image; determining a three-dimensional velocity of the target based on the image position deviation, the first depth data and the second depth data; and performing motion compensation on the target frame point cloud data based on the three-dimensional velocity. Without a deep learning model, computer computing power can be reduced, and cost can be saved.

Description

Technical Field

[0001] This application relates to the field of lidar technology, and in particular to methods, devices, media and electronic equipment for motion compensation of three-dimensional point cloud data. Background Technology

[0002] A lidar system is a radar system that uses laser beams to detect the position, velocity, and other characteristics of a target. Its working principle involves emitting a detection signal (laser beam) towards the target, then comparing the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, information about the target can be obtained, such as its distance, azimuth, altitude, velocity, attitude, and even shape, enabling target detection, tracking, and identification. A lidar system consists of a laser transmitter, an optical receiver, a turntable, and an information processing system.

[0003] Point cloud data is a dataset of spatial points obtained by scanning with a 3D LiDAR device. Each point contains 3D coordinate information, which is the X, Y, and Z coordinates we commonly refer to. Some points also contain color information, reflection intensity information, echo count information, etc.

[0004] Each frame of point cloud data is not exposed at the same time. When a LiDAR measures a moving target, there is a positional offset between points in each frame of point cloud data at different times. This positional offset is proportional to the speed of the moving target and the time difference between the points. When this positional offset is large, the shape of the target composed of multiple points will be distorted, i.e., motion distortion of the point cloud. Typically, the frame rate of a LiDAR is 10fps, meaning that the time difference between points in each frame of point cloud data is relatively large. Therefore, when the target speed is constant, the resulting positional offset between points is also large.

[0005] Typically, neural network models are used to predict the point correspondence between two adjacent frames of point cloud data. Based on the position changes of the corresponding points, the velocity of the target is predicted, and then motion compensation is performed on the two frames of point cloud data based on time. However, this method consumes a lot of computing power and is costly.

[0006] Therefore, there is a need for motion compensation methods, devices, media, and electronic equipment for 3D point cloud data, which can effectively compensate for the motion of 3D point cloud data, save computing power, and reduce costs. Summary of the Invention

[0007] This application provides a method, apparatus, medium, and electronic device for motion compensation of three-dimensional point cloud data, which can effectively compensate for motion of three-dimensional point cloud data, save computing power, and reduce costs.

[0008] In a first aspect, embodiments of this application provide a motion compensation method for three-dimensional point cloud data, the method comprising:

[0009] Acquire a first image, a second image, first point cloud data, and second point cloud data of the target; the first image and the first point cloud data respectively contain first position data and first depth data of the target point at a first time; the second image and the second point cloud data respectively contain second position data and second depth data of the target point at a second time.

[0010] Acquire target frame point cloud data of the target within the target time period; the target time period is associated with the first time point and / or the second time point.

[0011] Based on the first location data and the second location data, determine the image position deviation of the target point;

[0012] Based on the image position deviation, the first depth data, and the second depth data, the three-dimensional velocity of the target is determined from the first time moment to the second time moment;

[0013] Based on the three-dimensional velocity, motion compensation is performed on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data.

[0014] In some optional embodiments, determining the three-dimensional velocity of the target from the first time moment to the second time moment based on the image position deviation, the first depth data, and the second depth data includes:

[0015] Based on the first depth data and the second depth data, the depth displacement of the target is determined;

[0016] The image position deviation is projected into three-dimensional space to obtain a three-dimensional planar displacement;

[0017] Based on the three-dimensional planar displacement and the depth displacement, the three-dimensional displacement of the target is determined;

[0018] Based on the three-dimensional displacement, the three-dimensional velocity of the target is determined from the first time moment to the second time moment.

[0019] In some optional embodiments, the first image and the second image include focal length information;

[0020] The step of projecting the image position deviation into three-dimensional space includes:

[0021] Based on the first depth data, the second depth data, and the focal length information, the image position deviation is projected onto the three-dimensional space.

[0022] In some optional embodiments, based on the three-dimensional velocity, motion compensation is performed on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data, including:

[0023] Based on the multiple related point cloud data, multiple exposure times corresponding to the multiple related points are determined;

[0024] The time interval between each exposure time is determined, and based on the time interval and the three-dimensional velocity, multiple location information in the multiple related point cloud data is corrected.

[0025] In some optional embodiments, the target time period is temporally associated with the first time point and / or the second time point, including:

[0026] The time difference between the target time period and the first time and / or the second time meets a preset time difference threshold.

[0027] In some optional embodiments, before performing motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the three-dimensional velocity, the method further includes:

[0028] Based on the target frame rate in the target frame point cloud data, determine the speed threshold;

[0029] When the three-dimensional velocity reaches the velocity threshold, the step of performing motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the three-dimensional velocity is executed.

[0030] In some alternative embodiments, the target point is the centroid of the target.

[0031] Secondly, embodiments of this application provide a motion compensation device for three-dimensional point cloud data, characterized in that the device comprises:

[0032] The first acquisition module is used to acquire a first image, a second image, first point cloud data, and second point cloud data of the target; the first image and the first point cloud data respectively contain first position data and first depth data of the target point at a first time; the second image and the second point cloud data respectively contain second position data and second depth data of the target point at a second time.

[0033] The second acquisition module is used to acquire target frame point cloud data of the target within a target time period; the target time period is associated with the first time and / or the second time.

[0034] The first determining module is used to determine the image position deviation of the target point based on the first position data and the second position data;

[0035] The second determining module is used to determine the three-dimensional velocity of the target from the first time moment to the second time moment based on the image position deviation, the first depth data and the second depth data;

[0036] The motion compensation module is used to perform motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the three-dimensional velocity.

[0037] In some optional embodiments, the second determining module includes:

[0038] The first determining submodule is used to determine the depth displacement of the target based on the first depth data and the second depth data;

[0039] The second determining submodule projects the image position deviation into three-dimensional space to obtain a three-dimensional planar displacement.

[0040] The third determining submodule is used to determine the three-dimensional displacement of the target based on the three-dimensional planar displacement and the depth displacement;

[0041] The fourth determining submodule is used to determine the three-dimensional velocity of the target from the first time moment to the second time moment based on the three-dimensional displacement.

[0042] In some optional embodiments, the first image and the second image include focal length information;

[0043] The second determining submodule is further configured to: project the image position deviation onto the three-dimensional space based on the first depth data, the second depth data, and the focal length information.

[0044] In some optional embodiments, the motion compensation module includes:

[0045] The first motion compensation submodule is used to determine multiple exposure times corresponding to the multiple related points based on the multiple related point cloud data;

[0046] The second motion compensation submodule is used to determine the time interval between each exposure moment in the plurality of exposure moments, and to correct multiple position information in the plurality of related point cloud data based on the time interval and the three-dimensional velocity.

[0047] In some optional embodiments, the motion compensation module is further configured to: ensure that the time difference between the target time period and the first time and / or the second time satisfies a preset time difference threshold.

[0048] In some optional embodiments, the apparatus further includes:

[0049] The third determining module is used to determine the speed threshold based on the target frame rate in the target frame point cloud data;

[0050] The judgment module is used to perform the step of motion compensation based on the three-dimensional velocity for multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data when the three-dimensional velocity reaches the velocity threshold.

[0051] Thirdly, embodiments of this application provide a computer storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described motion compensation method for three-dimensional point cloud data.

[0052] Fourthly, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by the processor and executed to perform the motion compensation method for the three-dimensional point cloud data.

[0053] This application acquires a first image, a second image, first point cloud data, and second point cloud data of a target. The first image and the first point cloud data respectively contain first position data and first depth data of a target point on the target at a first time moment. The second image and the second point cloud data respectively contain second position data and second depth data of the target point at a second time moment. It acquires target frame point cloud data of the target over a target time period, which is associated with the first time moment and / or the second time moment. Based on the first position data and the second position data, it determines the image position deviation of the target point. Based on the image position deviation, the first depth data, and the second depth data, it determines the three-dimensional velocity of the target from the first time moment to the second time moment. Based on the three-dimensional velocity, it performs motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data. This allows for effective motion compensation of three-dimensional point cloud data without the need for a deep learning model, saving computing power and reducing costs. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1This is an application scenario diagram of a motion compensation method for three-dimensional point cloud data provided in the embodiments of this application;

[0056] Figure 2 This is a flowchart illustrating a motion compensation method for three-dimensional point cloud data provided in an embodiment of this application;

[0057] Figure 3 This is a schematic diagram of a process for determining three-dimensional velocity according to an embodiment of this application;

[0058] Figure 4 This is a schematic diagram of a motion compensation process provided in an embodiment of this application;

[0059] Figure 5 This is a schematic diagram of the structure of a motion compensation device for three-dimensional point cloud data provided in an embodiment of this application;

[0060] Figure 6 This is a hardware structure block diagram of an electronic device for implementing a motion compensation method for three-dimensional point cloud data, provided in an embodiment of this application. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0062] The term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. In the description of the invention, it should be understood that the terms "first," "second," "third," and "fourth," etc., in the specification, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0063] Before introducing the method of the embodiments of this application, we will first give an example of the application scenarios of the motion compensation method for three-dimensional point cloud data.

[0064] Please see Figure 1 , Figure 1This is an application scenario diagram of a motion compensation method for three-dimensional point cloud data provided in an embodiment of this application. Each frame of point cloud data of the lidar 11 is obtained by scanning around the lidar 11. The relative motion between the lidar 11 and the target 12 to be scanned will cause motion distortion of the point cloud of the lidar 11. Specifically, when the lidar 11 and the target 12 to be scanned remain relatively stationary, the scanning start point 13 and scanning end point 14 of the lidar 11 can be closed relatively well. However, when the lidar 11 moves relative to the target 12 to be scanned (e.g., the lidar 11 moves towards the target position 15), each frame of point cloud data of the lidar 11 will be distorted, causing the loop to no longer close, that is, the scanning start point 13 and scanning end point 14 will no longer close.

[0065] Each point in the point cloud has a positional shift at different times. The greater the relative speed between the lidar 11 and the target 12 being scanned, the more obvious the motion distortion of each point cloud frame becomes. Motion compensation is needed for each point cloud frame to compensate for the motion distortion.

[0066] In the prior art, when the position of the lidar 11 remains unchanged while the target 12 to be scanned moves, the corresponding relationship between points in two adjacent frames of the lidar 11 is usually predicted based on a neural network model. Based on the position change of the corresponding points, the speed of the target 12 to be scanned is predicted. Then, based on the time interval between points in each frame of the point cloud, motion compensation is performed on each frame of the point cloud. However, this method requires a lot of computing power and is costly.

[0067] To address the aforementioned problems, this application provides a motion compensation method for 3D point cloud data. Specifically, the method includes: acquiring a first image, a second image, first point cloud data, and second point cloud data of a target; the first image and the first point cloud data respectively contain first position data and first depth data corresponding to a target point on the target at a first time moment; the second image and the second point cloud data respectively contain second position data and second depth data corresponding to the target point at a second time moment; acquiring target frame point cloud data of the target within a target time period; the target time period being associated with the first time moment and / or the second time moment; determining the image position deviation of the target point based on the first position data and the second position data; determining the 3D velocity of the target from the first time moment to the second time moment based on the image position deviation, the first depth data, and the second depth data; and performing motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the 3D velocity. This method can effectively compensate for the motion of 3D point cloud data without requiring a deep learning model, saving computing power and reducing costs.

[0068] The following describes a specific embodiment of a motion compensation method for three-dimensional point cloud data according to this application. Figure 2 This is a flowchart illustrating a motion compensation method for 3D point cloud data provided in an embodiment of this application. This specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0069] S202: Acquire a first image, a second image, first point cloud data, and second point cloud data of the target; the first image and the first point cloud data respectively contain first position data and first depth data of the target point at a first time; the second image and the second point cloud data respectively contain second position data and second depth data of the target point at a second time.

[0070] The target's position is the same at the same time. Images of the target at the first and second moments are acquired using image acquisition devices (such as cameras); point cloud data of the target at the first and second moments are acquired using lidar. The image displacement information and point cloud displacement information of the target correspond between the first and second moments.

[0071] In some alternative embodiments, the target point is the centroid of the target.

[0072] Specifically, the prediction accuracy of the target's centroid (i.e., the geometric center after distortion) is high before and after motion distortion. Using the centroid as the target point is beneficial to improving the accuracy of predicting the target's three-dimensional velocity.

[0073] For example, in the first image, the first position information of the centroid at the first moment is (x1, y1), and in the first point cloud data, the first depth data of the centroid at the first moment is z1; in the second image, the second position information of the centroid at the second moment is (x2, y2), and in the second point cloud data, the second depth data of the centroid at the second moment is z1. The time difference between the first moment and the second moment is dti.

[0074] S204: Obtain target frame point cloud data of the target within the target time period; the target time period is associated with the first time and / or the second time.

[0075] Specifically, the closer the time is to the first and / or second moment, the closer the target's velocity within the target time period is to the velocity determined at the first and / or second moment. This results in better motion compensation for target frame point clouds exhibiting motion distortion.

[0076] S206: Based on the first location data and the second location data, determine the image position deviation of the target point.

[0077] The image position deviation of the target point determined in the first image and the second image is taken as the image position deviation of the target.

[0078] Specifically, the image position deviation includes horizontal displacement six = x2 - x1 and vertical displacement siy = y2 - y1.

[0079] S208: Based on the image position deviation, the first depth data, and the second depth data, determine the three-dimensional velocity of the target from the first time moment to the second time moment.

[0080] The two-dimensional first and second images cannot capture the depth position change of the target point; this change needs to be determined from the aforementioned first and second point cloud data. For example, the depth position change of the centroid is Z2-Z1.

[0081] Figure 3 This is a flowchart illustrating a method for determining three-dimensional velocity according to an embodiment of this application. In some optional embodiments, step S208 above determines the three-dimensional velocity of the target from the first time moment to the second time moment based on the image position deviation, the first depth data, and the second depth data, including... Figure 3 The following steps are shown:

[0082] S302: Determine the depth displacement of the target based on the first depth data and the second depth data.

[0083] For example, the depth displacement is the change in the depth position of the centroid mentioned above, i.e., Z2-Z1.

[0084] S304: Project the image position deviation into three-dimensional space to obtain three-dimensional planar displacement.

[0085] Specifically, the image positional deviation is converted into three-dimensional space to determine the target's true moving speed.

[0086] S306: Determine the three-dimensional displacement of the target based on the three-dimensional planar displacement and the depth displacement.

[0087] Specifically, three-dimensional displacement refers to the actual displacement of the target in three-dimensional space.

[0088] S308: Based on the three-dimensional displacement, determine the three-dimensional velocity of the target from the first moment to the second moment.

[0089] To obtain the displacement corresponding to the image position deviation in three-dimensional space, in some optional embodiments, the first image and the second image include focal length information; the above-mentioned projection of the image position deviation into three-dimensional space includes:

[0090] Based on the first depth data, the second depth data, and the focal length information, the image position deviation is projected onto the three-dimensional space.

[0091] For example, given the focal length f of the lens of the aforementioned image acquisition device, the velocity estimates vix in the horizontal direction, viy in the vertical direction, and viz in the longitudinal direction are determined based on the following formulas (1), (2), and (3), respectively.

[0092]

[0093]

[0094]

[0095] Where Z1 is the first depth data of the centroid, Z2 is the second depth data of the centroid, six is ​​the horizontal displacement, siy is the vertical displacement of the centroid, dti is the time difference between the first and second moments, and f is the focal length of the image acquisition device that generates the first and second images.

[0096] In this embodiment, the image position deviation is projected into three-dimensional space based on a ratio of depth data and focal length. This reduces computing power while ensuring the accuracy of three-dimensional planar displacement.

[0097] S210: Based on the three-dimensional velocity, perform motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data.

[0098] Specifically, when the target velocity changes, the closer the target time period is to the first moment and / or the second moment, the closer the target velocity of the target time period is to the three-dimensional velocity mentioned above; when the target velocity remains constant, the target time period can be any period of the target's motion.

[0099] Motion compensation for point clouds maps the exposure information of all points on a target within the point cloud onto a unified time axis. This process aims to improve the accuracy of motion compensation based on 3D velocity. Figure 4This is a schematic flowchart of motion compensation provided in an embodiment of this application; in some optional embodiments, in step S210, motion compensation is performed on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the three-dimensional velocity, including... Figure 4 The following steps are shown:

[0100] S402: Based on the multiple related point cloud data, determine multiple exposure times corresponding to the multiple related points.

[0101] Specifically, the displacement between points in the point cloud is determined by the time difference (exposure time difference) between points and the velocity change between points. The aforementioned three-dimensional velocity is used as the target's motion velocity between points to predict the three-dimensional displacement between points.

[0102] S404: Determine the time interval between each exposure moment in the plurality of exposure moments, and correct the multiple position information in the plurality of related point cloud data based on the time interval and the three-dimensional velocity.

[0103] For example, the point cloud coordinates (x, y, z) on the target have a time difference of d relative to the first reference point on the target. tln The displacement of point cloud n relative to the first reference point in the three axes is determined by the following formulas (4), (5) and (6):

[0104] dx = vix * d tln (4)

[0105] dy = viy * d tln (5)

[0106] dz=viz*d tln (6)

[0107] The corrected point cloud coordinates n are (x+dx, y+dy, z+dz).

[0108] To quickly filter point cloud data and rapidly determine target frame point cloud data, in some optional embodiments, the target time period is associated with the first time point and / or the second time point, including:

[0109] The time difference between the target time period and the first time and / or the second time meets a preset time difference threshold.

[0110] To increase the effectiveness of motion compensation and avoid performing motion compensation on target frame point clouds with insignificant motion distortion, in some optional embodiments, before performing motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the three-dimensional velocity, the method further includes:

[0111] Based on the target frame rate in the target frame point cloud data, determine the speed threshold;

[0112] When the three-dimensional velocity reaches the velocity threshold, the step of performing motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the three-dimensional velocity is executed.

[0113] In this embodiment, a velocity threshold is determined based on the frame rate of the LiDAR contained in the target frame point cloud data. This ensures that motion compensation is performed only before motion compensation is initiated, and only when the target's three-dimensional velocity reaches the velocity threshold, indicating significant motion distortion in the LiDAR's target frame point cloud. This approach allows motion compensation to be performed only on target frame point clouds exhibiting significant distortion, further reducing computational power and saving costs.

[0114] Figure 5 This is a schematic diagram of the structure of a motion compensation device for three-dimensional point cloud data provided in an embodiment of this application, as shown below. Figure 5 As shown, the motion compensation device for the 3D point cloud data includes:

[0115] The first acquisition module is used to acquire a first image, a second image, first point cloud data, and second point cloud data of the target; the first image and the first point cloud data respectively contain first position data and first depth data of the target point at a first time; the second image and the second point cloud data respectively contain second position data and second depth data of the target point at a second time.

[0116] The second acquisition module is used to acquire target frame point cloud data of the target within a target time period; the target time period is associated with the first time and / or the second time.

[0117] The first determining module is used to determine the image position deviation of the target point based on the first position data and the second position data;

[0118] The second determining module is used to determine the three-dimensional velocity of the target from the first time moment to the second time moment based on the image position deviation, the first depth data and the second depth data;

[0119] The motion compensation module is used to perform motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data based on the three-dimensional velocity.

[0120] In some optional embodiments, the second determining module includes:

[0121] The first determining submodule is used to determine the depth displacement of the target based on the first depth data and the second depth data;

[0122] The second determining submodule projects the image position deviation into three-dimensional space to obtain a three-dimensional planar displacement.

[0123] The third determining submodule is used to determine the three-dimensional displacement of the target based on the three-dimensional planar displacement and the depth displacement;

[0124] The fourth determining submodule is used to determine the three-dimensional velocity of the target from the first time moment to the second time moment based on the three-dimensional displacement.

[0125] In some optional embodiments, the first image and the second image include focal length information;

[0126] The second determining submodule is further configured to: project the image position deviation onto the three-dimensional space based on the first depth data, the second depth data, and the focal length information.

[0127] In some optional embodiments, the motion compensation module includes:

[0128] The first motion compensation submodule is used to determine multiple exposure times corresponding to the multiple related points based on the multiple related point cloud data;

[0129] The second motion compensation submodule is used to determine the time interval between each exposure moment in the plurality of exposure moments, and to correct multiple position information in the plurality of related point cloud data based on the time interval and the three-dimensional velocity.

[0130] In some optional embodiments, the motion compensation module is further configured to: ensure that the time difference between the target time period and the first time and / or the second time satisfies a preset time difference threshold.

[0131] In some optional embodiments, the apparatus further includes:

[0132] The third determining module is used to determine the speed threshold based on the target frame rate in the target frame point cloud data;

[0133] The judgment module is used to perform the step of motion compensation based on the three-dimensional velocity for multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data when the three-dimensional velocity reaches the velocity threshold.

[0134] The apparatus and method embodiments in this application are based on the same application concept.

[0135] Figure 6This is a hardware structure block diagram of an electronic device for implementing a motion compensation method for 3D point cloud data, provided in an embodiment of this application. The electronic device can be a server or a terminal device, and its internal structure diagram can be as follows. Figure 6 As shown. Figure 6 As shown, the electronic device 600 can vary considerably depending on its configuration or performance. It may include one or more Central Processing Units (CPUs) 610 (CPUs 610 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 630 for storing data, and one or more storage media 620 (e.g., one or more mass storage devices) for storing application programs 623 or data 622. The memory 630 and storage media 620 may be temporary or persistent storage. The program stored in the storage media 620 may include one or more modules, each module may include a series of instruction operations on a server. Furthermore, the CPU 610 may be configured to communicate with the storage media 620 and execute the series of instruction operations stored in the storage media 620 on the electronic device 600. Electronic device 600 may also include one or more power supplies 650, one or more wired or wireless network interfaces 650, one or more input / output interfaces 640, and / or one or more operating systems 621, such as Windows, Mac OS, Unix, Linux, FreeBSD, etc.

[0136] The input / output interface 640 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 600. In one example, the input / output interface 640 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 640 may be a radio frequency (RF) module for wireless communication with the Internet.

[0137] The power supply 660 can be logically connected to the processor 610 through the power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0138] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, electronic device 600 may also include... Figure 6 The more or fewer components shown, or having the same Figure 6The different configurations shown.

[0139] Embodiments of this application also provide a computer storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described motion compensation method for three-dimensional point cloud data.

[0140] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0141] The embodiments of this application also provide an electronic device, which includes at least a processor 610 and a memory 630. The memory 630 stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded by the processor 610 and executed to perform the motion compensation method for the three-dimensional point cloud data described above.

[0142] The present application, as described above, acquires a first image, a second image, first point cloud data, and second point cloud data of a target. The first image and the first point cloud data respectively contain first position data and first depth data of a target point at a first time moment. The second image and the second point cloud data respectively contain second position data and second depth data of the target point at a second time moment. It acquires target frame point cloud data of the target over a target time period, which is associated with the first time moment and / or the second time moment. Based on the first position data and the second position data, it determines the image position deviation of the target point. Based on the image position deviation, the first depth data, and the second depth data, it determines the three-dimensional velocity of the target from the first time moment to the second time moment. Based on the three-dimensional velocity, it performs motion compensation on multiple related point cloud data corresponding to multiple related points of the target in the target frame point cloud data. This allows for effective motion compensation of three-dimensional point cloud data without the need for a deep learning model, saving computing power and reducing costs.

[0143] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0144] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0145] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0146] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of motion compensation of three-dimensional point cloud data, characterized in that, The method comprises: acquiring a first image, a second image, first point cloud data and second point cloud data of a target; the first image and the first point cloud data are acquired by a registered image acquisition device and a laser radar at a same first time, and respectively contain first position data and first depth data corresponding to a target point on the target; the second image and the second point cloud data are acquired by the registered image acquisition device and the laser radar at a same second time, and respectively contain second position data and second depth data corresponding to the target point; acquiring target frame point cloud data of the target in a target time period; the target time period is associated with the first time and / or the second time; determining an image position deviation of the target point based on the first position data and the second position data; determining a depth displacement of the target based on the first depth data and the second depth data; projecting the image position deviation to a three-dimensional space based on the image position deviation, the first depth data and focal length information of the image acquisition device, to obtain a three-dimensional plane displacement; determining a three-dimensional displacement of the target from the first time to the second time based on the three-dimensional plane displacement and the depth displacement; determining a three-dimensional velocity of the target based on the three-dimensional displacement; and performing motion compensation on a plurality of related point cloud data corresponding to a plurality of related points of the target in the target frame point cloud data based on the three-dimensional velocity.

2. The method of claim 1, wherein, The motion compensation on the plurality of related point cloud data corresponding to the plurality of related points of the target in the target frame point cloud data based on the three-dimensional velocity comprises: determining a plurality of exposure times corresponding to the plurality of related points based on the plurality of related point cloud data; determining a time interval between each exposure time in the plurality of exposure times, and correcting a plurality of position information in the plurality of related point cloud data based on the time interval and the three-dimensional velocity.

3. The method of claim 1, wherein, The target time period is associated with the first time and / or the second time, comprising: a time difference between the target time period and the first time and / or the second time satisfies a preset time difference threshold.

4. The method of claim 1, wherein, Before the motion compensation on the plurality of related point cloud data corresponding to the plurality of related points of the target in the target frame point cloud data based on the three-dimensional velocity, the method further comprises: determining a velocity threshold based on a target frame rate in the target frame point cloud data; in a case where the three-dimensional velocity reaches the velocity threshold, performing the step of the motion compensation on the plurality of related point cloud data corresponding to the plurality of related points of the target in the target frame point cloud data based on the three-dimensional velocity.

5. The method of claim 1, wherein, The target point is a center of mass of the target.

6. A motion compensation apparatus for three-dimensional point cloud data, characterized by, The device comprises: The first acquisition module is configured to acquire a first image, a second image, first point cloud data, and second point cloud data of a target; the first image and the first point cloud data are collected by a registered image acquisition device and a laser radar at a same first time point, and respectively contain first position data and first depth data of a target point on the target corresponding to the first time point; the second image and the second point cloud data are collected by the registered image acquisition device and the laser radar at a same second time point, and respectively contain second position data and second depth data of the target point corresponding to the second time point; The second acquisition module is configured to acquire target frame point cloud data of the target in a target time period; the target time period is associated with the first time point and / or the second time point; The first determination module is configured to determine an image position deviation of the target point based on the first position data and the second position data; The first determination submodule is configured to determine a depth displacement of the target based on the first depth data and the second depth data; The second determination submodule is configured to project the image position deviation to a three-dimensional space based on the image position deviation, the first depth data, and focal length information of the image acquisition device, to obtain a three-dimensional plane displacement; The third determination submodule is configured to determine a three-dimensional displacement of the target based on the three-dimensional plane displacement and the depth displacement; The fourth determination submodule is configured to determine a three-dimensional velocity of the target from the first time point to the second time point based on the three-dimensional displacement; The motion compensation module is configured to perform motion compensation on a plurality of related point cloud data corresponding to a plurality of related points of the target in the target frame point cloud data based on the three-dimensional velocity.

7. A computer storage medium, characterized in that The computer storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to perform the motion compensation method of the three-dimensional point cloud data according to any one of claims 1-5.

8. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to perform the motion compensation method of the three-dimensional point cloud data according to any one of claims 1-5.

Images