Correction apparatus and method

Abstract

The invention relates to distortion correction of 3D point cloud data, and provides a correction apparatus and method for distortion correction of original point cloud data of a target object collected by a collection device. The correction device comprises a first correction device which is used for correcting the point cloud data so as to compensate for distortion of a dynamic target object caused by motion of the dynamic target object in the collected target objects.

Description

Technical Field

[0001] The present invention relates to information processing, and in particular to correction of environmental detection data. Background Art

[0002] 3D LiDAR (laser radar) has become a very important sensor device in modern autonomous vehicles and robotic applications. Compared with traditional visual cameras, 3D LiDAR can measure the surrounding environment more accurately. The measurement results detected by the 3D LiDAR sensor usually form 3D point cloud data, which can be used to reflect the geometric and physical information of objects or scenes in the surrounding environment in the 3D real world. 3D point cloud data is used as the input of 3D perception algorithms (such as simultaneous localization and mapping (SLAM), 3D object detection / lane detection, etc.) of autonomous vehicles and robots, and can realize the detection and tracking of objects in the environment. Therefore, more accurate 3D point cloud data is crucial for object perception. However, in practice, for example, in autonomous driving, the 3D point cloud data collected by the radar may be distorted due to various reasons, thereby affecting the accuracy of the downstream task execution results. Therefore, the correction of distorted point cloud data is an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a solution for correcting the distortion of 3D point cloud data of a dynamic target object with unknown motion parameters (such as speed, etc.) and estimating its motion state, such as speed, based on deep machine learning. According to one aspect of the present invention, a correction device is provided for performing distortion correction on original point cloud data of a target object acquired by an acquisition device, the correction device comprising: a first correction device for performing a first correction on the point cloud data to compensate for the distortion of the dynamic target object caused by the self-motion of the dynamic target object in the acquired target object.

[0004] In a preferred embodiment, the acquisition device is in a moving state or a stationary state; and the distortion correction device also includes a second correction device, which is used to perform a second correction on the point cloud data based on the movement speed of the acquisition device to compensate for the distortion of the target object caused by the movement of the acquisition device; wherein the first correction device performs the first correction on the point cloud data that has undergone the second correction.

[0005] In a preferred embodiment, the first correction device includes a detection module for performing object detection based on the point cloud data to obtain point cloud data of a dynamic target object; a feature extraction module for extracting a feature vector of the dynamic target object based on the point cloud data of the dynamic target object, wherein the feature vector includes geometric information and motion information of the dynamic object; and a motion compensation module for performing motion compensation on the point cloud data of the dynamic target object based on the feature vector to obtain point cloud data after the first correction.

[0006] In a preferred embodiment, the feature extraction module includes an encoder, configured to compress and encode the point cloud data of the dynamic target object and increase the dimension to output a first feature quantity, wherein the first feature quantity includes the geometric information and motion information of the dynamic target object; a converter, configured to convert the first feature quantity to increase the invariance of the first feature quantity, thereby generating a second feature quantity, wherein the second feature quantity includes the global feature and local feature of the dynamic target object; a decoder, configured to perform extended decoding and dimensionality reduction on the second feature quantity to generate a third feature quantity as the feature vector, wherein the third feature quantity contains multidimensional information about the dynamic target object.

[0007] In a preferred embodiment, the encoder includes at least one first multi-layer perceptual model, and the decoding unit includes at least one second multi-layer perceptual model; wherein the first multi-layer perceptual model compresses the point cloud data of the dynamic object, extracts the features of the point cloud data, and expands the feature dimension of the point cloud data by learning to generate the first feature quantity; wherein the converter includes a conversion unit configured to process the first feature quantity using a symmetric function to obtain the global feature; an information combination unit configured to combine the global feature with the local feature of the dynamic object to generate the second feature quantity, wherein the local feature includes the output of the at least one first multi-layer perceptual model or at least a part of the point cloud data corrected by the second correction device; wherein the second multi-layer perceptual model decompresses the second feature quantity and reduces the feature dimension of each point in the second feature quantity to generate the third feature quantity. wherein the motion compensation module processes the third feature quantity using the trained third multi-layer perceptual model to generate the corrected point cloud.

[0008] In a preferred embodiment, the motion compensation module includes a speed estimation submodule, which is configured to estimate the motion parameters of the dynamic object using a trained speed estimation model based on the feature vector extracted by the feature extraction module; the motion compensation submodule is configured to perform the first correction on the point cloud data of the dynamic object based on the estimated motion parameters using a trained speed compensation model, or to perform the first correction on the point cloud data of the dynamic object based on the feature vector extracted by the feature extraction module using a trained third MLP model.

[0009] In a preferred embodiment, the third MLP model is determined by joint training with the velocity estimation model using a training data set, wherein the training data set includes: a distorted point cloud data sample for each object instance, wherein the distorted point cloud data sample for each object instance is annotated with a correct bounding box corresponding to the true form of the object and the true velocity of the object corresponding to the sample.

[0010] According to another aspect of the present invention, a distortion correction method is provided for performing distortion correction on point cloud data of a target object acquired by an acquisition device, the distortion correction method comprising: performing a first correction on the point cloud data to compensate for the distortion of the dynamic target object caused by the self-motion of the dynamic target object in the acquired target object.

[0011] According to another aspect of the present invention, a machine-readable storage medium is provided, on which machine-readable instructions are stored, wherein the instructions, when executed by one or more processors, cause the processors to perform the correction method according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A diagram schematically showing the relationship between distortion and velocity of a dynamic target object;

[0013] Figure 2 A block diagram schematically shows a distortion correction device according to the present invention;

[0014] Figure 3 A block diagram schematically shows a dynamic distortion correction device according to an example of the present invention;

[0015] Figure 4 A comparison diagram schematically showing object distortion correction;

[0016] Figure 5 A flowchart showing a distortion correction method according to an example of the present invention is shown;

[0017] Figure 6 A flow chart of a dynamic distortion correction method according to an example of the present invention is shown. DETAILED DESCRIPTION

[0018] Since 3D LiDAR is usually installed on a moving car, there may be static target objects such as guardrails or lane lines in the surrounding environment of the moving car, and there may also be dynamic target objects, such as other vehicles with unknown speed states, so the 3D point cloud data collected for these target objects will be distorted. For the convenience of description, the distortion of the 3D point cloud data of the target object caused by the movement of the LiDAR installed on the moving car is called static distortion; and the distortion of the 3D point cloud data of the target object caused by the movement of the target object itself with an unknown motion state (such as unknown speed) is called dynamic distortion. Obviously, when the 3D LiDAR installed on the current vehicle moves with the current vehicle, the distortion of the point cloud data of the dynamic target object (such as other vehicles with unknown motion states in motion) collected by the LiDAR is caused by the movement of the LiDAR and the dynamic target object, which includes both static distortion and dynamic distortion. In another case, for example, when the 3D LiDAR is installed in a fixed position or on a stationary vehicle, the 3D point cloud data of the dynamic target object (such as other moving vehicles) collected by the 3D LiDAR only includes dynamic distortion.

[0019] Usually, since the vehicle's own speed V0 installed by the laser radar can be measured or obtained, for example, it can be detected or read by a vehicle-mounted IMU (inertial measurement unit) or a wheel odometer or a wheel speed sensor, etc., the motion compensation technology can be used to correct the 3D point cloud data of the target object collected by the laser radar based on the measured vehicle speed V0. However, this motion compensation can only compensate for the distortion of the point cloud data caused by the vehicle's own motion, so it is effective for static targets in the vehicle's surroundings and can eliminate the distortion of the point cloud data of the collected stationary targets; but for dynamic target objects in the vehicle's surroundings, there is still dynamic distortion of the point cloud data in the corrected point cloud data, for example, the target object reflected in the point cloud data is "stretched" along the direction of movement of the target object relative to the actual size of the target object. Figure 1 It is clearly shown that when the detected target object moves, its object outline is distorted; and Figure 1 As shown in the figure, the faster the speed of the target object, the greater the degree of distortion of the target object along its moving direction, which is manifested as the size of the target object along its moving direction being stretched longer. Since the moving state of the target object, such as the speed, is unknown, how to correct the distorted point cloud data of such dynamic target objects is a technical problem that needs to be solved in the field of autonomous driving. The correction device and method provided by the present invention can solve the distortion that may exist in the perception data of the moving object.

[0020] In the following description, the embodiment of the present invention is described by taking the dynamic target object as other vehicles in motion as an example; however, the present invention is obviously not limited thereto and can also be applied to any other moving object. In this example, the current vehicle CAR1 is traveling at a speed of V0, and the 3D LiDAR located on the vehicle obtains the 3D point cloud data RawCloud in its irradiation field of view. For example, the 3D point cloud RawCloud may contain N data points, and each data point can be regarded as a point cloud point in the irradiation field of view. There may be various irradiation target objects in the field of view, such as static targets such as isolation belts, and dynamic targets such as another vehicle CAR2 in motion, so the point cloud points obtained for each target object can be used to analyze or detect the corresponding object, including the shape, size and distance of the object, so as to perform subsequent object recognition or tracking, etc. In one example of the present invention, each point in the RawCloud point cloud is composed of a four-dimensional vector P = (x, y, z, i), where P represents the point and x, y, z represent the three-dimensional coordinates of the point in the Cartesian coordinate system in the field of view, and i represents the signal reflection intensity of the point received by the receiver in the 3D LiDAR. Of course, in another example, the three-dimensional coordinates of each point can also be represented by polar coordinates, that is, (α, β, ρ), where α represents the horizontal azimuth, β represents the vertical azimuth, and ρ represents the straight-line distance from the point to the coordinate origin, such as 3D LiDAR. As mentioned above, due to the movement of the vehicle CAR1 and the movement of the target object CAR2, the currently detected point cloud RawCloud has the problem of object distortion.

[0021] Figure 2 A block diagram of a distortion correction device according to an example of the present invention is shown. As shown in the figure, the distortion correction device 100 can be installed on the vehicle CAR1 to receive the detected 3D point cloud data {Pi} from the on-board 3D LiDAR. As an example, the distortion correction device 100 may include a static correction device 200 and a dynamic correction device 300. The static correction device 200 first performs static distortion compensation on the point cloud data RawCloud based on the motion parameters of the vehicle CAR1 when collecting the 3D point cloud RawCloud, such as speed V0 or heading angle, obtained from devices such as IMU or wheel odometer, so as to eliminate the influence of the movement of CAR1 on the point cloud RawCloud. The motion compensation technology known in the prior art can be used here to achieve static distortion compensation for the point cloud. For example, according to the speed and heading angle of the vehicle CAR1, the position change of the vehicle is compensated by performing displacement compensation and angle compensation, thereby achieving compensation for the distortion of the point cloud RawCloud caused by the movement of CAR1. Here, the point cloud data formed by static compensation of the original point cloud RawCloud is marked as StaticCloud, such as Figure 2 shown.

[0022] The dynamic correction module 300 receives the statically compensated point cloud data StaticCloud. It is not difficult to understand that for the static target objects in the surrounding environment contained in the point cloud StaticCloud, such as lane lines or lamp posts, after the static compensation processing of the static correction device 200, the point cloud data reflecting these static target objects have achieved distortion correction, while for dynamic targets in the environment, such as other moving vehicles CAR2, their point cloud data still have dynamic distortion caused by the self-motion of CAR2. Therefore, according to an embodiment of the present invention, the dynamic correction module 300 eliminates the dynamic distortion of the point cloud data of CAR2 caused by the self-motion of the target vehicle CAR2 by compensating the point cloud data StaticCloud corrected by the static correction device 200.

[0023] like Figure 2 As shown, in one example of the present invention, the dynamic correction module 300 includes a detection module 301, a feature extraction module 302 and a motion compensation module 303. The detection module 301 performs object detection based on the received StaticCloud to obtain point cloud data of dynamic objects therein. For example, the properties of the target object contained therein can be determined for the received StaticCloud, and the properties include motion properties, such as whether it is a dynamic object or a static object. Here, the detection module 301 can use a predetermined rule or algorithm known in the prior art to identify each object instance in the point cloud, for example, based on the maximum distance algorithm, a point cluster that meets certain conditions in the point cloud (for example, points whose distance between two points does not exceed a certain maximum distance are classified as a cluster) is identified as a target object instance, and further determines whether the object represented by the cluster has motion properties, for example, by comparing with an object template to determine whether the current object belongs to a moving object, such an object template may include data of various types of moving objects, such as cars, trucks, etc.; it can also be based on neural networks or machine learning to preliminarily identify these object classifications, so as to determine whether it is a moving car or a fixed building, etc. The object detection and classification methods known in the prior art are all applicable to the present invention, and will not be repeated here.

[0024] For each object instance identified from the point cloud data StaticCloud, the detection module 301 outputs the detection result of the target object, such as the point cloud data belonging to the target object defined by the bounding box BOX, that is, the pixel cluster belonging to the target object instance is contained in the bounding box, which is hereinafter referred to as ObjectCloud for ease of description. As an example, the bounding box BOX is composed of the coordinates of the center position of the box (x 0 ,y 0 ,z 0) and the length, width, height (l, w, h) of the box and its direction angle (θ). According to an example of the present invention, the detection module 301 can further determine whether the object in each bounding box may be distorted based on the classification attributes of the determined target object. For example, for a fixed building, such as a lane line or a lamp post, it is determined that the corresponding point cloud does not have distortion. If the target object is a vehicle on a driving lane, such as CAR2, it can be determined to be a dynamic object, and therefore it is determined that the point cloud ObjectCloud corresponding to CAR2 may have dynamic distortion, that is, the bounding box BOX defining CAR2 is elongated and deformed at this time. Therefore, the detection module 301 outputs the point cloud data ObjectCloud representing the dynamic object (such as CAR2) for subsequent dynamic distortion correction processing. It is assumed here that the point cloud data ObjectCloud contains n data points of CAR2, that is, ObjectCloud={Pi|i=1,...,n}. In the following description, CAR2 is used as an example to describe the dynamic target object.

[0025] The feature extraction module 302 is trained to extract feature vectors Vector of the point cloud data ObjectCloud of the dynamic object CAR2, including extracting feature information of different aspects or attributes of the point cloud from the point cloud data ObjectCloud. For example, the feature information can reflect the geometric information and motion information of the dynamic object CAR2. Figure 2As shown, in order to extract the feature vector Vector of the point cloud ObjectCloud, according to an example, the feature learning module 302 includes an encoder 3021, a converter 3022, and a decoder 3023. The encoder 3021 receives the point cloud data ObjectCloud of the dynamic target CAR2 from the detection module 301. Then the encoder 3021 performs compression encoding and dimension upgrading operations on the point cloud data ObjectCloud to output the feature quantity Enc_Tensor. In an example, the encoded feature quantity En_Tensor contains the basic geometric information and motion information of the moving target, such as the size, shape, and motion direction of the vehicle. It should be pointed out here that when the encoder 3021 performs encoding, it can not only obtain some attribute index information about CAR2 contained in the ObjectCloud point cloud, such as size, shape, etc., but also be extended to include some other index information calculated based on its point cloud, such as multiple output vectors of a machine learning model learned by executing deep learning (Deep Learning), such as Normal and other features of each point in the point cloud. In addition, according to the example of the present invention, when encoding the ObjectCloud point cloud, the encoder 3021 further compresses the amount of point cloud data contained in the ObjectCloud. As mentioned above, the ObjectCloud of CAR2 contains n data points. After compression, the encoder 3021 outputs m data points, where m is less than n. At the same time, in the dimensionality increase operation, as mentioned above, each data point contains more dimensional indicator information that has been expanded through encoding.

[0026] The converter 3022 is configured to convert the encoded feature quantity Enc_Tensor and generate a converted feature quantity Tran_Tensor. The purpose of the conversion here is to increase the invariance of the encoded feature quantity Enc_Tensor, and the conversion includes processing a point cloud with m data points to collect its global information. As an example, for example, the converter 3022 extracts the maximum index value of different indicators of the m data points as the global information of the m data points. In addition, in addition to including global information in the converted feature quantity Tran_Tensor, the converter 3022 also includes local information of the target object CAR2 in the converted feature quantity Tran_Tensor so as to better reflect the differentiation of object instances. The local information here can be, for example, the received ObjectCloud point cloud data or the feature information generated during the encoding process of the encoder 3021.

[0027] The decoder 3023 performs extended decoding and dimensionality reduction operations on the conversion feature quantity Tran_Tensor to generate a decoded feature quantity Dec_Tensor as the feature vector Vector output by the feature learning module 302. Here, after the decoder 3023 learns the conversion feature quantity Tran_Tensor, the feature quantity Dec_Tensor contains multi-dimensional feature information for the dynamic object CAR2; as an example, the multi-dimensional feature information here may include information such as the size, position, motion parameters or semantic category of the dynamic object CAR2. Since the decoding process is actually the inverse processing of the encoding process, the feature quantity Dec_Tensor can have the same data volume and dimension number as the ObjectCloud point cloud data received by the encoder 3021, that is, n×4; but the present invention is not limited to this, and the feature quantity Dec_Tensor learned by the decoder 3023 can also have different data volumes and dimensions, such as a data volume greater than or less than n and / or a dimension number greater than or less than 4.

[0028] The motion compensation module 303 is used to perform motion compensation on the feature vector Vector, i.e., the feature quantity Dec_Tensor generated by the decoder 3023 in this example, to generate corrected point cloud data, hereinafter referred to as ObjectCloud′, wherein the dynamic distortion of CAR2 caused by the self-motion of CAR2 in its point cloud data ObjectCloud is eliminated in the corrected point cloud data ObjectCloud′. Considering that it is difficult to establish an effective linear or nonlinear relationship between the degree of distortion of the point cloud collected by 3D LiDar and the speed of the moving target object in reality, it is difficult to use quantitative parameters to characterize the relationship between the two. Therefore, in this embodiment, the point cloud data is deeply learned by training a neural network, so as to automatically correct the distortion in the point cloud data. Therefore, in this example, the motion compensation module 303 can call the motion compensation neural network model MCM trained based on deep learning to directly process the point cloud data of CAR2 based on the feature quantity Dec_Tensor, so as to generate the corrected point cloud data ObjectCloud′ of the target object, such as CAR2, which eliminates the dynamic distortion, and is manifested as the bounding box BOX′ corresponding to ObjectCloud′ is corrected at this time, so as to adapt to the real physical size of the target object CAR2. ObjectCloud′, as the final data after dynamic distortion compensation, can be provided to subsequent applications, such as further identification or tracking of CAR2. According to a further embodiment of the present invention, the relationship between the distorted point cloud and the speed of the moving target object can be deeply learned by training a neural network, so as to estimate the speed V of the moving target. To this end, the dynamic correction device 300 can further include a speed estimation module 304, which is configured to call a speed estimation model VM that has learned the relationship between the distorted point cloud and the speed of the moving target object, and estimate the speed V of the dynamic target CAR2 by processing the feature quantity Dec_Tensor.

[0029] Figure 3 FIG. 3 is a partial schematic diagram of an exemplary embodiment of the dynamic correction device 300 for implementing the above-mentioned dynamic correction device 300, in which only the encoder 3021, the converter 3022, the decoder 3023, the motion compensation module 303 and the speed estimation module 304 are shown. Figure 3As shown, the encoder 3021 includes a first-level encoding unit mlp (4, 32) and a second-level encoding unit mlp (32, 64). Here, the encoding unit mlp (4, 32) and the encoding unit mlp (32, 64) are implemented by a trained multi-layer perceptron neural network (mlp) model, which is used to compress the input point cloud data step by step, extract the features of the point cloud data, and expand the feature dimension of the point cloud data to a higher dimension through learning. As shown in the figure, the encoding unit mlp (4, 32) receives the point cloud data after static compensation processing from the detection module 301. For example, in this example, it receives the point cloud data ObjectCloud containing the dynamic target object CAR2. As mentioned above, each point in the point cloud ObjectCloud can be a four-dimensional data point (x, y, z, i) containing three-dimensional coordinates and signal strength, that is, n×4, where n represents the number of data points in the point cloud ObjectCloud of CAR2. The encoding unit mlp (4, 32) compresses the data volume of the point cloud ObjectCloud from n to m by, for example, downsampling. 1 , here m 1 Less than n, and directly perform neural network operations on the point cloud data to extract the multi-dimensional feature information of the ObjectCloud point cloud, for example, 32 dimensions as shown in the figure, thereby outputting m 1 ×32 data point cloud. In another example, the encoding unit mlp(4,32) can also be designed to directly compress and learn the ObjectCloud point cloud data, thereby directly outputting m 1 ×32 data.

[0030] mlp(4,32) will generate m 1 ×32 point cloud data is provided to the next level coding unit mlp(32,64). Similar to mlp(4,32), mlp(32,64) 1 ×32 point cloud data to perform neural network operations and continue to extract multi-dimensional information of data points, for example, 64 dimensions as shown in the figure, and at the same time the data volume is increased from m 1 Further compression to m 2 , here m 2 Less than m 1 In another example, the coding unit mlp(32,64) can also be designed to directly 1 ×32 point cloud data is compressed and learned, so that m 2 64-dimensional data m 2 ×64. mlp(32,64) will generate m 2 The ×64 point cloud data is output as the encoded feature value Enc_Tensor of the encoder 3021. It should be noted here that Figure 3Only two encoding units mlp (4, 32) and encoding unit mlp (32, 64) of the encoder 3021 are schematically shown, but it does not exclude that the encoder 3021 may also selectively include other components, which may be necessary or preferred for the interconnection and conversion between MLPs. For example, a joint alignment unit or network (JointAlignment Network) is set before executing MLP or between the first and second level encoding operations to align the point cloud in space and realize the alignment of the input point and the point feature. The purpose of setting the joint alignment network is to take into account that the target object detected by LiDar has spatial invariance, so by aligning the input and the features of different points, the expression learned by the network also has the characteristics of this rigid transformation. Here, the T_Net affine transformation matrix commonly used in the prior art can be used to implement the joint alignment network, but it will not be repeated here. In addition, in this example, the encoder 3021 includes a secondary encoding unit, but it can also include more or fewer encoding units according to needs, such as the dimension of the point cloud to be encoded.

[0031] The converter 3022 includes a conversion unit 3031 and an information combination unit 3032, wherein the conversion unit 3031 receives the Enc_Tensor output by the encoder 3021, that is, (m 2 ×64) data and extract the global feature information therein. As an example, the conversion unit 3031 may use a symmetric function such as a maximum pooling function (Max Pooling) or other functions to extract (m 2 ×64) data points and generates 64-dimensional single-point features (1×64), which are provided to the combination unit 3032 as global features.

[0032] The combining unit 3032 receives the global features (1×64) of the point cloud provided by the information extracting unit 3031 and the data output by one of the encoding units in the encoder 3021 before the converter 3022 operates. For example, the data (m 2 ×64), obviously data (m 2 ×64) contains more local feature information of the object CAR2. The combining unit 3032 combines the global feature (1×64) with the local information (m 2 ×64), forming the conversion feature quantity Tran_Tensor(m 2 ×128). It should be noted here that, in other examples, the combining unit 3024 may also directly combine at least a portion of the point cloud data in the point cloud data ObjectCloud as local feature information with the global feature to form the conversion feature quantity Tran_Tensor.

[0033] In another example, a joint alignment network (not shown in the figure) can be further set in the converter 3022, for example, before or after the conversion unit 3031 extracts the global feature information, so as to align the input points with the point features. For example, the joint alignment network can receive the point cloud ObjectCloud from CAR2 as input, thereby enhancing the feature invariance during the conversion process. Here, the T_Net affine transformation matrix commonly used in the prior art can be used to implement the joint alignment network, but it will not be repeated here.

[0034] The decoder 3023 receives the conversion feature quantity output by the converter 3022, for example, Tran_Tensor(m 2 ×128), expand the point cloud data volume in the conversion feature quantity Tran_Tensor and compress the feature dimension of each point in Tran_Tensor to generate a decoded feature quantity Dec_Tensor. As shown in the figure, the decoder 3023 includes a first-level decoding unit mlp(128, 64) and a second-level decoding unit mlp(64, 4). Here, the decoding unit mlp(128, 64) and the decoding unit mlp(64, 4) are implemented by a trained multi-layer perceptron neural network mlp model, which can gradually expand the point cloud data volume in the conversion feature quantity Tran_Tensor and gradually compress the feature dimension of each point in Tran_Tensor. Specifically, the decoding unit mlp(128, 64) receives the feature quantity Tran_Tensor(m 2 ×128), and decode it into 64-dimensional data points, such as m 1 ×64, here m 1 Greater than m 2 Here the decoding unit mlp(128,64) can be designed to directly decode m 2 ×128 data for extended learning, thus directly outputting m 1 64-dimensional data m 1 ×64; or in another example, the decoding unit may also first decode (m 2 ×128) data, and then perform a data compression operation. Then the decoding unit mlp(64,4) performs a dimensionality reduction operation on the (m 1 ×64) decoded data is further processed and decoded into n×4 feature quantity Dec_Tensor. Here, the decoding unit mlp(64,4) can be designed to directly decode m 1 ×64 data, thereby directly outputting n×4 point cloud data; in another example, the decoding unit may also first perform (m 1×64) data, and then perform a data compression operation. It should be noted here that the decoded feature quantity Dec_Tensor obtained by the step-by-step decompression of the transformed feature quantity Tran_Tensor implemented by the decoder 3023 can have the same data volume and dimension as the original ObjectCloud point cloud, for example, n×4 in this example; but it can also be implemented with different data volumes and / or dimensions, for example, the feature quantity Dec_Tensor can be m 3 ×d, where m 3 represents the number of data points output by the decoder and m 3 It can be greater than or less than n, and d represents the dimension of each data point and can be greater than or less than the initial dimension 4 of the original point cloud ObjectCloud.

[0035] It should be noted here that since the combination unit 3032 can also be designed to combine the global feature (1×64) with other local information to form Tran_Tensor, for example, it can be combined with the (m 1 ×32) data to form (m 1 Therefore, the decoder 3023 also adjusts its decoding units at each level accordingly to adapt to the dimension change of Tran_Tensor.

[0036] The decoder 3023 provides the output feature quantity Dec_Tensor to the motion compensation module 303. According to an example of the present invention, considering that it is difficult to establish an effective linear or nonlinear relationship between the degree of distortion of the point cloud collected by 3D LiDar and the speed of the moving target object in reality, in this embodiment, a motion compensation model MCM is trained to perform deep learning on the point cloud data. The motion compensation module 303 can call the trained motion compensation model MCM to process the point cloud data in the Dec_Tensor output by the decoder 3023, thereby generating corrected point cloud data ObjectCloud′. Here, the motion compensation model MCM can be implemented using a multi-layer perception network architecture known in the prior art. As an example, the multi-layer perception network architecture may include a fully connected network layer, a rectified linear unit (ReLU), and preferably includes a Dropoutn layer to improve the generalization ability of the model and regularization to reduce the risk of overfitting. Figure 4The diagram schematically shows the bounding box change of the distortion compensation processing performed on CAR2. The box indicated by the dotted line indicates the bounding box BOX of the point cloud data ObjectCloud of CAR2 output by the detection module 301. Obviously, the box BOX is elongated compared with the size of the actual vehicle CAR2. The box shown by the solid line in the figure represents the updated bounding box BOX′ output by the motion compensation module 303, corresponding to the point cloud data ObjectCloud′ after distortion correction. It can be seen that the size of the box BOX′ is compressed, so that it matches the actual vehicle size. It should be pointed out here that the elliptical box in the figure represents the distribution range of the point cloud data ObjectCloud actually detected and output by the detection module 301, and the boxes BOX and BOX′ in the figure represent the boundary contours of the target vehicle determined before and after the corresponding point cloud correction.

[0037] In this example, the speed estimation unit 304 can call the trained and learned speed estimation model VM to process the point cloud data in the Dec_Tensor output by the decoder 3023, thereby generating a speed estimation V for the vehicle CAR2, including information such as the lateral speed and longitudinal speed of CAR2. The speed V can be provided as a reference parameter for subsequent tasks, wherein the speed estimation model VM can be generated by training using a multi-layer perception speed estimation neural network model.

[0038] In another embodiment of the present invention, the motion compensation module 303 does not generate the corrected point cloud ObjectCloud′ by directly processing the feature quantity Dec_Tensor using the training model MCM, but uses the motion parameters of CAR2 provided by the speed estimation unit 304, such as the speed estimation V or acceleration information, to eliminate the dynamic distortion in the initial point cloud ObjectCloud. To this end, after the speed estimation module 304 uses the trained multi-layer perceptual speed estimation model VM to process the point cloud data in the received decoded feature quantity Dec_Tensor and gives the motion parameters of the target CAR2, such as the speed estimation V, the motion compensation module 303 can use the estimated motion parameters such as the speed V to process the ObjectCloud point cloud data of CAR2 output by the detection module 301 to eliminate the dynamic distortion therein, thereby generating the corrected point cloud data ObjectCloud′. Here, the motion compensation technology known in the prior art can be used to realize the distortion compensation of CAR2 based on the speed estimation V. Therefore, the use of this solution can reduce the cost of the machine learning motion compensation model MCM. In this example, the speed estimation module 304 and the motion compensation module 303 may exist separately, but in another example, the speed estimation module 304 may also be integrated with the motion compensation module 303 as submodules to form a motion compensation module.

[0039] It should be pointed out here that in order to realize accurate correction of point cloud data distortion by the motion compensation model MCM, according to the present invention, the motion compensation model MCM and the speed estimation model VM can be jointly trained to generate the model MCM, and the speed estimation model VM can be determined accordingly. During the training process, the same LiDar sensor can be used to collect point cloud data samples in multiple scenes of multiple different object instances, and different LiDar sensors can be used to collect point cloud data samples in multiple scenes of multiple different object instances, wherein the correct bounding box corresponding to the real form of the object (i.e., the point cloud data corresponding to the real form object) and the real speed of the object corresponding to the sample are annotated on the distorted point cloud data sample of each object instance, as the prediction target or label of the model MCM and VM; thereby ensuring that the prediction results output by the model MCM and VM in training can simultaneously meet the dual requirements of correct correction of distortion and true estimation of speed.

[0040] It should be pointed out here that in the description of the above embodiments, the data acquisition device, such as LiDar, is in a moving state, such as a moving vehicle, for example. However, in other cases, for example, LiDar is located on a fixed facility, such as a building or a signal lamp post, then there is no static distortion in the point cloud used by the LiDar. Therefore, the distortion correction device 100 according to the present invention does not require a static correction device 200. In addition, in other application scenarios, the configuration of the dynamic correction device 300 is also variable. For example, the dynamic correction device 300 can directly receive the point cloud ObjectCloud for a dynamic target object provided externally, so the detection module 301 is not necessary at this time. Therefore, the configuration or structure of the distortion correction device 100 according to the present invention can be adapted according to actual needs and design purposes.

[0041] Figure 5 A flow chart of a data distortion correction method according to an embodiment of the present invention is shown, and the method is used to perform distortion correction on raw point cloud data of a target object collected by a collection device such as a laser radar. As shown in the figure, first, in step 501, the raw point cloud data RawCloid is corrected using the speed V0 of the vehicle CAR1 carrying the laser radar to compensate for the static distortion of the target object caused by the movement of the laser radar, and a statically corrected point cloud StaticCloud is output, where the target object includes a static target and a dynamic target in motion.

[0042] Then, in step 503, object detection is performed on the statically corrected point cloud StaticCloud to determine the target object instances contained therein, wherein data distortion correction is achieved for the detected static target objects; and for dynamic objects, such as another detected moving vehicle CAR2, since it may contain dynamic distortion, the point cloud ObjectCloud of the dynamic object, such as CAR2, is output.

[0043] As mentioned above, if the acquisition device such as LiDar is in a fixed position or static state, step 501 is not necessary, and the method example of the present invention can directly receive the point cloud ObjectCloud determined as a dynamic target object to perform the process of eliminating dynamic distortion.

[0044] In step 505, the point cloud ObjectCloud of the dynamic object CAR2 output in step 503 is processed to compensate for the dynamic distortion in the point cloud of CAR2 caused by the motion of CAR2 itself. Figure 6 The flowchart of an exemplary dynamic distortion correction method is shown.

[0045] like Figure 6 As shown, in step 601, a point cloud ObjectCloud of a target object CAR2 that has been subjected to static distortion processing is received, for example, ObjectCloud includes n point cloud points or data points of the object CAR2.

[0046] In step 603, the point cloud data in ObjectCloud is compressed and dimensionalized to output a feature value Enc_Tensor, wherein when encoding the ObjectCloud point cloud, the point cloud data contained in the ObjectCloud is compressed, for example, n data points in the ObjectCloud are compressed into m data points, where m is less than n, and each data point contains more dimensional indicator information expanded by encoding, thereby forming a feature value Enc_Tensor. In one example, the following can be used: Figure 3 The multi-level coding unit mlp (4, 32) and coding unit mlp (32, 64) composed of the multi-layer perceptron neural network MLP model shown in the figure compress and dimensionally expand the input ObjectCloud point cloud data. First, the coding unit mlp (4, 32) receives the point cloud data ObjectCloud of CAR2, directly performs neural network operations on the point cloud data in ObjectCloud, and increases the data points in BOX to 32 dimensions. At the same time, the data volume is compressed from n to m by methods such as downsampling. 1 , thus outputting (m 1 ×32) point cloud data. Furthermore, mlp(32, 64) is used for (m1 ×32) point cloud data to perform neural network operations and continue to extract 64-dimensional information of data points. 1 Compress to m 2 , here m 2 Less than m 1 , thus outputting (m 2 ×64) point cloud data is output as the feature vector Enc_Tensor.

[0047] In step 605, the feature vector Enc_Tensor is transformed to increase the invariance of the first feature quantity, thereby generating a feature vector Tan_Tensor, where the transformed vector Tan_Tensor includes the global feature information and local feature information of the point cloud data of CAR2. In one implementation, a maximum pooling function (Max Pooling) or other symmetric functions can be used to extract the (m 2 ×64) data points and generate 64-dimensional single-point features (1×64) as global features. Then, the global features (1×64) are combined with the data point cloud before the conversion operation, such as the data point (m 2 ×64) to form (m 2 ×128) point cloud data as the conversion feature value Tran_Tensor.

[0048] In step 607, the conversion feature quantity Tran_Tensor(m 2 ×128) performs extended decoding, expands the point cloud data in the conversion feature quantity Tran_Tensor and compresses the feature dimension of each point in Tran_Tensor to generate a decoded feature quantity Dec_Tensor, wherein Dec_Tensor contains multi-dimensional feature information for the dynamic object CAR2. According to an example of the present invention, the decoding process here is implemented by a multi-level decoding unit composed of a trained multi-layer perceptron neural network MLP model, for example, a decoding unit mlp (128, 64) and a decoding unit mlp (64, 4) are used to expand the point cloud data in the conversion feature quantity Tran_Tensor step by step and compress the feature dimension of each point in Tran_Tensor step by step. For example, mlp (128, 64) is used to receive the combined point cloud (m 2 ×128), and decode it into 64-dimensional data points, such as m 1 ×64, here m 1 Greater than m 2 Then mlp(64,4) is used to 1×64) point cloud data is further processed and decoded into 4-dimensional data points, i.e. n×4, as the decoded feature value Dec_Tensor output by the decoding operation, where n is the amount of data contained in the original ObjectCloud and is greater than m 1 It should be noted here that the amount of point cloud data obtained by decompressing the transformation feature Tran_Tensor step by step may be different from the original ObjectCloud point cloud. 3 with dimension d, where m 3 can be smaller or larger than n, and d can be larger or smaller than the initial dimension 4 of the original ObjectCloud midpoint.

[0049] In step 609, motion compensation is performed on Dec_Tensor to generate corrected point cloud data ObjectCloud′ of CAR2, thereby eliminating the dynamic distortion caused by CAR2. In this embodiment, the relationship between the degree of distortion of the point cloud data and the motion (e.g., speed) of the dynamic target is deeply learned by training the multi-layer perceptual neural network mlp, thereby directly processing the point cloud data in the decoded amount Dec_Tensor by calling the multi-layer perceptual motion compensation model MCM learned through such training, thereby generating corrected point cloud data ObjectCloud′. In addition, the multi-layer perceptual speed estimation model VM learned through training can also be called to process the point cloud data in the decoded output Dec_Tensor, thereby generating a speed estimate V for the vehicle CAR2, which can be used as a reference parameter for subsequent tasks.

[0050] In another embodiment of the present invention, in step 609, the trained multi-layer perceptual speed estimation model VM is first used to process the point cloud data in the received decoded feature value Dec_Tensor and give a speed estimate V of the target CAR2, and then the speed estimate V is used to process the ObjectCloud point cloud data of CAR2 to eliminate the dynamic distortion therein, thereby generating the corrected CAR2 point cloud data ObjectCloud′, thereby reducing the cost of the machine learning MCM model.

[0051] In the above embodiment, although the concept and specific embodiment of the present invention for compensating the distortion of a dynamic target object are illustrated by taking the moving car CAR2 as an example, it is obvious that the present invention is not limited thereto, but is applicable to any situation involving dynamic distortion, including walking robots, etc. In addition, although the exemplary method and module architecture of the present invention are described above in conjunction with specific examples, it can be understood that the method steps and components described here can be implemented as electronic hardware, computer software, or a combination of the two. In addition, a machine-readable medium provided in another embodiment of the present invention stores machine-readable instructions, and when the machine-readable instructions are executed by the processor, the processor executes any of the aforementioned methods disclosed herein. In addition, the present invention is shown and described in detail above through the accompanying drawings and preferred embodiments, but the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that more embodiments of the present invention can be obtained by combining the above different embodiments, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A correction device for performing distortion correction on point cloud data of a target object acquired by an acquisition device, the correction device comprising: The first correction device is used to perform a first correction on the point cloud data to compensate for the distortion of the dynamic target object caused by the self-motion of the dynamic target object among the collected target objects.

2. The calibration device as claimed in claim 1, wherein: The acquisition device is in a moving state or a stationary state.

3. The calibration device of claim 2, further comprising: a second correction device, configured to perform a second correction on the point cloud data based on a movement speed of the acquisition device to compensate for a distortion of the target object caused by the movement of the acquisition device; The first correction device performs the first correction on the point cloud data that has undergone the second correction.

4. The calibration device according to any one of claims 1 to 3, wherein the first calibration device comprises: A detection module, configured to perform object detection based on the point cloud data to obtain point cloud data of a dynamic target object; A feature extraction module, used to extract a feature vector of the dynamic target object based on the point cloud data of the dynamic target object, wherein the feature vector includes geometric information and motion information of the dynamic object; A motion compensation module is used to perform motion compensation on the point cloud data of the dynamic target object based on the feature vector to obtain the point cloud data after the first correction.

5. The correction device as claimed in claim 4, wherein the feature extraction module comprises: An encoder configured to compress and encode the point cloud data of the dynamic target object and perform dimension upgrading to output a first feature quantity, wherein the first feature quantity includes geometric information and motion information of the dynamic target object; a converter configured to convert the first feature quantity to increase the invariance of the first feature quantity, thereby generating a second feature quantity, wherein the second feature quantity includes a global feature and a local feature of the dynamic target object; A decoder is configured to perform extended decoding and dimensionality reduction on the second feature quantity to generate a third feature quantity as the feature vector, wherein the third feature quantity contains multi-dimensional information about the dynamic target object.

6. The correction device of claim 5, wherein the encoder comprises at least one first multi-layer perceptual model, and the decoding unit comprises at least one second multi-layer perceptual model; The first multi-layer perception model generates the first feature quantity by compressing the point cloud data of the dynamic object, extracting features of the point cloud data, and expanding the feature dimension of the point cloud data through learning; The converter comprises: a conversion unit, configured to process the first feature quantity using a symmetric function to obtain the global feature; an information combining unit configured to combine the global feature with a local feature of the dynamic object to generate the second feature quantity, wherein the local feature comprises an output of the at least one level first multi-layer perception model or at least a portion of the point cloud data corrected by the second correction device; wherein the second multi-layer perception model decompresses the second feature quantity and reduces the feature dimension of each point in the second feature quantity to generate the third feature quantity; and The motion compensation module processes the third feature quantity using a trained third multi-layer perception model to generate a corrected point cloud.

7. The correction device of claim 6, wherein the symmetric function comprises a maximum pooling function, and the converter further comprises: The joint alignment unit is used to receive the point cloud data of the dynamic target object so as to align the input points with the point features.

8. The calibration device of claim 7, wherein the object point cloud data of the dynamic object output by the detection module is defined by a first bounding box, including the position, size and orientation angle of the dynamic object; The corrected point cloud data output by the motion compensation module is defined by a second bounding box, wherein: A length of the second bounding box along a moving direction of the dynamic object is smaller than a length of the first bounding box along the moving direction of the dynamic object.

9. The calibration device of claim 6, further comprising: The speed estimation module is configured to estimate the speed of the dynamic target object using a trained speed estimation model based on the feature vector extracted by the feature extraction module.

10. The correction device of claim 6, wherein the motion compensation module comprises: a speed estimation submodule configured to estimate the motion parameters of the dynamic object using a trained speed estimation model based on the feature vector extracted by the feature extraction module; The motion compensation submodule is configured to perform the first correction on the point cloud data of the dynamic object based on the estimated motion parameters using a trained velocity compensation model, or to perform the first correction on the point cloud data of the dynamic object based on the feature vector extracted by the feature extraction module using a trained third MLP model.

11. The correction device according to claim 9 or 10, wherein the third MLP model is determined by joint training with the speed estimation model using a training data set, wherein the training data set comprises: A distorted point cloud data sample of each object instance, wherein the distorted point cloud data sample of each object instance is annotated with a correct bounding box corresponding to the true form of the object and a true speed of the object corresponding to the sample.

12. The correction device of claim 5, wherein the multi-dimensional information includes size information, position information, motion information and semantic category information of the dynamic object.

13. A correction method for performing distortion correction on point cloud data of a target object acquired by an acquisition device, the correction method comprising: A first correction is performed on the point cloud data to compensate for the distortion of the dynamic target object caused by the self-motion of the dynamic target object among the acquired target objects.

14. The calibration method according to claim 13, wherein: The acquisition device is in a moving state or a stationary state.

15. The calibration method of claim 14, further comprising: performing a second correction on the point cloud data based on the movement speed of the acquisition device to compensate for the distortion of the target object caused by the movement of the acquisition device; The first correction is performed on the point cloud data that has undergone the second correction.

16. The calibration method according to any one of claims 13 to 15, wherein the first calibration comprises: Performing object detection based on the point cloud data to obtain point cloud data of a dynamic target object; Extracting a feature vector of the dynamic target object based on the point cloud data of the dynamic target object, wherein the feature vector includes geometric information and motion information of the dynamic object; Motion compensation is performed on the point cloud data of the dynamic target object based on the feature vector to obtain the first corrected point cloud data.

17. The correction method of claim 16, wherein said extracting a feature vector of said dynamic target object comprises: Using an encoder to compress and encode the point cloud data of the dynamic target object and perform dimension upgrading to output a first feature quantity, wherein the first feature quantity includes geometric information and motion information of the dynamic target object; Converting the first feature quantity to increase the invariance of the first feature quantity, thereby generating a second feature quantity, wherein the second feature quantity includes a global feature and a local feature of the dynamic target object; The second feature quantity is subjected to expansion decoding and dimensionality reduction by a decoder to generate a third feature quantity as the feature vector, wherein the third feature quantity contains multi-dimensional information on the dynamic target object.

18. The correction method of claim 17, wherein the encoder comprises at least one first multi-layer perceptual model, and the decoder comprises at least one second multi-layer perceptual model; The first multi-layer perception model generates the first feature quantity by compressing the point cloud data of the dynamic object, extracting features of the point cloud data, and expanding the feature dimension of the point cloud data through learning; The converting of the first feature quantity comprises: Processing the first feature quantity using a symmetric function to obtain the global feature; Combining the global feature with the local feature of the dynamic object to generate the second feature quantity, wherein the local feature includes at least a portion of the output of the at least one level first multi-layer perception model or the point cloud data corrected by the second correction device; wherein the second multi-layer perception model decompresses the second feature quantity and reduces the feature dimension of each point in the second feature quantity to generate the third feature quantity; and The performing of motion compensation on the point cloud data of the dynamic target object includes: processing the third feature quantity using a trained third multi-layer perception model to generate a corrected point cloud.

19. The correction method of claim 18, wherein the symmetric function comprises a maximum pooling function, wherein converting the first feature quantity further comprises: Point cloud data of the dynamic target object is received and input points are aligned with point features using a joint alignment network.

20. The correction method of claim 19, wherein the object point cloud data of the dynamic object obtained by the detection is defined by a first bounding box, including the position, size and orientation angle of the dynamic object; The corrected point cloud data is defined by a second bounding box, wherein: A length of the second bounding box along a moving direction of the dynamic object is smaller than a length of the first bounding box along the moving direction of the dynamic object.

21. The calibration method of claim 18, further comprising: Based on the extracted feature vector, a trained velocity estimation model is used to estimate the velocity of the dynamic target object.

22. The correction method of claim 18, wherein the motion compensation comprises: estimating motion parameters of the dynamic object using a trained velocity estimation model based on the extracted feature vectors; The first correction is performed on the point cloud data of the dynamic object based on the estimated motion parameters using a trained velocity compensation model, or the first correction is performed on the point cloud data of the dynamic object based on the extracted feature vector using the trained third MLP model.

23. The correction method of claim 21 or 22, wherein the third MLP model is determined by joint training with the velocity estimation model using a training data set, wherein the training data set comprises: A distorted point cloud data sample of each object instance, wherein the distorted point cloud data sample of each object instance is annotated with a correct bounding box corresponding to the true form of the object and a true speed of the object corresponding to the sample.

24. The correction method of claim 17, wherein the multi-dimensional information includes size information, position information, motion information and semantic category information of the dynamic object.

25. A machine-readable storage medium having machine-readable instructions stored thereon, wherein the instructions, when executed by one or more processors, cause the processors to perform the correction method of any one of claims 13-24.

26. A calibration device comprising: a storage medium having a machine-readable program stored thereon; At least one processor is configured to implement the correction method of one of claims 13-24 by executing the readable program.

Images