Point cloud data prediction method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a point cloud data prediction method, electronic equipment and a computer readable storage medium. According to the method, the current aerial view of the moving object at the current moment and the historical aerial view of the moving object at the historical moment are obtained, and the historical moment is earlier than the current moment; performing fusion processing on the current aerial view and the historical aerial view according to the movement deviation data of the moving object between the current moment and the historical moment to obtain a fusion feature map; performing rendering processing on the fused feature map to obtain a rendered fused feature map; and predicting predicted point cloud data of the moving object at the next moment according to the rendered fusion feature map and the moving deviation data of the moving object between the current moment and the next moment, wherein the next moment is later than the current moment. Therefore, the adaptability to downstream tasks can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud data processing, and in particular to a point cloud data prediction method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the field of autonomous driving, point cloud data prediction is one of the most basic self-supervised tasks, which predicts future point clouds based on historical point clouds. It is of great significance in tasks such as target tracking, target prediction, and target detection.

[0003] Current point cloud prediction methods typically process the feature information of a target image at a specific moment to generate a predicted point cloud. This approach only processes single-frame images and fails to capture temporal information, making it less suitable for downstream tasks such as target tracking or prediction. Summary of the Invention

[0004] The main technical problem solved by this application is to provide a point cloud data prediction method, electronic device and computer-readable storage medium, which can improve adaptability.

[0005] To solve the above technical problems, this application provides a point cloud data prediction method.

[0006] In one embodiment, a point cloud data prediction method includes: obtaining a current bird's-eye view of a moving object at a current moment and a historical bird's-eye view at a historical moment, the historical moment being earlier than the current moment; fusing the current bird's-eye view and the historical bird's-eye view based on movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map; rendering the fused feature map to obtain a rendered fused feature map; predicting predicted point cloud data of the moving object at a next moment based on the rendered fused feature map and movement deviation data of the moving object between the current moment and a next moment, the next moment being later than the current moment.

[0007] In one embodiment, the step of fusing the current bird's-eye view and the historical bird's-eye view based on the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map includes: obtaining the offset between the historical bird's-eye view and the current bird's-eye view based on the movement deviation data; aligning the image features of the current bird's-eye view with the image features of the historical bird's-eye view based on the offset between the historical bird's-eye view and the current bird's-eye view to obtain the image features of the historical bird's-eye view aligned with the image features of the current bird's-eye view; and performing feature fusion processing on the image features of the aligned historical bird's-eye view and the image features of the current bird's-eye view to obtain the fused feature map.

[0008] In one embodiment, the movement deviation data includes relative displacement and relative movement angle, and the step of obtaining the offset between the historical bird's-eye view and the current bird's-eye view based on the movement deviation data includes: obtaining the translation angle of the moving object between the current moment and the historical moment based on the relative displacement; determining the offset angle based on the difference between the relative movement angle and the translation angle; and determining the offset between the historical bird's-eye view and the current bird's-eye view based on the relative displacement and the offset angle.

[0009] In one embodiment, the step of determining the offset between the historical bird's-eye view and the current bird's-eye view based on the relative displacement and the offset angle includes: obtaining the translation distance of the moving object between the current moment and the historical moment based on the relative displacement; obtaining the offset distance of the moving object between the current moment and the historical moment based on the translation distance and the offset angle; and determining the offset based on the size information of the current bird's-eye view and the offset distance.

[0010] In one embodiment, the relative displacement includes the relative displacement of the moving object in a first direction and the relative displacement in a second direction, the first direction and the second direction are perpendicular to each other, and the step of obtaining the translation distance of the moving object between the current moment and the historical moment based on the relative displacement includes: obtaining the sum of the squares of the relative displacement in the first direction and the relative displacement in the second direction; and determining the root square value of the sum of the squares as the translation distance between the current moment and the historical moment.

[0011] In one embodiment, the offset includes a first offset in a third image direction and a second offset in a fourth image direction, the third image direction and the fourth image direction are perpendicular to each other, and the step of aligning the image features of the current bird's-eye view and the image features of the historical bird's-eye view according to the offset between the historical bird's-eye view and the current bird's-eye view to obtain the image features of the historical bird's-eye view aligned with the image features of the current bird's-eye view includes: transforming the image features of the historical bird's-eye view according to the first offset to obtain alternative image features of the historical bird's-eye view; transforming the alternative image features of the historical bird's-eye view according to the second offset to obtain the image features of the aligned historical bird's-eye view.

[0012] In one embodiment, the step of rendering the fused feature map to obtain a rendered fused feature map includes: performing probability statistical processing on the foreground occlusion points of each grid in the fused feature map to obtain a foreground occlusion probability; performing feature averaging processing on the fused feature map to obtain a desired feature; and determining the product of the desired feature and the foreground occlusion probability as the rendered fused feature map.

[0013] In one embodiment, the step of predicting the predicted point cloud data of the moving object at the next moment based on the rendered fusion feature map and the movement deviation data of the moving object between the current moment and the next moment includes: encoding the movement deviation data between the current moment and the next moment to obtain high-dimensional feature information; inputting the high-dimensional feature information and the rendered fusion feature map into a feature decoder to obtain image features of the predicted bird's-eye view of the moving object at the next moment; and performing feature processing on the image features of the predicted bird's-eye view at the next moment to obtain the predicted point cloud data at the next moment.

[0014] In order to solve the above technical problems, the present application provides an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above point cloud data prediction method.

[0015] In order to solve the above technical problems, the present application provides a computer-readable storage medium, comprising: storing program data, which is used to implement the above-mentioned point cloud data prediction method when executed by a processor.

[0016] The above scheme fuses the current bird's-eye view and historical bird's-eye view images based on the movement deviation data between the current moment and the previous moment, generating a fused feature map. The fused feature map is then rendered to generate a rendered fused feature map. The predicted point cloud data for the moving object at the next moment is then predicted based on the rendered fused feature map and the movement deviation data between the current moment and the next moment. Thus, by fusing the current bird's-eye view and historical bird's-eye view images using the movement deviation data, the fused feature map incorporates time information. The predicted point cloud data for the next moment is then predicted based on the rendered fused feature map, thereby improving the prediction accuracy of the point cloud data and, in turn, the adaptability of the predicted point cloud data for downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0018] Figure 1 is a flow chart of an exemplary embodiment of a point cloud data prediction method shown in the present application;

[0019] Figure 2 is a schematic diagram of an exemplary embodiment of the translation distance and translation angle shown in the present application;

[0020] Figure 3 is a schematic diagram of an exemplary embodiment of a foreground point shown in the present application;

[0021] Figure 4 is a flowchart of an exemplary embodiment of a rendering process shown in this application;

[0022] Figure 5 is a schematic structural diagram of an exemplary embodiment of a feature decoder shown in this application;

[0023] Figure 6 is a flowchart of an exemplary embodiment of point cloud data prediction shown in this application;

[0024] Figure 7 is a block diagram of a point cloud data prediction device shown in an exemplary embodiment of the present application;

[0025] Figure 8 This is a structural diagram of an embodiment of an electronic device provided by the present application;

[0026] Figure 9 It is a structural diagram of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] First, it's important to note that in the field of autonomous driving, point cloud data prediction—predicting future point clouds from historical point clouds—is one of the most fundamental self-supervised tasks. It plays a crucial role in tasks such as object tracking, prediction, and detection. Current point cloud prediction methods typically process the feature information of a target image at a specific moment to produce a predicted point cloud. This approach only processes single-frame images and fails to capture temporal information, making it less suitable for downstream tasks such as object tracking or prediction.

[0029] Based on this, this application provides a point cloud data prediction method, electronic device and computer readable storage medium. Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of a point cloud data prediction method shown in the present application.

[0030] The execution entity of a point cloud data prediction method may be a terminal device, server, or other processing device, wherein the terminal device may be a computer, mobile device, terminal, computing device, vehicle-mounted device, etc. The execution entity of the point cloud data prediction method may also be a point cloud data prediction device. In some possible implementations, the point cloud data prediction method may be implemented by a processor calling computer-readable instructions stored in a memory. The execution entity of the point cloud data prediction method may also be a big data cluster. A big data cluster is a computer system architecture formed by multiple computers connected via a network. The big data cluster can be deployed on a private cloud built on K8S (Kubernetes, a container orchestration engine).

[0031] Specifically, a point cloud data prediction method of this embodiment includes the following steps:

[0032] Step S110 : Acquire a current bird's-eye view of the mobile object at the current moment and a historical bird's-eye view at a historical moment, where the historical moment is earlier than the current moment.

[0033] A mobile object is an object that can move, such as a vehicle.

[0034] The current bird's-eye view image refers to an image of a moving object observed from a bird's-eye view at the current moment.

[0035] A historical bird's-eye view image is an image of a moving object observed from a bird's-eye view at a historical moment, where the historical moment can be the moment before the current moment.

[0036] The point cloud data prediction device obtains a current bird's-eye view of the mobile object at the current moment and historical bird's-eye views of the mobile object at historical moments. Specifically, the point cloud data prediction device obtains at least one horizontal view of the mobile object at the current moment, performs view conversion processing on the image features of the at least one horizontal view, and obtains the current bird's-eye view of the mobile object at the current moment; the point cloud data prediction device obtains at least one horizontal view of the mobile object at historical moments, performs view conversion processing on the image features of the at least one horizontal view, and obtains the historical bird's-eye views of the mobile object at historical moments.

[0037] In one embodiment, a point cloud data prediction device acquires multi-view images captured by multiple cameras, inputs the multi-view images into a feature encoder, and obtains image features output by the feature encoder. For example, the image features may include [m, n, c, h, w], where m represents the number of image frames, n represents the number of cameras, c represents the number of feature channels, and h and w represent the height and width of the image. For example, the feature encoder may be a swin-transformer (sliding window-transformer) network.

[0038] In one embodiment, a point cloud data prediction device performs a Bird's Eye View (BEV) view conversion on the image features of multi-view images to obtain image features in a BEV space. Specifically, the point cloud data prediction device uses a method based on Lift, Splat, Shoot (LSS) to perform BEV view conversion on the image features of the multi-view images. LSS is an algorithmic framework used in autonomous driving and computer vision, primarily for converting multi-view images into a bird's-eye view representation. For example, the point cloud data prediction device downsamples the 2D planar images of each camera to obtain feature points of the 2D planar images; explicitly estimates the depth distribution of the feature points of the 2D planar images in 3D space; then, based on the depth distribution of the feature points in 3D space, maps the feature points into the camera's view cone, forming a view cone in 3D space containing the image features; then, based on the camera's intrinsic and extrinsic parameters, the view cone points are assigned to a BEV grid, and sum-pooling is performed on the multiple view cone points in each grid cell of the grid to form a BEV feature map. The BEV feature map includes the image features of the bird's-eye view.

[0039] Step S120 : fusing the current bird's-eye view image and the historical bird's-eye view image based on the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map.

[0040] Motion deviation data is the deviation between the current motion data of a mobile object and its motion data at a previous moment. The motion deviation data may include a first offset in the height direction and a second offset in the width direction. The motion deviation data may also include the relative displacement and relative motion angle of the mobile object between the current moment and the previous moment.

[0041] The fused feature map is a feature map that fuses the image features of the current bird's-eye view and the image features of the historical bird's-eye view.

[0042] The point cloud data prediction device fuses the current bird's-eye view with the historical bird's-eye view based on the motion deviation data of the moving object between the current moment and the historical moment, generating a fused feature map. Specifically, the motion deviation data includes a first offset in the height direction and a second offset in the width direction. The point cloud data prediction device performs an affine transformation on the image features of the historical bird's-eye view based on the motion deviation data to obtain image features of the historical bird's-eye view aligned with the image features of the current bird's-eye view. The device then fuses the aligned image features of the historical bird's-eye view with the image features of the current bird's-eye view using a convolutional layer to generate a fused feature map.

[0043] Step S130: Render the fused feature map to obtain a rendered fused feature map.

[0044] The rendered fused feature map refers to the image obtained after the fused feature map is rendered.

[0045] The point cloud data prediction device renders the fused feature map to obtain a rendered fused feature map. Specifically, the point cloud data prediction device uses a preset renderer to render the fused feature map to obtain a rendered fused feature map. The preset renderer may be a latent space renderer, etc.

[0046] Step S140: predicting the predicted point cloud data of the moving object at the next moment based on the rendered fusion feature map and the movement deviation data of the moving object between the current moment and the next moment, where the next moment is later than the current moment.

[0047] The next moment is the moment after the current moment. The next moment can be adjacent to the current moment and later than the current moment.

[0048] The point cloud data prediction device predicts the predicted point cloud data of the moving object at the next moment based on the rendered fused feature map and the movement deviation data of the moving object between the current moment and the next moment. Specifically, the point cloud data prediction device inputs the rendered fused feature map and the movement deviation data of the moving object between the current moment and the next moment into a feature decoder to obtain the predicted features of the moving object at the next moment; maps the predicted features through a mapping layer to obtain occupied voxels; and saves the coordinates of the center points of all occupied voxels as the predicted point cloud data for the next moment.

[0049] As can be seen, a point cloud data prediction method in this embodiment fuses the current bird's-eye view and historical bird's-eye view images based on the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map; renders the fused feature map to obtain a rendered fused feature map; and predicts the predicted point cloud data of the moving object at the next moment based on the rendered fused feature map and the movement deviation data of the moving object between the current moment and the next moment. Thus, by fusing the current bird's-eye view and historical bird's-eye view images using the movement deviation data, the fused feature map contains time information, and the predicted point cloud data for the next moment is predicted based on the rendered fused feature map, thereby improving the prediction accuracy of the point cloud data and, in turn, improving the adaptability of downstream tasks based on the predicted point cloud data.

[0050] The point cloud data prediction device fuses the current bird's-eye view and the historical bird's-eye view based on the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map, including the following steps: obtaining the offset between the historical bird's-eye view and the current bird's-eye view based on the movement deviation data; aligning the image features of the current bird's-eye view with the image features of the historical bird's-eye view based on the offset between the historical bird's-eye view and the current bird's-eye view to obtain the image features of the historical bird's-eye view aligned with the image features of the current bird's-eye view; and performing feature fusion processing on the aligned image features of the historical bird's-eye view and the image features of the current bird's-eye view to obtain a fused feature map. Thus, by aligning the image features of the current bird's-eye view and the historical bird's-eye view and then performing feature fusion, the fusion processing of the current bird's-eye view and the historical bird's-eye view is achieved.

[0051] The motion deviation data includes relative displacement and relative movement angle. The relative displacement can include the relative displacement of the moving object in a first direction and the relative displacement in a second direction. For example, the relative displacement can be expressed as (delta_x, delta_y), where delta_x represents the relative displacement in the first direction and delta_y represents the relative displacement in the second direction.

[0052] Before fusing the current bird's-eye view and historical bird's-eye view based on the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map, the point cloud data prediction device further includes: obtaining the movement deviation data of the moving object between the current moment and the historical moment. As an example, the point cloud data prediction device uses a positioning device to respectively collect the position of the moving object at the current moment and the position at the historical moment, and determines the difference between the current position and the position at the historical moment as the relative displacement; the point cloud data prediction device uses an angle measurement device to respectively collect the movement angle of the moving object at the current moment and the movement angle at the historical moment, and determines the difference between the current movement angle and the movement angle at the historical moment as the relative movement angle. As another example, the moving object is a vehicle, and the point cloud data prediction device obtains the relative displacement and relative movement angle of the moving object at the current moment relative to the historical moment from the vehicle's odometer information. The relative movement angle is the relative angle of the vehicle's front end at the current moment relative to the historical moment.

[0053] The point cloud data prediction device obtains the offset between the historical bird's-eye view and the current bird's-eye view based on the movement deviation data, including: obtaining the translation angle of the moving object between the current moment and the historical moment based on the relative displacement; determining the offset angle based on the difference between the relative movement angle and the translation angle; and determining the offset between the historical bird's-eye view and the current bird's-eye view based on the relative displacement and the offset angle.

[0054] The point cloud data prediction device obtains the translation angle of the moving object between the current moment and the historical moment based on the relative displacement. Specifically, the point cloud data prediction device obtains the ratio between the relative displacement of the moving object in the second direction and the relative displacement in the first direction, and applies an inverse tangent function to the ratio to obtain the translation angle.

[0055] In one embodiment, if Figure 2 As shown, the straight-line distance between the historical position A of the moving object and the current position B is the translation distance of the moving object at the current moment relative to the historical moment, the first direction is the horizontal direction, the second direction is the vertical direction, and the angle between the translation distance and the first direction is the translation angle.

[0056] The point cloud data prediction device determines the offset angle according to the difference between the relative movement angle and the translation angle. Specifically, the point cloud data prediction device determines the offset angle according to the difference between the relative movement angle and the translation angle.

[0057] For example, the offset angle satisfies the following formula:

[0058] bev_angle=delta_angle-trans_angle

[0059] In the above formula, bev_angle represents the offset angle, delta_angle represents the relative movement angle, and trans_angle represents the translation angle.

[0060] The point cloud data prediction device determines the offset between the historical bird's-eye view and the current bird's-eye view based on the relative displacement and the offset angle, including: obtaining the translation distance of the moving object between the current moment and the historical moment based on the relative displacement; obtaining the offset distance of the moving object between the current moment and the historical moment based on the translation distance and the offset angle; and determining the offset based on the size information of the current bird's-eye view and the offset distance.

[0061] The point cloud data prediction device obtains the translation distance of the moving object between the current moment and the historical moment based on the relative displacement, including: obtaining the sum of the squares of the relative displacement in the first direction and the relative displacement in the second direction; and determining the root square value of the sum of the squares as the translation distance between the current moment and the historical moment.

[0062] For example, the translation distance satisfies the following formula:

[0063]

[0064] In the above formula, L t Characterizes the translation distance.

[0065] The point cloud data prediction device obtains the offset distance of the moving object between the current moment and the historical moment based on the translation distance and the offset angle. Specifically, the offset distance includes the offset distance in a first direction and the offset distance in a second direction. The point cloud data prediction device determines the offset distance in the first direction by multiplying the translation distance by the cosine value of the offset angle, and determines the offset distance in the second direction by multiplying the translation distance by the sine value of the offset angle.

[0066] For example, the translation distance, offset angle, and offset distance in the first direction satisfy the following formula:

[0067] L x =L t ×cosbev_angle

[0068] In the above formula, L x Characterizes the offset distance in the first direction.

[0069] The translation distance, offset angle, and offset distance in the second direction satisfy the following formula:

[0070] L y =L t ×sinbev_angle

[0071] In the above formula, L yIndicates the offset distance in the second direction.

[0072] The point cloud data prediction device determines the offset based on the size information of the current bird's-eye view and the offset distance. In one embodiment, the size information of the current bird's-eye view includes the grid height and grid width of the bird's-eye view. For example, the grid size of the bird's-eye view may be 0.5 meters by 0.5 meters. The offset includes a first offset in the third image direction and a second offset in the fourth image direction. The third and fourth image directions are perpendicular to each other, with the third image direction being the height direction and the fourth image direction being the width direction.

[0073] Specifically, the point cloud data prediction device determines the ratio of the offset distance in the first direction to the grid height of the bird's-eye view as the first offset; and determines the ratio of the offset distance in the second direction to the grid width of the bird's-eye view as the second offset.

[0074] For example, the first offset is an offset in the height direction, and the first offset satisfies the following formula:

[0075] shift_h=L x ÷h

[0076] In the above formula, shift_h represents the first offset, and h represents the grid height of the bird's-eye view.

[0077] The second offset is the offset in the width direction, and the second offset satisfies the following formula:

[0078] shift_W=L y ÷w

[0079] In the above formula, shift_W represents the second offset, and w represents the grid width of the bird's-eye view.

[0080] The point cloud data prediction device aligns the image features of the current bird's-eye view with the image features of the historical bird's-eye view according to the offset between the historical bird's-eye view and the current bird's-eye view to obtain the image features of the historical bird's-eye view aligned with the image features of the current bird's-eye view, including: transforming the image features of the historical bird's-eye view according to a first offset to obtain alternative image features of the historical bird's-eye view; and transforming the alternative image features of the historical bird's-eye view according to a second offset to obtain the image features of the aligned historical bird's-eye view.

[0081] Exemplarily, the point cloud data prediction device translates the historical feature map corresponding to the historical bird's-eye view by a first offset in the third image direction, and translates the historical feature map corresponding to the historical bird's-eye view by a second offset in the fourth image direction to obtain an aligned historical feature map, and determines the image features of the aligned historical feature map as the image features of the aligned historical bird's-eye view.

[0082] As another example, the offset includes a deviation angle on a preset clockwise direction. The point cloud data prediction device rotates the historical feature map corresponding to the historical bird's-eye view map by the deviation angle on the preset clockwise direction to obtain an aligned historical feature map. The preset clockwise direction can be clockwise or counterclockwise.

[0083] The point cloud data prediction device performs feature fusion processing on the image features of the aligned historical bird's-eye view image and the image features of the current bird's-eye view image to obtain a fused feature map. Specifically, the point cloud data prediction device uses a convolution layer to perform convolution processing on the image features of the aligned historical bird's-eye view image and the image features of the current bird's-eye view image to obtain a fused feature map.

[0084] The point cloud data prediction device renders the fused feature map to obtain a rendered fused feature map, including the following steps: performing probabilistic statistical processing on the foreground occlusion points of each grid in the fused feature map to obtain a foreground occlusion probability; performing feature averaging on the fused feature map to obtain a desired feature; and multiplying the desired feature by the foreground occlusion probability to determine the rendered fused feature map. Thus, by introducing the foreground occlusion probability, the height distribution becomes sharper and clearer, effectively alleviating the problem of the radial shape of the BEV feature.

[0085] Foreground occlusion points refer to the grid points that are blocked between the origin and the target point in the image. Among them, the points between the origin and the target point are foreground points. For example, Figure 3 As shown in the left figure, the i-th grid in the fusion feature map is g (i) , on the ray from the origin o to the i-th grid, the foreground point is the ray from the origin o to the grid g (i) The grid points between . Figure 3 As shown in the right figure, the origin o and the grid g (i) The connected lines are uniformly sampled to obtain the foreground point x (i,0) 、x (i,1) 、x (i,2) and x (i,3) .

[0086] The point cloud data prediction device performs probabilistic statistical processing on the foreground occlusion points of each grid in the fused feature map to obtain the foreground occlusion probability. Specifically, the point cloud data prediction device inputs the features in the fused feature map into a linear layer, obtains a probability map through the mapping of the linear layer, and uses the conditional probability formula to perform probabilistic statistical processing on the foreground occlusion points of each grid in the probability map to obtain the foreground occlusion probability. Specifically, the foreground occlusion probability satisfies the following formula:

[0087]

[0088] In the above formula, Represents the foreground occlusion probability of the jth grid between the origin 0 and the i-th grid, p (i,j) Represents the probability of occupying the jth grid, p (i,k) Represents the probability of occupying the kth grid.

[0089] As can be seen, the foreground occlusion probability of each grid in this embodiment is the joint probability of the foreground point being free multiplied by the probability of the grid being occupied. In other words, when the current point is occluded, the probability of the grid being occupied is lowered, allowing the depth distribution to reflect the actual location of the point cloud.

[0090] In one embodiment, the rendering process is as follows Figure 4 As shown, the fusion feature map F obtained by the point cloud data prediction device bev For radial BEV features, in Step 1, the point cloud data prediction device maps the fused feature map through a linear layer to obtain a probability map p; Step 2, based on the occupancy probability of each grid in the probability map, the conditional probability function is used to calculate the foreground occlusion probability of each grid to obtain a conditional probability map p^; Step 3, the feature expectation function is used to obtain the expected features of the line formed from the origin to each grid point, and the features on the same line are averaged to obtain the feature averaged map Step-4: Multiply the expected feature by the foreground occlusion probability, and the resulting product is the rendered fusion feature map Thus, the BEV features with latent space information are obtained

[0091] The point cloud data prediction device predicts the predicted point cloud data of the moving object at the next moment based on the rendered fusion feature map and the movement deviation data of the moving object between the current moment and the next moment, including: encoding the movement deviation data between the current moment and the next moment to obtain high-dimensional feature information; inputting the high-dimensional feature information and the rendered fusion feature map into a feature decoder to obtain image features of the predicted bird's-eye view of the moving object at the next moment; and performing feature processing on the image features of the predicted bird's-eye view of the next moment to obtain the predicted point cloud data at the next moment.

[0092] The movement deviation data between the current moment and the next moment may include a relative displacement and a relative movement angle between the current moment and the next moment. Specifically, the point cloud data prediction device receives odometer information of the moving object at the next moment, and the odometer information includes the relative displacement and the relative movement angle between the current moment and the next moment.

[0093] The point cloud data prediction device encodes the motion deviation data between the current moment and the next moment to obtain high-dimensional feature information. Specifically, the point cloud data prediction device uses a linear or nonlinear transformation to map the motion deviation data between the current moment and the next moment into a high-dimensional space to obtain high-dimensional feature information. For example, the point cloud data prediction device uses a convolutional layer, an embedding layer, or a fully connected layer to process the motion deviation data between the current moment and the next moment to obtain high-dimensional feature information.

[0094] The feature decoder includes at least deformable self-attention, temporal cross attention and a feedforward neural network. Figure 5 The network structure of the feature decoder is combined with Figure 5 As shown, the point cloud data prediction device converts high-dimensional feature information e t+1 Added to FutureBEV Queries (Future BEV Queries) as the input of the feature decoder, the high-dimensional feature information e t+1 Input MLP (Multilayer Perceptron) to get the output of MLP; input the output of MLP into Deformable Self-Attention and the first Add&Norm (residual connection and normalization layer) respectively to get the output of Deformable Self-Attention; input the output of Deformable Self-Attention and the output of MLP into the first Add&Norm to get the result Q of the first Add&Norm output; the fused feature map after rendering Perform dimensionality change to obtain dimensions K and V, input the dimension-transformed rendered fusion feature map and the result Q into Temporal Cross-Attention to obtain the output; input the output of Temporal Cross-Attention and the result Q into the second Add&Norm to obtain the output of the second Add&Norm, input the output of the second Add&Norm into the FeedForward Network to obtain a decoding feature, repeat the above decoding steps 6 times to obtain the image features of the predicted bird's-eye view of the moving object at the next moment

[0095] The point cloud data prediction device performs feature processing on the image features of the predicted bird's-eye view at the next moment to obtain the predicted point cloud data at the next moment. Specifically, the point cloud data prediction device maps the image features of the predicted bird's-eye view at the next moment using a mapping layer to obtain occupied voxels. The coordinates of the center points of all occupied voxels are stored as the predicted point cloud data at the next moment. The mapping layer can be composed of an MLP.

[0096] After the point cloud data prediction device performs feature processing on the image features of the predicted bird's-eye view at the next moment to obtain the predicted point cloud data at the next moment, it also includes: obtaining the distance between each grid and the previous grid; determining the product of the step length corresponding to the distance between each grid and the previous grid and the conditional occlusion probability of the corresponding grid as the depth of each grid, obtaining the depth of all grids, and obtaining the depth distribution of the predicted point cloud data at the next moment based on the depth of all grids.

[0097] In one embodiment, the occupied voxel can be represented as P', P'∈R X*Y*Z , where X, Y, and Z represent the number of grids in the X, Y, and Z directions, respectively.

[0098] After performing feature processing on the image features of the predicted bird's-eye view at the next moment to obtain the predicted point cloud data at the next moment, the point cloud data prediction device further includes: obtaining the collected point cloud data at the next moment; obtaining a target loss value based on the collected point cloud data and the predicted point cloud data at the next moment, and training the initial point cloud data prediction network with the goal of minimizing the target loss value to obtain a target point cloud data prediction network. The target point cloud data prediction network can be a backbone network in a model such as a target detection model or a target tracking model.

[0099] In one embodiment, a point cloud data prediction device obtains a current bird's-eye view of a mobile object at a current moment, historical bird's-eye views at previous moments, and collected point cloud data at a next moment, where the previous moment is earlier than the current moment; inputs the current bird's-eye view of the mobile object at the current moment and the historical bird's-eye views at previous moments into an initial point cloud data prediction network; the initial point cloud data prediction network fuses the current bird's-eye view and the historical bird's-eye views based on movement deviation data of the mobile object between the current moment and the previous moment to obtain a fused feature map; renders the fused feature map to obtain a rendered fused feature map; and predicts predicted point cloud data of the mobile object at the next moment, where the next moment is later than the current moment, based on the rendered fused feature map and movement deviation data of the mobile object between the current moment and the next moment. The point cloud data prediction device obtains a target loss value based on the collected point cloud data and the predicted point cloud data at the next moment, and trains the initial point cloud data prediction network with the goal of minimizing the target loss value to obtain a target point cloud data prediction network. The point cloud data prediction method of this example is used to obtain point cloud data, which can achieve self-supervision without labeling the data during model training, thereby saving resources.

[0100] Collected point cloud data refers to point cloud data collected by sensors, and the collected point cloud data includes moving objects.

[0101] The point cloud data prediction device obtains a target loss value based on the collected point cloud data and the predicted point cloud data at the next moment. As an example, the point cloud data prediction device obtains a cross entropy loss and a chamfer distance loss based on the collected point cloud data and the predicted point cloud data at the next moment, respectively, and uses the sum of the cross entropy loss and the chamfer distance loss as the target loss value. As another example, the point cloud data prediction device obtains a cross entropy loss and a chamfer distance loss based on the collected point cloud data and the predicted point cloud data at the next moment, respectively, and uses at least one of the cross entropy loss and the chamfer distance loss as the target loss value.

[0102] In one embodiment, the cross entropy loss satisfies the following formula:

[0103]

[0104] In the above formula, represents the cross entropy loss, T represents the number of frames, n represents the number of point clouds collected in the tth frame, P t (.) represents the occupied voxel, g (i) Characterize the collected point cloud, x (i,j) Representing predicted point clouds.

[0105] In one embodiment, the chamfer distance loss satisfies the following formula:

[0106]

[0107] In the above formula, d CD (S1, S2) represents the chamfer distance loss, S1 represents the set of collected point clouds, x is a point in the collected point cloud set, S2 represents the set of predicted point clouds, and y is a point in the predicted point cloud set.

[0108] In one embodiment, combining Figure 6 As shown, multiple images (Images) are collected between time tm and time t, and the multiple images are input into the feature encoder to obtain image features of the historical bird's-eye view and image features of the current bird's-eye view; the image features of the historical bird's-eye view and the image features of the current bird's-eye view are time-series fused to obtain a fused feature map; the fused feature map is rendered to obtain a rendered fused feature map; the rendered fused feature map is input into the feature decoder to obtain image features of the predicted (Prediction) point cloud data; the collected point cloud data at time t+1 is collected, where t+1->t+m, and the image features (gt) of the collected point cloud data at time t+1 are obtained; the loss (Loss) between the image features of the collected point cloud data and the image features of the predicted point cloud data is calculated, so that the prediction accuracy of the predicted point cloud data can be judged according to the loss.

[0109] Figure 7FIG is a block diagram of a point cloud data prediction device shown in an exemplary embodiment of the present application. Figure 7 As shown, the exemplary point cloud data prediction device 700 includes: an acquisition module 710, a fusion module 720, a rendering module 730 and a prediction module 740. Specifically:

[0110] The acquisition module 710 is configured to acquire a current bird's-eye view of the mobile object at a current moment and a historical bird's-eye view of the mobile object at a historical moment, where the historical moment is earlier than the current moment.

[0111] The fusion module 720 is used to fuse the current bird's-eye view image and the historical bird's-eye view image according to the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map.

[0112] The rendering module 730 is used to render the fused feature map to obtain a rendered fused feature map.

[0113] The prediction module 740 is used to predict the predicted point cloud data of the moving object at the next moment based on the rendered fusion feature map and the movement deviation data of the moving object between the current moment and the next moment, where the next moment is later than the current moment.

[0114] In this exemplary point cloud data prediction device, the current bird's-eye view and historical bird's-eye view images are fused based on the object's motion deviation data between the current moment and previous moments to produce a fused feature map. The fused feature map is then rendered to produce a rendered fused feature map. The predicted point cloud data for the moving object at the next moment is predicted based on the rendered fused feature map and the object's motion deviation data between the current moment and the next moment. Thus, by fusing the current bird's-eye view and historical bird's-eye view images using the motion deviation data, the fused feature map incorporates time information. The predicted point cloud data for the next moment is then predicted based on the rendered fused feature map, thereby improving the accuracy of point cloud data prediction and, in turn, the adaptability of the predicted point cloud data for downstream tasks.

[0115] The functions of each module can be found in the embodiment of the point cloud data prediction method, which will not be repeated here.

[0116] In order to implement the point cloud data prediction method of the above embodiment, this application proposes another electronic device, please refer to Figure 8 , Figure 8 It is a structural diagram of an embodiment of an electronic device provided by this application.

[0117] The electronic device 800 includes a memory 801 and a processor 802 , wherein the memory 801 and the processor 802 are coupled.

[0118] The memory 801 is used to store program data, and the processor 802 is used to execute the program data to implement the point cloud data prediction method of the above embodiment.

[0119] In this embodiment, the processor 802 may also be referred to as a CPU (Central Processing Unit). The processor 802 may be an integrated circuit chip having signal processing capabilities. The processor 802 may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or the processor 802 may be any conventional processor.

[0120] This application also provides a computer-readable storage medium, such as Figure 9 As shown, the computer-readable storage medium 900 is used to store program data 901. When the program data 901 is executed by the processor, it is used to implement the point cloud data prediction method in the method embodiment of the present application.

[0121] The method involved in the embodiment of the point cloud data prediction method of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0122] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A point cloud data prediction method, characterized in that: The method comprises: Obtaining a current bird's-eye view of the mobile object at a current moment and a historical bird's-eye view at a historical moment, the historical moment being earlier than the current moment; fusing the current bird's-eye view image and the historical bird's-eye view image according to the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature image; Rendering the fused feature map to obtain a rendered fused feature map; Predicted point cloud data of the moving object at a next moment is predicted based on the rendered fusion feature map and movement deviation data of the moving object between the current moment and the next moment, where the next moment is later than the current moment.

2. The point cloud data prediction method according to claim 1, characterized in that: The step of fusing the current bird's-eye view and the historical bird's-eye view according to the movement deviation data of the moving object between the current moment and the historical moment to obtain a fused feature map comprises: Acquire an offset between the historical bird's-eye view and the current bird's-eye view according to the movement deviation data; performing alignment processing on the image features of the current bird's-eye view and the image features of the historical bird's-eye view according to the offset between the historical bird's-eye view and the current bird's-eye view, to obtain the image features of the historical bird's-eye view aligned with the image features of the current bird's-eye view; Feature fusion processing is performed on the image features of the aligned historical bird's-eye view and the image features of the current bird's-eye view to obtain the fused feature map.

3. The point cloud data prediction method according to claim 2, characterized in that: The movement deviation data includes a relative displacement and a relative movement angle. The step of obtaining an offset between the historical bird's-eye view and the current bird's-eye view according to the movement deviation data includes: Acquire a translation angle of the moving object between the current moment and the historical moment according to the relative displacement; determining an offset angle according to a difference between the relative movement angle and the translation angle; An offset between the historical bird's-eye view and the current bird's-eye view is determined according to the relative displacement and the offset angle.

4. The point cloud data prediction method according to claim 3, characterized in that: The step of determining the offset between the historical bird's-eye view and the current bird's-eye view according to the relative displacement and the offset angle comprises: Acquire a translation distance of the moving object between the current moment and the historical moment according to the relative displacement; Acquire an offset distance of the moving object between the current moment and the historical moment according to the translation distance and the offset angle; The offset is determined according to the size information of the current bird's-eye view and the offset distance.

5. The point cloud data prediction method according to claim 4, characterized in that: The relative displacement includes a relative displacement of the moving object in a first direction and a relative displacement in a second direction, the first direction and the second direction being perpendicular to each other, and the step of obtaining a translation distance of the moving object between the current moment and the historical moment based on the relative displacement includes: Obtaining a sum of squares of the relative displacement in the first direction and the relative displacement in the second direction; The root square value of the square sum is determined as the translation distance between the current moment and the historical moment.

6. The point cloud data prediction method according to claim 2, characterized in that: The offset includes a first offset in a third image direction and a second offset in a fourth image direction, the third image direction and the fourth image direction are perpendicular to each other, and the step of aligning the image features of the current bird's-eye view with the image features of the historical bird's-eye view based on the offset between the historical bird's-eye view and the current bird's-eye view to obtain the image features of the historical bird's-eye view aligned with the image features of the current bird's-eye view includes: transforming the image features of the historical bird's-eye view according to the first offset to obtain candidate image features of the historical bird's-eye view; The candidate image features of the historical bird's-eye view are transformed according to the second offset to obtain the aligned image features of the historical bird's-eye view.

7. The point cloud data prediction method according to claim 1, characterized in that: The step of rendering the fused feature map to obtain a rendered fused feature map includes: Performing probability statistics processing on the foreground occlusion points of each grid in the fused feature map to obtain a foreground occlusion probability; Performing feature averaging processing on the fused feature map to obtain desired features; The product of the expected feature and the foreground occlusion probability is determined as the rendered fusion feature map.

8. The point cloud data prediction method according to claim 1, characterized in that: The step of predicting predicted point cloud data of the moving object at the next moment based on the rendered fusion feature map and movement deviation data of the moving object between the current moment and the next moment includes: Encoding the movement deviation data between the current moment and the next moment to obtain high-dimensional feature information; Inputting the high-dimensional feature information and the rendered fused feature map into a feature decoder to obtain image features of a predicted bird's-eye view of the moving object at the next moment; Feature processing is performed on the image features of the predicted bird's-eye view at the next moment to obtain predicted point cloud data at the next moment.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor calls the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images