Point cloud fusion method and device, storage medium, terminal and computer program product

Abstract

The invention discloses a point cloud fusion method and device, a storage medium, a terminal and a computer program product, and the method comprises the steps: obtaining a plurality of frames of original point clouds with time sequence, and converting the plurality of frames of original point clouds to the same target coordinate system, thereby obtaining a plurality of frames of converted point clouds; the multiple frames of transformation point clouds are respectively projected to a target view from a target coordinate system, corresponding multiple frames of first projection features are obtained, and the dimension of the target coordinate system is larger than that of the target view; performing weight prediction on each frame of first projection feature to obtain a weight corresponding to the frame of first projection feature, and then performing weighted fusion on multiple frames of first projection features by adopting each weight to obtain a fused projection feature; projecting the fused projection feature from the target view to the target coordinate system to obtain a second projection feature; and determining a point cloud fusion result based on the second projection feature. According to the scheme, a point cloud fusion result with more comprehensive scene information and richer details can be obtained, and the accuracy of subsequent semantic segmentation is further improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a point cloud fusion method and device, a storage medium, a terminal, and a computer program product. Background Art

[0002] Point cloud semantic segmentation technology plays an extremely important role in the field of autonomous driving. Semantic segmentation technology processes point cloud data and classifies each point into different semantic categories, such as roads, vehicles, pedestrians, buildings, traffic signs, etc. This fine classification helps the autonomous driving system understand the environment more accurately and helps the vehicle make correct decisions, such as slowing down, avoiding or continuing to drive, thereby achieving safer navigation and obstacle avoidance. For example, in a complex traffic environment, autonomous driving vehicles need to identify and distinguish different types of objects, such as distinguishing whether there are pedestrians or stationary obstacles in front. Point cloud semantic segmentation technology can provide this level of detail to help vehicles make correct decisions, such as slowing down, avoiding or continuing to drive.

[0003] In the existing technology, semantic segmentation is usually performed based on single-frame point cloud data. This method has the following main disadvantages:

[0004] (1) Insufficient information: A single frame of point cloud data only contains scene information at a certain moment and lacks continuity in the time dimension. (2) Sparsity problem: Since the point cloud data obtained by sensors such as lidar are usually sparse, a single frame of data may not cover all the details of the scene, especially at long distances or in complex environments, which further limits the accuracy of segmentation. (3) Lack of contextual information: A single frame of point cloud data cannot provide sufficient contextual information, such as the relationship between objects, historical changes in the scene, etc., which are crucial for accurately understanding the environment.

[0005] Therefore, although single-frame point cloud semantic segmentation technology performs well in static scenes, its limitations become particularly obvious in complex environments that need to process dynamic and multi-scan data. Summary of the invention

[0006] One of the purposes of the embodiments of the present invention is to provide a point cloud cross-frame fusion solution, which can effectively integrate point cloud data from different time frames to obtain a point cloud fusion result with more comprehensive scene information and richer details, thereby improving the accuracy of subsequent semantic segmentation.

[0007] To achieve the above-mentioned purpose, an embodiment of the present invention provides a point cloud fusion method, comprising the following steps: obtaining a multi-frame original point cloud with time sequence, and transforming the multi-frame original point cloud to the same target coordinate system respectively to obtain a multi-frame transformed point cloud; projecting the multi-frame transformed point cloud from the target coordinate system to the target view respectively to obtain the corresponding multi-frame first projection features, wherein the dimension of the target coordinate system is greater than the dimension of the target view; performing weight prediction on the first projection feature of each frame to obtain the weight corresponding to the first projection feature of the frame, and then using each weight to perform weighted fusion on the multi-frame first projection features to obtain a fused projection feature; projecting the fused projection feature from the target view to the target coordinate system to obtain a second projection feature; and determining the point cloud fusion result based on the second projection feature.

[0008] Optionally, the multiple frames of transformed point clouds are projected from the target coordinate system to the target view respectively to obtain corresponding multiple frames of first projection features, including: for each frame of transformed point cloud, performing first feature extraction on the frame transformed point cloud to obtain first feature data; projecting the first feature data from the target coordinate system to the target view to obtain corresponding first preliminary projection features; and aggregating the first preliminary projection features in the target view to obtain the first projection features corresponding to the frame transformed point cloud.

[0009] Optionally, the first preliminary projection feature includes multiple projection points, each projection point has feature values ​​of multiple dimensions; the first preliminary projection feature is aggregated in the target view to obtain the first projection feature corresponding to the frame transformation point cloud, including: gridding the target view to obtain multiple grids; for each grid containing a projection point, in each dimension of the projection point, the largest feature value is selected from the feature values ​​of each projection point in the grid in the dimension as the feature value of the grid in the dimension; the feature values ​​of each grid of the target view in the multiple dimensions are used as the first projection feature corresponding to the frame transformation point cloud.

[0010] Optionally, the target view includes a range view, the target coordinate system is a three-dimensional coordinate system, the range view is a two-dimensional view, and the first feature data includes a plurality of first feature points; the first feature data is projected from the target coordinate system to the target view using the following formula:

[0011] theta = z;

[0012] phi = arctan2(y,x);

[0013] Among them, x represents the x-axis coordinate value of the first feature point in the first feature data in the target coordinate system, y represents the y-axis coordinate value of the first feature point in the target coordinate system, z represents the z-axis coordinate value of the first feature point in the target coordinate system, arctan2() represents the inverse tangent function, theta represents the first coordinate value of the projection point corresponding to the first feature point in the range view, and phi represents the second coordinate value of the projection point corresponding to the first feature point in the range view.

[0014] Optionally, the fused projection feature is projected from the target view to the target coordinate system to obtain a second projection feature, including: performing a second feature extraction on the fused projection feature to obtain second feature data; projecting the second feature data from the target view to the target coordinate system to obtain a second preliminary projection feature; performing bilinear interpolation processing on the second preliminary projection feature in the target coordinate system to obtain the second projection feature.

[0015] Optionally, weight prediction for the first projection feature of each frame is achieved by using a pre-trained weight prediction network corresponding to the first projection feature of the frame; multiple pre-trained weight prediction networks corresponding to the first projection features of the multiple frames are determined in the following manner: an initialization model is determined, and a training data set is constructed using a multi-frame sample point cloud with time sequence and semantic segmentation annotation data of one frame of the sample point cloud, wherein the initialization model at least includes a first semantic segmentation network and multiple weight prediction networks to be trained, and the multiple weight prediction networks to be trained correspond to the multi-frame sample point clouds one-to-one, and are respectively used to perform weight prediction on the first sample projection feature determined based on the corresponding sample point cloud, and the first semantic segmentation network is used to perform semantic segmentation on the sample point cloud fusion result determined based on the multi-frame sample point cloud, and output semantic segmentation prediction data; the initialization model is iteratively trained using a preset loss function and the training data set to iteratively optimize the parameters of the multiple weight prediction networks to be trained, thereby obtaining multiple pre-trained weight prediction networks; wherein, in each round of iterative training, the function value of the loss function is determined based at least on the semantic segmentation annotation data and the semantic segmentation prediction data output by the first semantic segmentation network.

[0016] Optionally, in each round of iterative training, the function value of the loss function is determined based at least on the semantic segmentation annotation data and the semantic segmentation prediction data output by the first semantic segmentation network, including: calculating a first loss value based on the difference between the semantic segmentation annotation data and the semantic segmentation prediction data; using a second semantic segmentation network to perform semantic segmentation on the sample point cloud to which the semantic segmentation annotation data belongs to obtain semantic segmentation reference data; calculating a second loss value based on the difference between the semantic segmentation prediction data and the semantic segmentation reference data; and performing a weighted operation on the first loss value and the second loss value to obtain the function value of the loss function.

[0017] Optionally, the target view includes a bird's-eye view and a range view, and a second projection feature obtained by projecting the bird's-eye view to the target coordinate system is recorded as a second bird's-eye view feature, and a second projection feature obtained by projecting the range view to the target coordinate system is recorded as a second range view feature; based on the second projection feature, a point cloud fusion result is determined, including: performing feature fusion on the second bird's-eye view feature and the second range view feature to obtain the point cloud fusion result.

[0018] Optionally, each weight is used to perform weighted fusion on the first projection features of the multiple frames to obtain a fused projection feature, including: respectively determining the feature data belonging to the static target and the dynamic target in the first projection features of the multiple frames, recorded as the first feature data and the second feature data; using the each weight to perform weighted operation on the first feature data in the first projection features of the multiple frames to obtain a static target fused projection feature; selecting a frame of first projection features from the multiple frames of first projection features, and using the second feature data in the selected first projection feature as the dynamic target fused projection feature; merging the static target fused projection feature and the dynamic target fused projection feature to obtain the fused projection feature.

[0019] Optionally, after determining the point cloud fusion result, the method further includes: performing semantic segmentation on the point cloud fusion result.

[0020] An embodiment of the present invention also provides a point cloud fusion device, including: a point cloud data acquisition module, used to acquire multiple frames of original point clouds with time sequence, and transform the multiple frames of original point clouds to the same target coordinate system respectively, to obtain multiple frames of transformed point clouds; a first projection module, used to project the multiple frames of transformed point clouds from the target coordinate system to the target view respectively, to obtain corresponding multiple frames of first projection features, wherein the dimension of the target coordinate system is greater than the dimension of the target view; a weighted fusion module, used to perform weight prediction on the first projection features of each frame, to obtain the weight corresponding to the first projection features of the frame, and then use each weight to perform weighted fusion on the multiple frames of first projection features, to obtain a fused projection feature; a second projection module, used to project the fused projection feature from the target view to the target coordinate system, to obtain a second projection feature; a point cloud fusion result determination module, used to determine the point cloud fusion result based on the second projection feature.

[0021] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned point cloud fusion method are executed.

[0022] An embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned point cloud fusion method when running the computer program.

[0023] An embodiment of the present invention further provides a computer program product, including a computer program, which executes the steps of the above-mentioned point cloud fusion method when executed by a processor.

[0024] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0025] In an embodiment of the present invention, by using a multi-frame original point cloud with time sequence as the basic data for point cloud fusion, effectively integrating point cloud data from different time frames, a point cloud fusion result with more comprehensive scene information and richer details than a single-frame point cloud can be obtained. Furthermore, based on the point cloud fusion result, it is helpful to better understand the dynamic changes of the scene in the subsequent semantic segmentation process, improve the accuracy and reliability of the semantic segmentation results, and improve the autonomous driving system's perception of the environment. Furthermore, compared with the existing conventional scheme that directly performs data fusion and feature extraction on point cloud data in a higher-dimensional target coordinate system (for example, in a three-dimensional lidar coordinate system), this implementation scheme projects the point cloud data to a relatively lower-dimensional target view. On the one hand, the lower-dimensional target view is easier to implement structured data processing than the high-dimensional space, and more information such as neighbor information, spatial information, etc. can be extracted during the feature extraction process; on the other hand, the lower-dimensional space also helps to improve the efficiency of data processing and deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of a point cloud fusion method in an embodiment of the present invention;

[0027] Figure 2 yes Figure 1 A flowchart of a specific implementation of step S12;

[0028] Figure 3 yes Figure 1 A flowchart of a specific implementation of step S13;

[0029] Figure 4 is a flowchart of a specific implementation method of determining a pre-trained weight prediction network in an embodiment of the present invention;

[0030] Figure 5 It is a structural schematic diagram of a point cloud fusion device in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0032] Reference Figure 1 , Figure 1 1 is a flow chart of a point cloud fusion method in an embodiment of the present invention. The point cloud fusion method can be applied to a terminal device with a point cloud data processing function, such as but not limited to a computer, a vehicle-mounted terminal, a mobile phone, a tablet computer, a server, a cloud platform, etc.

[0033] The point cloud fusion method may include steps S11 to S15:

[0034] Step S11: acquiring multiple frames of original point clouds with time sequence, and transforming the multiple frames of original point clouds to the same target coordinate system respectively, to obtain multiple frames of transformed point clouds;

[0035] Step S12: projecting the multi-frame transformed point clouds from the target coordinate system to the target view respectively to obtain corresponding multi-frame first projection features, wherein the dimension of the target coordinate system is greater than the dimension of the target view;

[0036] Step S13: performing weight prediction on the first projection feature of each frame to obtain the weight corresponding to the first projection feature of the frame, and then using each weight to perform weighted fusion on the first projection features of multiple frames to obtain a fused projection feature;

[0037] Step S14: projecting the fused projection feature from the target view to the target coordinate system to obtain a second projection feature;

[0038] Step S15: Determine the point cloud fusion result based on the second projection feature.

[0039] In the specific implementation of step S11, the time-sequential multi-frame original point cloud may be point cloud data collected by a laser radar installed on the autonomous driving vehicle during the driving process of the autonomous driving vehicle. Each frame of the original point cloud has its own collection time and a laser radar coordinate system corresponding to the position of the laser radar at the collection time. It can be understood that during the driving process of the vehicle, the position of the laser radar is relatively fixed to the position of the vehicle and changes with the position of the vehicle. Accordingly, the laser radar coordinate system is related to the position of the laser radar. In other words, the laser radar coordinate system to which each frame of the original point cloud belongs corresponds to the position of the laser radar at the collection time of the frame of the original point cloud.

[0040] Transforming the multiple frames of original point clouds to the same target coordinate system may specifically refer to: transforming the multiple frames of original point clouds from their respective laser radar coordinate systems to the target coordinate system. Specifically, the target coordinate system may be the laser radar coordinate system to which one of the multiple frames of original point clouds belongs, for example, the laser radar coordinate system to which the last frame in the multiple frames of original point clouds belongs; or, the target coordinate system may also be other appropriate preset coordinate systems.

[0041] In some embodiments, the laser radar coordinate system to which each frame of the original point cloud belongs can be determined in the following manner: the coordinate origin is the center of mass or geometric center of the laser radar or the laser beam emission center, the x-axis points horizontally to the front of the laser beam emission surface, the y-axis points horizontally to the left side of the laser radar, and the z-axis points vertically upward.

[0042] In some embodiments, the dimension of the target coordinate system may be three-dimensional, and the dimension of the target view may be two-dimensional.

[0043] Reference Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation of step S12 in the embodiment; the step S12 may specifically include steps S121 to S123.

[0044] In step S121, for each frame of transformed point cloud, a first feature extraction is performed on the frame of transformed point cloud to obtain first feature data.

[0045] Each frame of the transformed point cloud may contain a number of 3D points, as well as the 3D coordinate information, reflection intensity information, etc. of each 3D point. Specifically, each frame of the transformed point cloud may be represented by an n×m matrix, where n represents the number of 3D points contained in the transformed point cloud, and m represents the dimension of each 3D point, that is, each 3D point has m-dimensional eigenvalues. For example, when m=4, the eigenvalues ​​of each 3D point in 4 dimensions are: the coordinate value of the 3D point in the target coordinate system, and the reflection intensity information of the 3D point.

[0046] In a specific implementation, the first feature extraction can be implemented using a first feature extraction network (e.g., a multilayer perceptron (MLP)). The first feature data obtained after the first feature extraction can be specifically represented by a matrix of h×w×c, where h×w represents the number of first feature points in the first feature data, and c represents the dimension of each first feature point, that is, each first feature point has c dimensional feature values. For example, c can be selected from 16, 32, 64, etc.

[0047] In step S122, the first feature data is projected from the target coordinate system to the target view to obtain a corresponding first preliminary projection feature.

[0048] Furthermore, the target view may include a range view, the target coordinate system is a three-dimensional coordinate system, the range view is a two-dimensional view, and the first feature data includes a plurality of first feature points; the first feature data is projected from the target coordinate system to the target view using the following formula:

[0049] theta = z;

[0050] phi = arctan2(y,x);

[0051] Among them, x represents the x-axis coordinate value of the first feature point in the first feature data in the target coordinate system, y represents the y-axis coordinate value of the first feature point in the target coordinate system, z represents the z-axis coordinate value of the first feature point in the target coordinate system, arctan2() represents the inverse tangent function, theta represents the first coordinate value of the projection point corresponding to the first feature point in the range view, and phi represents the second coordinate value of the projection point corresponding to the first feature point in the range view.

[0052] The range view is a two-dimensional distance image, each column in the distance image represents a distance value measured by a group of laser rangefinders at a certain time point, and each row represents a distance value measured by each laser rangefinder at a different steering position. Taking the target coordinate system as a three-dimensional coordinate system as an example, projecting the first feature data from the target coordinate system to the range view can be specifically understood as converting a point in a three-dimensional space to a two-dimensional space represented by a distance image through spherical coordinates.

[0053] In step S123, the first preliminary projection features are aggregated in the target view to obtain the first projection features corresponding to the frame transformation point cloud.

[0054] Compared with the existing conventional schemes that directly perform data fusion and feature extraction on each frame of transformed point cloud in a higher-dimensional target coordinate system (for example, in a three-dimensional lidar coordinate system), this implementation scheme performs data fusion and feature extraction by projecting the transformed point cloud into a relatively lower-dimensional target view. On the one hand, compared with high-dimensional coordinate systems, lower-dimensional target views are easier to implement structured data processing, and more information such as neighbor information, spatial information, etc. can be extracted during the feature extraction process; on the other hand, lower-dimensional space also helps to increase the speed of data processing and deployment.

[0055] Furthermore, the first preliminary projection feature includes multiple projection points, each projection point has feature values ​​of multiple dimensions; the step S123 may specifically include: gridding the target view to obtain multiple grids; for each grid including the projection point, in each dimension of the projection point, selecting the largest feature value from the feature values ​​of each projection point in the grid in the dimension as the feature value of the grid in the dimension; using the feature values ​​of each grid of the target view in the multiple dimensions as the first projection feature corresponding to the frame transformation point cloud.

[0056] In a specific implementation, the target view may be gridded using a preset grid size. For example, for a square grid, the grid size may refer to the side length of the square. The grid size may be appropriately set according to actual needs.

[0057] In a specific implementation, each projection point in the first preliminary projection feature has characteristic values ​​of c dimensions, and correspondingly, each feature point in the first projection feature obtained by aggregation processing also has characteristic values ​​of c dimensions.

[0058] As a non-limiting embodiment, the first preliminary projection feature includes M projection points, each projection point has eigenvalues ​​in 32 dimensions, that is, in each dimension of the projection point, the M projection points have corresponding M eigenvalues ​​in the dimension. For each grid containing projection points, in each dimension of the projection point, the largest eigenvalue is selected from the M eigenvalues ​​of each projection point in the grid in the dimension as the eigenvalue of the grid in the dimension. For a grid that does not contain a projection point, a preset value (e.g., 0) can be used as the eigenvalue of the grid in each dimension.

[0059] It should be noted that, in a specific implementation, in addition to using the above-mentioned gridding scheme to aggregate the first preliminary projection feature, other appropriate aggregation schemes may also be used. For example, all the projection points in the first preliminary projection feature are grouped into m projection points as a group (for example, randomly grouped) to obtain multiple groups of projection points; for each group of projection points, in each dimension of the projection point, the largest eigenvalue is selected from the eigenvalues ​​of each projection point in the group in the dimension (or the average of the eigenvalues ​​of each projection point in the group in the dimension is used) as the eigenvalue of the feature point corresponding to the group of projection points in the dimension; wherein the multiple groups of projection points obtained by grouping correspond one-to-one to the multiple feature points in the first projection feature obtained by aggregation.

[0060] Continue to refer to Figure 1 In the specific implementation of step S13, a pre-trained weight prediction network can be used to predict the weight of the first projection feature of each frame, or a weight can be assigned to the first projection feature of each frame in combination with appropriate information of the first projection feature of each frame (for example, the number of projection points included). For example, the more projection points are included, the greater the weight corresponding to the first projection feature.

[0061] Reference Figure 3 , Figure 3 yes Figure 1 A flowchart of a specific implementation of step S13 in the step S13; in the step S13, the weighted fusion of the first projection features of the multiple frames is performed using various weights to obtain the fused projection features, which may include steps S131 to S134.

[0062] In step S131, feature data belonging to a static target and a dynamic target in the plurality of frames of first projection features are determined respectively, and recorded as first feature data and second feature data.

[0063] In step S132, the weights are used to perform a weighted operation on the first feature data in the multiple frames of first projection features to obtain a static target fused projection feature.

[0064] In a specific implementation, the weighted operation may be a weighted sum, a weighted product, or the like.

[0065] As mentioned above, each first feature data corresponds to a feature point in the first projection feature, and each feature point has feature values ​​of multiple dimensions. In other words, each first feature data has feature values ​​of multiple dimensions. Taking weighted summation as an example, the weights are used to perform weighted summation on the first feature data in the first projection features of the multiple frames. Specifically, it may include: on each dimension of the first feature data, the weights are used to perform weighted summation on the feature values ​​of each first feature data in the first projection features of the multiple frames in the dimension to obtain the weighted feature value of the dimension.

[0066] In step S133, a frame of first projection features is selected from the multiple frames of first projection features, and the second feature data in the selected first projection features is used as the dynamic target fusion projection feature.

[0067] The selected frame of first projection features may be the frame of first projection features that is the latest in time sequence among the multiple frames of first projection features.

[0068] In step S134, the static target fusion projection feature and the dynamic target fusion projection feature are merged to obtain the fusion projection feature.

[0069] In the embodiment of the present invention, during the cross-frame point cloud fusion process, only the feature data belonging to the static target is weighted and fused. Since the dynamic target is in motion, directly using the feature information of multiple frames of point clouds for weighted fusion may cause ghosting or target blurring. Therefore, for the dynamic target, the feature information of one frame of point cloud is retained. In this way, a more information-rich and enhanced point cloud fusion result can be obtained, and the ghosting problem in the area where the dynamic target is located can be effectively avoided.

[0070] Continue to refer to Figure 1 In the specific implementation of step S14, the fused projection feature is projected from the target view to the target coordinate system to obtain a second projection feature.

[0071] Furthermore, the step S14 may specifically include: performing a second feature extraction on the fused projection feature to obtain second feature data; projecting the second feature data from the target view to the target coordinate system to obtain a second preliminary projection feature; performing bilinear interpolation processing on the second preliminary projection feature in the target coordinate system to obtain the second projection feature.

[0072] In a specific implementation, the second feature extraction can be implemented using an existing semantic feature extraction network (also referred to as a second feature extraction network, for example, a two-dimensional fully convolutional network (2D Fully Convolutional Networks, 2D FCN).

[0073] In the embodiment of the present invention, for step S12 and step S14, since the transformed point cloud is in the target coordinate system and the fused projection feature obtained by weighted fusion is in the target view, the spatial dimensions of the two are different, and the corresponding feature extraction method and projection data processing method are also different. Therefore, by adopting a more targeted and accurate feature extraction and data processing method, it is helpful to obtain a better point cloud fusion result.

[0074] In the specific implementation of step S15, the point cloud fusion result is determined based on the second projection feature.

[0075] In a specific implementation, there is only one target view (for example, only a bird's-eye view or only a range view), and the method of determining the point cloud fusion result in step S15 is: directly using the second projection feature as the point cloud fusion result.

[0076] In another specific implementation, the target view includes a bird's-eye view (Bird's-Eye-View, BEV) and a range view, and the second projection feature obtained by projecting the bird's-eye view to the target coordinate system is recorded as a second bird's-eye view feature, and the second projection feature obtained by projecting the range view to the target coordinate system is recorded as a second range view feature; the step S15 may include: performing feature fusion on the second bird's-eye view feature and the second range view feature to obtain the point cloud fusion result.

[0077] The bird's-eye view is a two-dimensional view of an object or scene from a high place, which is used to describe a scene viewed from a high place, just like a bird looking down at the ground from the air. This perspective can provide a comprehensive view of a wide area, allowing people to clearly see the details and layout on the ground. Taking the target coordinate system as a three-dimensional coordinate system as an example, projecting the first feature data from the target coordinate system to the bird's-eye view can be specifically understood as converting a point in three-dimensional space into a view viewed from above.

[0078] The bird's-eye view provides a global perspective and a unified coordinate framework without occlusion or scaling issues, which helps identify occluded or intersecting vehicles; and compared to point cloud data in three-dimensional space, the data projected into the BEV view is more structured, faster to process, easier to deploy, and efficient in data fusion.

[0079] Among them, the method of feature fusion of the second bird's-eye view feature and the second range view feature can be to fuse them by adopting feature data weighted operation, feature concatenation, two-layer or multi-layer perceptron (MultilayerPerceptron, MLP) and the like. The embodiment of the present invention does not specifically limit the method of feature fusion.

[0080] It should be pointed out that in addition to the above two views, other appropriate view forms may be used in combination with scene requirements, such as cylindrical view, point view, etc.

[0081] In an embodiment of the present invention, different types of views are used for point cloud projection and feature data fusion. Since different types of views can provide information from different perspectives and have their own characteristics in terms of detail information retention and data processing efficiency, it is helpful to obtain more comprehensive and rich point cloud fusion results, and further improve the accuracy and reliability of subsequent semantic segmentation.

[0082] Further, in Figure 1 In the illustrated embodiment, after executing step S15 to determine the point cloud fusion result, the method further includes: performing semantic segmentation on the point cloud fusion result.

[0083] In an embodiment of the present invention, through the point cloud cross-frame fusion technology, a point cloud fusion result containing more complete, rich and enhanced detail information is obtained. Based on the point cloud fusion result, the limitations of single-frame point cloud segmentation technology (especially processing dynamic information with temporal sequence) can be effectively overcome, which helps to enhance the ability of semantic segmentation algorithm to utilize time information in multi-scan point cloud data, so that objects can be more accurately identified and classified, and environmental perception and decision-making capabilities in applications such as autonomous driving systems can be improved.

[0084] Reference Figure 4 , Figure 4 It is a flowchart of a specific implementation method of determining a pre-trained weight prediction network in an embodiment of the present invention; specifically, weight prediction of the first projection feature of each frame is achieved by using the pre-trained weight prediction network corresponding to the first projection feature of the frame; the method of determining multiple pre-trained weight prediction networks corresponding to the first projection features of multiple frames may include the following steps S41 to S42.

[0085] In step S41, an initialization model is determined, and a training data set is constructed using a multi-frame sample point cloud with time sequence and semantic segmentation annotation data of one frame of the sample point cloud, wherein the initialization model includes at least a first semantic segmentation network and a plurality of weight prediction networks to be trained, and the plurality of weight prediction networks to be trained correspond one-to-one to the multi-frame sample point cloud.

[0086] Among them, the multiple weight prediction networks to be trained are respectively used to perform weight prediction on the first sample projection features determined based on the corresponding sample point cloud, and the first semantic segmentation network is used to perform semantic segmentation on the sample point cloud fusion results determined based on the multiple frames of sample point clouds, and output semantic segmentation prediction data.

[0087] In step S42, the initialization model is iteratively trained using a preset loss function and the training data set to iteratively optimize the parameters of the multiple weight prediction networks to be trained, thereby obtaining multiple pre-trained weight prediction networks.

[0088] In each round of iterative training, the function value of the loss function is determined based at least on the semantic segmentation annotation data and the semantic segmentation prediction data output by the first semantic segmentation network.

[0089] Furthermore, in each round of iterative training, the method for determining the function value of the loss function may include: calculating a first loss value based on the difference between the semantic segmentation annotation data and the semantic segmentation prediction data; using a second semantic segmentation network to perform semantic segmentation on the sample point cloud to which the semantic segmentation annotation data belongs to obtain semantic segmentation reference data; calculating a second loss value based on the difference between the semantic segmentation prediction data and the semantic segmentation reference data; and performing a weighted operation on the first loss value and the second loss value to obtain the function value of the loss function.

[0090] Among them, calculating the first loss value based on the difference between the semantic segmentation annotation data and the semantic segmentation prediction data, and calculating the second loss value based on the difference between the semantic segmentation prediction data and the semantic segmentation reference data can both be implemented using existing loss function formulas, such as the L1 loss function formula, the L2 loss function formula, the mean square error loss function formula, etc.

[0091] In a specific implementation, each round of training for the initialized model may specifically include: transforming the multiple frames of sample point clouds to the same sample target coordinate system to obtain multiple frames of transformed sample point clouds; projecting the multiple frames of transformed sample point clouds from the sample target coordinate system to the target view to obtain corresponding multiple frames of first sample projection features, wherein the dimensions of the sample target coordinate system and the target view are different; inputting the first sample projection features of each frame into the corresponding weight prediction network to obtain the sample prediction weight corresponding to the first sample projection features of the frame, and then using each sample prediction weight to perform weighted fusion on the first sample projection features of the multiple frames to obtain fused sample projection features; projecting the fused sample projection features from the target view to the sample target coordinate system to obtain second sample projection features; determining the sample point cloud fusion result based on the second sample projection features; inputting the sample point cloud fusion result into the first semantic segmentation network for semantic segmentation to obtain the semantic segmentation prediction data.

[0092] Furthermore, the multi-frame transformed sample point clouds are projected from the sample target coordinate system to the target view respectively to obtain corresponding multi-frame first sample projection features, which may specifically include: for each frame of sample transformed point cloud, performing first feature extraction on the frame sample transformed point cloud to obtain first sample feature data; projecting the first sample feature data from the sample target coordinate system to the target view to obtain corresponding first preliminary sample projection features; and aggregating the first preliminary sample projection features in the target view to obtain the first sample projection features corresponding to the frame transformed point cloud.

[0093] Furthermore, the fused sample projection feature is projected from the target view to the sample target coordinate system to obtain a second sample projection feature, which may specifically include: performing a second feature extraction on the fused sample projection feature to obtain second sample feature data; projecting the second sample feature data from the target view to the sample target coordinate system to obtain a second preliminary sample projection feature; and performing bilinear interpolation processing on the second preliminary sample projection feature in the sample target coordinate system to obtain the second sample projection feature.

[0094] Furthermore, the initialization model may also include a first feature extraction network to be trained (e.g., MLP) and a second feature extraction network to be trained (e.g., 2D FCN); wherein the first feature extraction network is used to perform a first feature extraction on each frame of sample transformation point cloud to obtain the first sample feature data, and the second feature extraction network is used to perform a second feature extraction on the fused sample projection features to obtain the second sample feature data.

[0095] Among them, in the process of iterative training, in addition to iteratively optimizing the parameters of the multiple weight prediction networks to be trained, the parameters of the first semantic segmentation network, the first feature extraction network, and the second feature extraction network can also be iteratively optimized, thereby obtaining an optimization model including a pre-trained first semantic segmentation network, a pre-trained first feature extraction network, a pre-trained second feature extraction network, and a pre-trained weight prediction network. By using the optimization model to perform point cloud fusion of an embodiment of the present invention and semantically segmenting the point cloud fusion result, a more accurate semantic segmentation result can be obtained.

[0096] For more detailed implementation of each round of training for the initialization model, please refer to the previous article Figures 1 to 3 The content about performing point cloud fusion on the multiple frames of original point clouds is not repeated here.

[0097] In an embodiment of the present invention, the loss used to implement model training and optimization not only considers the difference between the semantic segmentation annotation data and the semantic segmentation prediction data of the sample point cloud fusion result of multiple frames of sample point clouds, but also introduces the difference between the semantic segmentation result based on a single frame of sample point clouds (i.e., the semantic segmentation reference data) and the semantic segmentation prediction data. Therefore, by combining multiple differences to calculate the function value of the loss function, compared with calculating the loss based on only a single factor, it helps to improve the accuracy of model training and obtain an optimized model with better prediction effect.

[0098] Reference Figure 5 , Figure 5 : is a schematic diagram of the structure of a point cloud fusion device in an embodiment of the present invention. The device may include:

[0099] The point cloud data acquisition module 51 is used to acquire multiple frames of original point clouds with time sequence, and transform the multiple frames of original point clouds to the same target coordinate system to obtain multiple frames of transformed point clouds;

[0100] A first projection module 52, configured to project the multi-frame transformed point clouds from the target coordinate system to a target view to obtain corresponding multi-frame first projection features, wherein the dimension of the target coordinate system is greater than the dimension of the target view;

[0101] The weighted fusion module 53 is used to perform weight prediction on the first projection feature of each frame to obtain the weight corresponding to the first projection feature of the frame, and then use the weights to perform weighted fusion on the first projection features of multiple frames to obtain a fused projection feature;

[0102] A second projection module 54, configured to project the fused projection feature from the target view to the target coordinate system to obtain a second projection feature;

[0103] The point cloud fusion result determination module 55 is used to determine the point cloud fusion result based on the second projection feature.

[0104] For the principle, specific implementation and beneficial effects of the point cloud fusion device, please refer to the previous article and Figures 1 to 4 The relevant description about the point cloud fusion method shown is not repeated here.

[0105] The embodiment of the present invention further provides a storage medium, such as a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor. Figures 1 to 4 The steps of the point cloud fusion method shown in any embodiment. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0106] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0107] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0108] The embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figures 1 to 4 The steps of the point cloud fusion method shown in any embodiment.

[0109] The embodiment of the present invention also provides a computer program product, including a computer program, wherein the computer program is executed by a processor Figures 1 to 4 The steps of the point cloud fusion method shown in any embodiment.

[0110] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0111] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0112] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0113] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0114] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A point cloud fusion method, characterized in that: include: Acquire multiple frames of original point clouds with time sequence, and transform the multiple frames of original point clouds to the same target coordinate system respectively to obtain multiple frames of transformed point clouds; Projecting the multi-frame transformed point clouds from the target coordinate system to the target view respectively to obtain corresponding multi-frame first projection features, wherein the dimension of the target coordinate system is greater than the dimension of the target view; Performing weight prediction on the first projection feature of each frame to obtain the weight corresponding to the first projection feature of the frame, and then using each weight to perform weighted fusion on the first projection features of multiple frames to obtain a fused projection feature; Projecting the fused projection feature from the target view to the target coordinate system to obtain a second projection feature; Based on the second projection feature, a point cloud fusion result is determined.

2. The method according to claim 1, characterized in that Projecting the multi-frame transformed point clouds from the target coordinate system to the target view respectively to obtain corresponding multi-frame first projection features, including: For each frame of transformed point cloud, performing first feature extraction on the frame of transformed point cloud to obtain first feature data; Projecting the first feature data from the target coordinate system to the target view to obtain a corresponding first preliminary projection feature; The first preliminary projection features are aggregated in the target view to obtain first projection features corresponding to the frame transformation point cloud.

3. The method according to claim 2, characterized in that The first preliminary projection feature includes a plurality of projection points, each projection point having feature values ​​of a plurality of dimensions; Aggregating the first preliminary projection features in the target view to obtain the first projection features corresponding to the frame transformation point cloud includes: Meshing the target view to obtain a plurality of meshes; For each grid containing the projection point, in each dimension of the projection point, the largest eigenvalue is selected from the eigenvalues ​​of each projection point in the grid in that dimension as the eigenvalue of the grid in that dimension; The feature values ​​of each grid of the target view in the multiple dimensions are used as first projection features corresponding to the frame transformation point cloud.

4. The method according to claim 2, characterized in that: The target view includes a range view, the target coordinate system is a three-dimensional coordinate system, the range view is a two-dimensional view, and the first feature data includes a plurality of first feature points; The first feature data is projected from the target coordinate system to the target view using the following formula: theta = z; phi = arctan2(y,x); Among them, x represents the x-axis coordinate value of the first feature point in the first feature data in the target coordinate system, y represents the y-axis coordinate value of the first feature point in the target coordinate system, z represents the z-axis coordinate value of the first feature point in the target coordinate system, arctan2() represents the inverse tangent function, theta represents the first coordinate value of the projection point corresponding to the first feature point in the range view, and phi represents the second coordinate value of the projection point corresponding to the first feature point in the range view.

5. The method according to claim 1, characterized in that Projecting the fused projection feature from the target view to the target coordinate system to obtain a second projection feature includes: Performing second feature extraction on the fused projection feature to obtain second feature data; Projecting the second feature data from the target view to the target coordinate system to obtain a second preliminary projection feature; Perform bilinear interpolation processing on the second preliminary projection feature in the target coordinate system to obtain the second projection feature.

6. The method according to claim 1, characterized in that The weight prediction of the first projection feature of each frame is achieved by using a pre-trained weight prediction network corresponding to the first projection feature of the frame; The following method is used to determine the multiple pre-trained weight prediction networks corresponding to the first projection features of the multiple frames: Determine an initialization model, and construct a training data set using multiple frames of sample point clouds with time sequence and semantic segmentation annotation data of one frame of sample point clouds, wherein the initialization model at least includes a first semantic segmentation network and a plurality of weight prediction networks to be trained, the plurality of weight prediction networks to be trained correspond to the multiple frames of sample point clouds one by one, and are respectively used to perform weight prediction on first sample projection features determined based on corresponding sample point clouds, the first semantic segmentation network is used to perform semantic segmentation on sample point cloud fusion results determined based on the multiple frames of sample point clouds, and output semantic segmentation prediction data; Using a preset loss function and the training data set, the initialization model is iteratively trained to iteratively optimize the parameters of the multiple weight prediction networks to be trained, thereby obtaining multiple pre-trained weight prediction networks; In each round of iterative training, the function value of the loss function is determined based at least on the semantic segmentation annotation data and the semantic segmentation prediction data output by the first semantic segmentation network.

7. The method according to claim 6, characterized in that In each round of iterative training, determining the function value of the loss function based at least on the semantic segmentation annotation data and the semantic segmentation prediction data output by the first semantic segmentation network includes: Calculating a first loss value based on a difference between the semantic segmentation annotation data and the semantic segmentation prediction data; Using a second semantic segmentation network, semantic segmentation is performed on the sample point cloud to which the semantic segmentation annotation data belongs, so as to obtain semantic segmentation reference data; Calculating a second loss value based on a difference between the semantic segmentation prediction data and the semantic segmentation reference data; A weighted operation is performed on the first loss value and the second loss value to obtain a function value of the loss function.

8. The method according to claim 1, characterized in that The target view includes a bird's-eye view and a range view, a second projection feature obtained by projecting the bird's-eye view to the target coordinate system is recorded as a second bird's-eye view feature, and a second projection feature obtained by projecting the range view to the target coordinate system is recorded as a second range view feature; Based on the second projection feature, determining a point cloud fusion result includes: Feature fusion is performed on the second bird's-eye view feature and the second range view feature to obtain the point cloud fusion result.

9. The method according to claim 1, characterized in that: The weights are used to perform weighted fusion on the first projection features of the multiple frames to obtain fused projection features, including: Determine feature data belonging to a static target and a dynamic target in the first projection features of the multiple frames respectively, and record them as first feature data and second feature data; Using the weights to perform weighted operation on the first feature data in the multiple frames of first projection features to obtain a static target fused projection feature; Selecting a frame of first projection features from the multiple frames of first projection features, and using second feature data in the selected first projection features as dynamic target fusion projection features; The static target fusion projection feature and the dynamic target fusion projection feature are combined to obtain the fusion projection feature.

10. The method according to claim 1, characterized in that After determining the point cloud fusion result, the method further includes: Perform semantic segmentation on the point cloud fusion result.

11. A point cloud fusion device, characterized in that: include: A point cloud data acquisition module is used to acquire multiple frames of original point clouds with time sequence, and transform the multiple frames of original point clouds to the same target coordinate system to obtain multiple frames of transformed point clouds; A first projection module, used to project the multi-frame transformed point clouds from the target coordinate system to a target view respectively to obtain corresponding multi-frame first projection features, wherein the dimension of the target coordinate system is greater than the dimension of the target view; A weighted fusion module is used to perform weight prediction on the first projection feature of each frame to obtain the weight corresponding to the first projection feature of the frame, and then use each weight to perform weighted fusion on the first projection features of multiple frames to obtain a fused projection feature; A second projection module, used for projecting the fused projection feature from the target view to the target coordinate system to obtain a second projection feature; The point cloud fusion result determination module is used to determine the point cloud fusion result based on the second projection feature.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the point cloud fusion method according to any one of claims 1 to 10 are executed.

13. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the point cloud fusion method according to any one of claims 1 to 10 are performed.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the point cloud fusion method according to any one of claims 1 to 10 are executed.

Images