Point cloud registration method and device based on deep collaborative attention network

Abstract

The application relates to the technical field of intelligent sensing, and provides a point cloud registration method and device based on a deep collaborative attention network. The method comprises the following steps: acquiring first point cloud data and second point cloud data; inputting the first point cloud data and the second point cloud data into a pre-trained deep collaborative attention network model to obtain a point cloud registration result between the first point cloud data and the second point cloud data; wherein the deep collaborative attention network model comprises a feature extraction network and a registration network; the feature extraction network comprises a multi-scale aggregation coding module; multi-scale point cloud features of the first point cloud data and the second point cloud data are extracted through different size pooling operations; and the registration network is used for obtaining the point cloud registration result according to the multi-scale point cloud features of the first point cloud data and the second point cloud data. The multi-scale point cloud features are acquired by using the multi-scale aggregation coding module, and the problem of poor registration precision of low-overlap point clouds is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent perception, in particular to a point cloud registration method based on a deep collaborative attention network, a point cloud registration device based on a deep collaborative attention network, an intelligent perception system applied to engineering machinery and a computer readable storage medium. BACKGROUND

[0002] The intelligent perception system brings great opportunities for improving the automation and intelligent level of the engineering machinery field, completing efficient automatic control, autonomous operation and machine group collaboration.

[0003] At present, the intelligent perception system in the engineering machinery field mainly uses monocular cameras, binocular cameras and single laser radars. Under actual working conditions, due to interference such as dust, light and irregular vibration of engineering equipment, the anti-interference performance of the intelligent perception system based on visual cameras is poor, and it is difficult to accurately locate the three-dimensional coordinates of the target. Due to the limited longitudinal field of view angle of the single laser radar and the low overlap rate of the sparse point cloud obtained, it will lead to large registration error and it is difficult to accurately describe the construction scene. The existing point cloud registration method is mainly applied to the point cloud registration task in the high overlap rate scene, and it is difficult to face the point cloud registration task in the low overlap rate scene which is more suitable for the actual engineering machinery field. Therefore, there is an urgent need for a registration method for low overlap rate point clouds to solve the problem of poor registration accuracy caused by the small number of corresponding point clouds in the low overlap rate scene, and to provide a high-precision three-dimensional perception scene for intelligent construction. SUMMARY

[0004] The purpose of the present application is to provide a point cloud registration method and device based on a deep collaborative attention network to solve the problem of poor point cloud registration accuracy in a low overlap rate scene.

[0005] In one aspect, the present application provides a point cloud registration method based on a deep collaborative attention network, comprising:

[0006] obtaining first point cloud data and second point cloud data, wherein there are overlapping point clouds between the first point cloud data and the second point cloud data;

[0007] inputting the first point cloud data and the second point cloud data into a pre-trained deep collaborative attention network model to obtain a point cloud registration result between the first point cloud data and the second point cloud data, wherein the deep collaborative attention network model comprises a feature extraction network and a registration network, the feature extraction network comprises at least one multi-scale aggregation encoding module, the multi-scale aggregation encoding module is used to extract multi-scale point cloud features of the first point cloud data and the second point cloud data through different size pooling operations, and the registration network is used to obtain a point cloud registration result according to the multi-scale point cloud features of the first point cloud data and the second point cloud data.

[0008] In the embodiment of the present application, the multi-scale aggregation encoding module is specifically used for:

[0009] The first feature of the input point cloud data is extracted through a convolution operation, and the input point cloud data includes the first point cloud data and the second point cloud data;

[0010] Different size pooling operations are performed on the first feature to obtain second features of different scales;

[0011] The dimensions of the second features of different scales are unified by using an upsampling operation to obtain the multi-scale point cloud feature.

[0012] In the embodiment of the present application, the multi-scale point cloud feature is obtained by using the upsampling operation to unify the dimensions of the second features of different scales, which includes: the dimensions of the second features of different scales are unified by using the upsampling operation to obtain third features, the scale of the third features is the same as the scale of the first features; the first features and the third features are added to obtain the multi-scale point cloud feature.

[0013] In the embodiment of the present application, the feature extraction network includes N multi-scale aggregation encoding modules, and the N multi-scale aggregation encoding modules are connected in series, the output of the i-th multi-scale aggregation encoding module is used as the input of the (i+1)-th multi-scale aggregation encoding module, i is a positive integer less than or equal to N.

[0014] In the embodiment of the present application, the output of the first multi-scale aggregation encoding module and the output of the second multi-scale aggregation encoding module are fused to serve as the input of the third multi-scale aggregation encoding module; in the case that i is greater than or equal to 3, the input of the i-th multi-scale aggregation encoding module and the output of the i-th multi-scale aggregation encoding module are fused to serve as the input of the (i+1)-th multi-scale aggregation encoding module.

[0015] In the embodiment of the present application, the registration network includes an attention-based overlapping saliency module;

[0016] The attention-based overlapping saliency module is used to obtain interaction attention information between the first point cloud data and the second point cloud data based on the multi-scale point cloud features of the first point cloud data and the second point cloud data, and to register the point cloud data based on the interaction attention information.

[0017] In the embodiment of the present application, the attention-based overlapping saliency module is specifically used for:

[0018] receive a first multi-scale point cloud feature and a second multi-scale point cloud feature, the first multi-scale point cloud feature being a multi-scale point cloud feature extracted by the feature extraction network from the first point cloud data, and the second multi-scale point cloud feature being a multi-scale point cloud feature extracted by the feature extraction network from the second point cloud data;

[0019] obtain an interaction attention weight between the first multi-scale point cloud feature and the second multi-scale point cloud feature as interaction attention information between the first point cloud data and the second point cloud data;

[0020] enhance the first multi-scale point cloud feature and the second multi-scale point cloud feature based on the interaction attention weight respectively to obtain a first enhanced feature and a second enhanced feature;

[0021] register the first enhanced feature and the second enhanced feature to obtain the point cloud registration result.

[0022] In the embodiment of the application, before the interaction attention weight between the first multi-scale point cloud feature and the second multi-scale point cloud feature is obtained, the method further comprises:

[0023] optimizing the first multi-scale point cloud feature and the second multi-scale point cloud feature into a first optimized feature and a second optimized feature respectively;

[0024] The interaction attention weight between the first multi-scale point cloud feature and the second multi-scale point cloud feature is obtained, specifically comprising: obtaining an interaction attention weight between the first optimized feature and the second optimized feature;

[0025] The first multi-scale point cloud feature and the second multi-scale point cloud feature are enhanced based on the interaction attention weight respectively to obtain a first enhanced feature and a second enhanced feature, specifically comprising: the first optimized feature and the second optimized feature are enhanced based on the interaction attention weight respectively to obtain a first enhanced feature and a second enhanced feature.

[0026] In the embodiment of the application, the first multi-scale point cloud feature and the second multi-scale point cloud feature are optimized into a first optimized feature and a second optimized feature respectively, comprising:

[0027] convolution operations are performed on the first multi-scale point cloud feature and the second multi-scale point cloud feature respectively;

[0028] activation functions are used to activate the features after the convolution operations to obtain activated features;

[0029] The activation feature of the first multi-scale point cloud feature is point multiplied with the first multi-scale point cloud feature to obtain a first optimized feature, and the activation feature of the second multi-scale point cloud feature is point multiplied with the second multi-scale point cloud feature to obtain a second optimized feature.

[0030] In the embodiment of the application, the convolution operation is respectively performed on the first multi-scale point cloud feature and the second multi-scale point cloud feature, including:

[0031] The feature channel number of the convolution kernel for optimizing the multi-scale point cloud feature is obtained.

[0032] The convolution kernel size is determined according to the feature channel number.

[0033] The first multi-scale point cloud feature and the second multi-scale point cloud feature are subjected to the convolution operation based on the convolution kernel size.

[0034] In the embodiment of the application, the interaction attention weight value between the first multi-scale point cloud feature and the second multi-scale point cloud feature is obtained, including: the activation feature of the first multi-scale point cloud feature is concatenated with the activation feature of the second multi-scale point cloud feature to obtain a first fusion feature; and the full connection and activation are performed on the first fusion feature to obtain the interaction attention weight value.

[0035] In the embodiment of the application, the first enhanced feature and the second enhanced feature are registered to obtain the point cloud registration result, including: the first enhanced feature and the second enhanced feature are concatenated to obtain a second fusion feature; and the multi-layer perception is performed on the second fusion feature to obtain a rotation quaternion, and the point cloud registration result includes the rotation quaternion.

[0036] Another aspect of the application provides a point cloud registration device based on a deep collaborative attention network, including:

[0037] The point cloud acquisition device is used for acquiring a plurality of point cloud data, including first point cloud data and second point cloud data, and there is an overlapping point cloud between the first point cloud data and the second point cloud data.

[0038] The deep collaborative attention network model is used for obtaining a point cloud registration result between the first point cloud data and the second point cloud data according to the input first point cloud data and second point cloud data.

[0039] The deep collaborative attention network model comprises a feature extraction network and a registration network, the feature extraction network comprises at least one multi-scale aggregation encoding module, the multi-scale aggregation encoding module is used for extracting multi-scale point cloud features of the first point cloud data and the second point cloud data through different size pooling operations, and the registration network is used for obtaining a point cloud registration result according to the multi-scale point cloud features of the first point cloud data and the second point cloud data.

[0040] The embodiment of the application also provides an intelligent perception system applied to an engineering machine, and the intelligent perception system adopts the point cloud registration method based on the deep collaborative attention network to register collected point cloud data.

[0041] The embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores computer instructions, and the computer instructions are used for implementing the point cloud registration method based on the deep collaborative attention network when being executed.

[0042] The point cloud registration method based on the deep collaborative attention network provided by the application uses a multi-scale aggregation encoding (MS-AE) module to construct a feature extraction network, and obtains multi-scale point cloud features with strong robustness. The MS-AE module is a basic unit of the point cloud feature extraction network, and different layers of the MS-AE module dig multi-level point cloud information from shallow to deep, relieve the interference of point cloud disorder, and improve the robustness of point cloud features, so that the problem of poor registration accuracy caused by low overlap rate or abnormal points between point clouds can be effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application. In the drawings:

[0044] Figure 1 A flowchart of the point cloud registration method based on the deep collaborative attention network provided by the embodiment of the application is shown in the figure;

[0045] Figure 2 A structure schematic diagram of the deep collaborative attention network model provided by the embodiment of the application is shown in the figure;

[0046] Figure 3 A structure schematic diagram of the multi-scale aggregation encoding (MS-AE) module provided by the embodiment of the application is shown in the figure;

[0047] Figure 4 A structure schematic diagram of the attention-based overlap saliency (AOS) module provided by the embodiment of the application is shown in the figure;

[0048] Figure 5A flowchart of a training method of a deep collaborative attention network model provided by an embodiment of the present application is shown in the figure.

[0049] Figure 6 A schematic diagram of iteration of a deep collaborative attention network model provided by an embodiment of the present application is shown in the figure.

[0050] Figure 7 A block diagram of a point cloud registration device based on a deep collaborative attention network provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] In order to make the technical solutions and advantages in the embodiments of the present application clearer, the exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0052] Terminology analysis:

[0053] Point cloud registration refers to inputting two point clouds (one point cloud as a source and the other point cloud as a target), and outputting a transformation to make the coincidence degree of the two point clouds as high as possible. Or, for two coordinate systems under different perspectives (such as a world coordinate system and a camera coordinate system), a transformation is obtained to make the two coordinate systems transform to a unified perspective.

[0054] Figure 1 A flowchart of a point cloud registration method based on a deep collaborative attention network provided by an embodiment of the present application is shown in the figure. Figure 1 As shown in the figure, the point cloud registration method based on a deep collaborative attention network provided by the present embodiment includes the following steps:

[0055] S110, obtaining first point cloud data and second point cloud data.

[0056] Specifically, the point cloud data under different perspectives of the same region in a working condition scene can be collected by the same laser radar device or multiple laser radar devices. For example, two point clouds are obtained by collecting data of the same position from different angles by two laser radar devices, and the two point clouds are defined as the first point cloud data and the second point cloud data. There are overlapping point clouds between the first point cloud data and the second point cloud data, that is, there are overlapping data between the two point clouds.

[0057] S120, inputting the first point cloud data and the second point cloud data into a pre-trained deep collaborative attention network model to obtain a point cloud registration result between the first point cloud data and the second point cloud data.

[0058] The deep collaborative attention network model comprises a feature extraction network and a registration network. The feature extraction network comprises at least one multi-scale aggregation encoding (MS-AE) module, and the MS-AE module is used for extracting multi-scale point cloud features of the first point cloud data and the second point cloud data through different size pooling operations. The registration network is used for obtaining a point cloud registration result according to the multi-scale point cloud features of the first point cloud data and the second point cloud data.

[0059] In the embodiment of the application, the MS-AE module is used to construct the feature extraction network, so that multi-scale point cloud features with strong robustness can be obtained. As a basic unit of the point cloud feature extraction network, the MS-AE module at different layers excavates multi-level point cloud information from shallow to deep, alleviates the interference of the unordered point cloud, improves the robustness of the point cloud features, and effectively solves the problem of poor registration accuracy caused by low overlap rate or abnormal points between point clouds.

[0060] In an embodiment, as shown in FIG. 1, the deep collaborative attention network model comprises a parameter-shared double-branch feature extraction network and a registration network based on an attention-based overlapping salient (AOS) module. The first branch feature extraction network and the second branch feature extraction network are both stacked by a plurality of MS-AE modules (5 MS-AE modules are adopted in the embodiment), and are used to extract multi-scale point cloud features of the first point cloud data and the second point cloud data, respectively. The AOS module is used to realize information interaction between the first point cloud data and the second point cloud data, highlight useful information and suppress redundant information. Then, a multi-layer perceptron is used to combine global fusion features to predict a rotation quaternion, and the point cloud registration result is obtained by using the rotation quaternion to perform point cloud registration. Figure 2 Figure 2 The network architecture of the AOS module and the specific working principle thereof will be described in detail in the following embodiments, but it is worth emphasizing that in the embodiment of the application, the registration network is not limited to the AOS module, but can also be a conventional registration network based on PCRNet, etc. For the purpose of simplifying the description, in the following embodiments, the registration network based on the AOS module will be mainly taken as an example for description.

[0061] In an embodiment, the feature extraction network of each branch comprises N MS-AE modules, the N MS-AE modules are connected in series, the output of the i th MS-AE module is used as the input of the i+1 th MS-AE module, and the output of the i th MS-AE module and the output of the i+1 th MS-AE module are fused and then used as the input of the i+2 th MS-AE module, where i is a positive integer less than or equal to N.

[0062] In an embodiment, the feature extraction network of each branch comprises N MS-AE modules, the N MS-AE modules are connected in series, the output of the i th MS-AE module is used as the input of the i+1 th MS-AE module, and the output of the i th MS-AE module and the output of the i+1 th MS-AE module are fused and then used as the input of the i+2 th MS-AE module, where i is a positive integer less than or equal to N. ​

[0063] In one specific embodiment, the output of the first MS-AE module is fused with the output of the second MS-AE module as the input of the third MS-AE module. In the case where i is greater than or equal to 3, the input of the i-th MS-AE module is fused with the output of the i-th MS-AE module as the input of the (i+1)-th MS-AE module.

[0064] As shown in Figure 2 In conjunction with one example, each branch of the feature extraction network includes 5 MS-AE modules, and the 5 MS-AE modules are connected in series, the output of the first MS-AE module is the input of the second MS-AE module. The output of the first MS-AE module is fused with the output of the second MS-AE module as the input of the third MS-AE module. The input of the third MS-AE module is fused with the output of the third MS-AE module as the input of the fourth MS-AE module, and the input of the fourth MS-AE module is fused with the output of the fourth MS-AE module as the input of the fifth MS-AE module. The output of the fifth MS-AE module is the input of the AOS module. Of course, it is easy to understand that the number of MS-AE modules can be adjusted as needed, which will not be illustrated one by one here.

[0065] The present embodiment utilizes the stacked structure of multiple MS-AE modules connected in series to obtain multi-level multi-scale features, further alleviates the interference of the disorder of point clouds, and improves the robustness of point cloud features.

[0066] In one embodiment, the MS-AE module extracts the first feature of the input point cloud data (the first point cloud data and the second point cloud data) through convolution operation, performs pooling operation of different sizes on the first feature to obtain second features of different scales, and uses up-sampling operation to unify the dimensions of the second features of different scales to obtain multi-scale point cloud features. The MS-AE module improves the amount of information describing a single point cloud through convolution operation, making the matching of the description features more reliable.

[0067] In one specific embodiment, the MS-AE module uses up-sampling operation to unify the dimensions of the second features of different scales to obtain third features of the same scale as the first features, and adds the first features extracted by convolution operation and the third features to obtain multi-scale point cloud features. The MS-AE module constructs a residual structure by adding the first features extracted by convolution operation and the third features of unified dimensions, which improves the training efficiency of the network model.

[0068] In an embodiment, the AOS module receives the first multi-scale point cloud features extracted by the feature extraction network from the first point cloud data and the second multi-scale point cloud features extracted by the feature extraction network from the second point cloud data, obtains the interaction attention weight (i.e., interaction attention information) between the first multi-scale point cloud features and the second multi-scale point cloud features, respectively enhances the first multi-scale point cloud features and the second multi-scale point cloud features based on the interaction attention weight, obtains the first enhanced features and the second enhanced features, and registers the first enhanced features and the second enhanced features to obtain the point cloud registration result. The AOS module enhances the first multi-scale point cloud features and the second multi-scale point cloud features by using the interaction attention weight, highlights the overlapping area between the first point cloud and the second point cloud, and solves the problem of poor registration accuracy between point clouds with low overlap.

[0069] In a specific embodiment, in the process of obtaining the interaction attention weight between the first multi-scale point cloud features and the second multi-scale point cloud features, the first multi-scale point cloud features and the second multi-scale point cloud features are first optimized into the first optimized features and the second optimized features, and then the interaction attention weight between the first optimized features and the second optimized features is obtained. In the process of enhancing the first multi-scale point cloud features and the second multi-scale point cloud features, the first optimized features and the second optimized features are respectively enhanced based on the interaction attention weight to obtain the first enhanced features and the second enhanced features, so as to highlight the overlapping area between the first point cloud and the second point cloud.

[0070] In a specific embodiment, the first enhanced features and the second enhanced features obtained by enhancing the optimized features based on the interaction attention weight are concatenated to obtain the second fusion features, the multi-layer perception is used to perceive the second fusion features, the rotation quaternion is predicted, the point cloud registration is performed according to the rotation quaternion, and the point cloud registration result is obtained.

[0071] In a specific embodiment, when the first multi-scale point cloud features and the second multi-scale point cloud features are optimized, the convolution operation is first performed on the first multi-scale point cloud features and the second multi-scale point cloud features, the activation function is used to activate the features after the convolution operation to obtain the activation features, the point multiplication operation is performed on the activation features of the first multi-scale point cloud features and the first multi-scale point cloud features to obtain the first optimized features, and the point multiplication operation is performed on the activation features of the second multi-scale point cloud features and the second multi-scale point cloud features to obtain the second optimized features.

[0072] In one specific implementation, during the convolution operation on the first and second multi-scale point cloud features, the number of feature channels of the convolution kernel used to optimize the multi-scale point cloud features is obtained. The convolution kernel size is determined based on the number of feature channels, and the convolution operation is performed on the first and second multi-scale point cloud features based on the convolution kernel size. In this process, determining the convolution kernel size based on the number of feature channels allows for adaptive adjustment of the convolution kernel size, thereby achieving adaptive convolution operation and resulting in better feature optimization.

[0073] In one specific implementation, after obtaining the activation features of the first multi-scale point cloud feature and the activation features of the second multi-scale point cloud feature, the activation features of the first multi-scale point cloud feature and the activation features of the second multi-scale point cloud feature are concatenated to obtain the first fused feature, and the first fused feature is fully connected and activated to obtain the interactive attention weight.

[0074] In one specific implementation, the activation feature of the first fused feature is multiplied by the first optimized feature to obtain the first optimized fused feature. The activation feature of the first fused feature is then multiplied by the second optimized feature to obtain the second optimized fused feature. The first and second optimized fused features are concatenated and then subjected to global average pooling. The globally average pooled features are then fused to obtain the second fused feature. The second fused feature is used as the global fused feature, and a rotation quaternion is obtained based on this global fused feature.

[0075] In a specific example, such as Figure 3 As shown, the MS-AE module includes sub-modules such as 1D convolution operations, 1D pooling operations of different sizes, and upsampling operations. The operation steps of each sub-module are as follows:

[0076] 1) Input the first and second point cloud data into the MS-AE module, and use 1D convolution operations to extract features from the point cloud data to obtain the first feature describing a single point cloud. Its operating formula is:

[0077]

[0078] Where i represents the point cloud index, j represents the MS-AE module index, i∈{1,2}, j∈{1,…,5}, Let be the input of the j-th MS-AE module in the feature extraction network for point cloud i, and let Conv(·) represent the convolution operation. express Features extracted by convolution operations.

[0079] 2) Using 1D pooling operations with filters of different sizes for the first feature The extraction is performed to obtain second features of different scales, and the dimension of the second features of different scales is unified through an upsampling operation to obtain multi-scale point cloud features; and the operation formula is:

[0080]

[0081] wherein, k l represents the filter size of the pooling operation, k l ∈{4, 8, 16}, represents that the 1-dimensional pooling operation is performed on the point cloud feature to obtain the second feature, and Up(·) represents the upsampling operation, represents the multi-scale point cloud feature.

[0082] 3) residual operation (addition operation) is performed based on the single point cloud feature and the multi-scale point cloud feature to improve the training efficiency of the network model, and the residual operation is represented as:

[0083]

[0084] In the embodiment of the application, a plurality of MS-AE modules are used to form a double-branch feature extraction network as shown in Figure 2 , multi-scale point cloud features X5 1 of the first point cloud and multi-scale point cloud features X5 2 of the second point cloud are extracted.

[0085] The AOS module adopts a two-stage strategy to optimize the multi-scale point cloud features X5 1 of the first point cloud and the multi-scale point cloud features X5 2 of the second point cloud, and highlights the role of the overlapping area in the regression transformation matrix through feature interaction.

[0086] In a specific example, as shown in Figure 4 , the specific operation process of the AOS module is as follows:

[0087] 1) the first multi-scale point cloud feature and the second multi-scale point cloud feature are respectively subjected to global mean processing, convolution operation is performed on the features after the global mean processing based on the adaptive convolution kernel size, and the features after the convolution operation are activated by using an activation function to obtain activated features , and the operation formula is:

[0088]

[0089] wherein, Let f represent the multi-scale point cloud features of point cloud i, GAP(·) represent the global mean operation, Conv(·) represent the convolution operation, f is the kernel size of the convolution operation, and σ(·) is the Sigmoid activation function.

[0090] The kernel size f is determined based on the number of feature channels in the kernel to achieve adaptive convolution operations. The formula for calculating the kernel size f is as follows:

[0091] Where C is the number of feature channels, |·| odd Indicates rounding down.

[0092] Activation features of the first multi-scale point cloud features Features of the first multi-scale point cloud Performing a dot product operation yields the first optimized feature. Activation features of the second multi-scale point cloud features With second multi-scale point cloud features Performing a dot product operation yields the second optimized feature. Its operating formula is:

[0093]

[0094] in, Represents the multi-scale point cloud features of point cloud i. The activation feature represents the multi-scale point cloud features of point cloud i.

[0095] 2) Activation features of the first multi-scale point cloud features Activation features of second-scale point cloud features The first fused feature is obtained by concatenating the features, and the interactive attention weights are obtained by performing fully connected and activated operations on the first fused feature. Its operating formula is:

[0096]

[0097] in, Indicates feature concatenation, FC i (·) represents a fully connected layer, and σ(·) is the Sigmoid activation function.

[0098] 3) Based on interaction attention weights For the first optimized feature With the second optimization feature Enhancement is performed to obtain the first enhanced feature. With the second enhancement feature To enhance the overlapping region between the first and second point clouds, the operation formula is as follows:

[0099]

[0100]

[0101] wherein, represents the interaction attention weight of the first point cloud, represents the interaction attention weight of the second point cloud.

[0102] 4) concatenating the first enhanced feature with the second enhanced feature to obtain a second fusion feature, perceiving the second fusion feature based on a multi-layer perception MLP(·) to predict a rotation quaternion E, and performing point cloud registration based on the rotation quaternion E to obtain a point cloud registration result, and the operation formula is:

[0103]

[0104] wherein, E is a rotation quaternion, MLP(·) represents a multi-layer perception, GAP(·) represents a global mean operation, represents feature concatenation.

[0105] In an embodiment, in the step S120, an iterative strategy similar to an iterative closest point (ICP) algorithm is used to solve the transformation matrix, the point cloud data is input into the deep collaborative attention network model for iterative operation to obtain the point cloud registration result. As shown in Figure 6 the specific process of the deep collaborative attention network model iteration is that, in the first iteration operation, the first point cloud data and the second point cloud data (the first point cloud as the target point cloud and the second point cloud as the source point cloud) are input into the network model to obtain the intermediate transformation matrix E(1) of the first iteration. In the second iteration operation, the product of the first point cloud data input in the first iteration period and the intermediate transformation matrix E(1) obtained in the first iteration period is taken as the intermediate point cloud data, and the intermediate point cloud data and the second point cloud data are input into the network model to obtain the intermediate transformation matrix E(2) of the second iteration. In this way, n rounds of iteration operations are performed, in the nth iteration operation, the product of the intermediate point cloud data input in the n-1th iteration period and the intermediate transformation matrix obtained in the n-1th iteration period is taken as the intermediate point cloud data in the nth iteration period, and the intermediate point cloud data and the second point cloud data are input into the network model to obtain the intermediate transformation matrix E(n) of the third iteration. After n rounds of iteration operations, the intermediate transformation matrix converges, the iteration operation is ended, and the optimal transformation matrix is obtained. The product E of the intermediate transformation matrices obtained in all rounds of iteration operations is determined as the point cloud registration result.

[0106] The embodiment of the present application also provides a training method of the deep collaborative attention network model. As shown inFigure 5 The training method includes the following steps:

[0107] S510, obtaining point cloud data to construct a training set; S520, inputting the training sample into an initial deep collaborative attention network model to obtain a predicted transformation matrix;

[0108] S530, calculating a loss value of a loss function according to the predicted transformation matrix and the annotation result;

[0109] S540, updating parameters of the initial deep collaborative attention network model according to the loss value until the loss value converges, to obtain a trained deep collaborative attention network model.

[0110] In the step S510, the training set includes training samples and annotation results corresponding to the training samples, each training sample includes first training point cloud data and second training point cloud data, and the annotation result is used to reflect the transformation relationship between the first training point cloud data and the second training point cloud data.

[0111] In the step S530, the loss function is used to calculate the distance between the first transformed point cloud and the second transformed point cloud, wherein the first transformed point cloud is obtained by transforming the first training point cloud data using the predicted transformation matrix, and the second transformed point cloud is obtained by transforming the first training point cloud data using the annotation result.

[0112] In an embodiment, the second transformed point cloud is obtained by inputting the annotation result into a pre-constructed fully connected network, outputting a bijective function, and transforming the first training point cloud data based on the bijective function to obtain the second transformed point cloud. Transforming the point cloud based on the bijective function can improve the robustness of the deep collaborative attention network model obtained by training.

[0113] In an embodiment, the loss function is specifically used to calculate the bulldozer distance, Euclidean distance or KL distance (Kullback-Leibler Divergence, Kullback-Leibler difference) between the first transformed point cloud and the second transformed point cloud.

[0114] In a specific embodiment, the bulldozer movement distance function is used as the loss function of the deep collaborative attention network model to realize end-to-end training of the model. In the process of calculating the loss function, a bijective function is determined to minimize the point pair distance based on the bulldozer distance. Specifically, the L2 norm between each component element of the first point cloud data or the second point cloud data after rotation quaternion transformation and the value mapped by the bijective function is calculated, and the minimum value in the mean of the calculated L2 norm is taken as the bulldozer distance between one of the first point cloud data and the second point cloud data after rotation quaternion transformation and the other. Its calculation formula is:

[0115]

[0116] wherein X 1 represents the first point cloud, X 2 represents the second point cloud, EMD(·) is a bulldozer distance function, and ψ(·) is a bijective function, is obtained by rotating the quaternion E, and the parameter x is 1 a constituent element of X , and ||x-ψ(x)||2 represents an L2 norm.

[0117] The embodiment utilizes the bulldozer distance function as a network loss function to train the network model, maps and changes the parameter x through the bijective function, minimizes the Euclidean distance between the two groups of elements, so as to obtain a better approximation of the real rotation quaternion, improve the point cloud registration accuracy under a low overlap rate, and improve the convergence and robustness of network training.

[0118] The embodiment of the application also provides a point cloud registration device based on a deep collaborative attention network. Figure 7 As shown in the figure, the point cloud registration device comprises a point cloud acquisition module and a deep collaborative attention network model. The point cloud acquisition module is used to acquire a plurality of point cloud data, and the plurality of point cloud data comprises first point cloud data and second point cloud data. There is an overlapping point cloud between the first point cloud data and the second point cloud data. The deep collaborative attention network model is used to obtain a point cloud registration result between the first point cloud data and the second point cloud data according to the input first point cloud data and second point cloud data. The deep collaborative attention network model comprises a feature extraction network and an attention-based overlapping saliency (AOS) module. The feature extraction network is used to extract multi-scale point cloud features of the point cloud data. The AOS module is used to acquire interactive attention information between the plurality of point cloud data based on the multi-scale point cloud features of the plurality of point cloud data, and perform registration on the plurality of point cloud data based on the interactive attention information.

[0119] In an embodiment, referring to Figure 2 , the feature extraction network of the deep collaborative attention network model is a parameter-shared double-branch feature extraction network. The feature extraction network of each branch is stacked by a plurality of MS-AE module layers to extract multi-scale point cloud features of the first point cloud data and the second point cloud data, respectively. Then, the AOS module is used to realize information interaction between the first point cloud data and the second point cloud data, highlight useful information and suppress redundant information. A multi-layer perceptron is used to combine global fusion features to predict a rotation quaternion, and the rotation quaternion is used to perform point cloud registration to obtain a point cloud registration result.

[0120] In an embodiment, the feature extraction network of each branch comprises N MS-AE modules, the N MS-AE modules are connected in series, the output of the i th MS-AE module is used as the input of the i + 1 th MS-AE module. The output of the i th MS-AE module and the output of the i + 1 th MS-AE module are fused and used as the input of the i + 2 th MS-AE module, where i is a positive integer less than or equal to N.

[0121] In an embodiment, the output of the 1 st MS-AE module and the output of the 2 nd MS-AE module are fused and used as the input of the 3 rd MS-AE module. In the case where i is greater than or equal to 3, the input of the i th MS-AE module and the output of the i th MS-AE module are fused and used as the input of the i + 1 th MS-AE module.

[0122] With reference to Figure 2 , in an example, the feature extraction network of each branch comprises 5 MS-AE modules, the 5 MS-AE modules are connected in series, the output of the 1 st MS-AE module is used as the input of the 2 nd MS-AE module. The output of the 1 st MS-AE module and the output of the 2 nd MS-AE module are fused and used as the input of the 3 rd MS-AE module. The input of the 3 rd MS-AE module and the output of the 3 rd MS-AE module are fused and used as the input of the 4 th MS-AE module, the input of the 4 th MS-AE module and the output of the 4 th MS-AE module are fused and used as the input of the 5 th MS-AE module. The output of the 5 th MS-AE module is used as the input of the AOS module.

[0123] In an embodiment, the MS-AE module extracts the first feature of the input point cloud data (the first point cloud data and the second point cloud data) through convolution operation, performs pooling operation on the first feature to obtain second features of different scales, and uses up-sampling operation to unify the dimensions of the second features of different scales to obtain multi-scale point cloud features. The MS-AE module improves the amount of information describing a single point cloud through convolution operation, so that the matching of the description features is more reliable.

[0124] In an embodiment, the MS-AE module uses up-sampling operation to unify the dimensions of the second features of different scales to obtain third features of the same scale as the first features, and adds the first features and the third features to obtain multi-scale point cloud features. The MS-AE module constructs a residual structure by adding the first features extracted by convolution operation and the third features obtained by unifying the dimensions, so as to improve the training efficiency of the network model.

[0125] In an embodiment, the AOS module receives the first multi-scale point cloud features extracted by the feature extraction network from the first point cloud data and the second multi-scale point cloud features extracted by the feature extraction network from the second point cloud data, obtains the interaction attention weight (i.e., interaction attention information) between the first multi-scale point cloud features and the second multi-scale point cloud features, respectively enhances the first multi-scale point cloud features and the second multi-scale point cloud features based on the interaction attention weight, obtains the first enhanced features and the second enhanced features, and registers the first enhanced features and the second enhanced features to obtain the point cloud registration result. The AOS module enhances the first multi-scale point cloud features and the second multi-scale point cloud features by using the interaction attention weight, highlights the overlapping area between the first point cloud and the second point cloud, and solves the problem of poor registration accuracy between point clouds with low overlap.

[0126] In a specific embodiment, in the process of obtaining the interaction attention weight between the first multi-scale point cloud features and the second multi-scale point cloud features, the first multi-scale point cloud features and the second multi-scale point cloud features are first optimized into the first optimized features and the second optimized features, and then the interaction attention weight between the first optimized features and the second optimized features is obtained. In the process of enhancing the first multi-scale point cloud features and the second multi-scale point cloud features, the first optimized features and the second optimized features are respectively enhanced based on the interaction attention weight to obtain the first enhanced features and the second enhanced features, so as to highlight the overlapping area between the first point cloud and the second point cloud.

[0127] In a specific embodiment, the first enhanced features and the second enhanced features obtained by enhancing the optimized features based on the interaction attention weight are concatenated to obtain the second fusion features, the multi-layer perception is used to perceive the second fusion features, the rotation quaternion is predicted, the point cloud registration is performed according to the rotation quaternion, and the point cloud registration result is obtained.

[0128] In a specific embodiment, when the first multi-scale point cloud features and the second multi-scale point cloud features are optimized, the convolution operation is first performed on the first multi-scale point cloud features and the second multi-scale point cloud features, the activation function is used to activate the features after the convolution operation to obtain the activation features, the point multiplication operation is performed on the activation features of the first multi-scale point cloud features and the first multi-scale point cloud features to obtain the first optimized features, and the point multiplication operation is performed on the activation features of the second multi-scale point cloud features and the second multi-scale point cloud features to obtain the second optimized features.

[0129] In a specific embodiment, in the process of respectively performing convolution operations on the first multi-scale point cloud feature and the second multi-scale point cloud feature, the feature channel number of the convolution kernel for optimizing the multi-scale point cloud feature is obtained, the convolution kernel size is determined according to the feature channel number, and the first multi-scale point cloud feature and the second multi-scale point cloud feature are subjected to convolution operations based on the convolution kernel size. In the above process, the convolution kernel size is adaptively adjusted according to the feature channel number of the convolution kernel, thereby realizing adaptive convolution operation to achieve better feature optimization effect.

[0130] In a specific embodiment, after obtaining the activated features of the first multi-scale point cloud feature and the activated features of the second multi-scale point cloud feature, the activated features of the first multi-scale point cloud feature and the activated features of the second multi-scale point cloud feature are concatenated to obtain a first fusion feature, and the first fusion feature is subjected to full connection and activation to obtain an interaction attention weight.

[0131] It should be noted that in the embodiments of the present application, the registration network of the deep collaborative attention network model is not limited to the AOS module, but can also be a conventional registration network based on PCRNet, etc. The specific structures of the MS-AE module and the AOS module of the deep collaborative attention network model of the present embodiment can be understood by referring to the corresponding descriptions and effects of the embodiments shown in Figure 2 to Figure 4

[0132] The point cloud registration device based on the deep collaborative attention network provided in the embodiments of the present application uses the MS-AE module to construct a feature extraction network, obtains a multi-scale point cloud feature with strong robustness, alleviates the interference of the disorder of the point cloud, and improves the robustness of the point cloud feature; uses the AOS module based on attention to realize information interaction of each group of point cloud features, highlight the key information of each group of point cloud features, explicitly mine the point cloud overlapping area information, highlight the role of the overlapping area in the regression transformation matrix, and solve the problem of poor registration accuracy caused by the small number of corresponding point clouds in the low overlap rate scene.

[0133] The embodiments of the present application also provide an intelligent perception system applied to engineering machinery, which uses the point cloud registration method based on the deep collaborative attention network to register the collected point cloud data. The point cloud registration method and the point cloud registration device combine the attention mechanism to explicitly mine the point cloud segment overlapping area information, solve the problem of poor registration accuracy caused by the small number of corresponding point clouds in the low overlap rate scene. The registration device is installed in the intelligent perception system of the engineering machinery or the point cloud registration method is used, which provides a high-precision three-dimensional perception scene for intelligent construction, feeds back the registration result to the engineering machinery equipment to perform subsequent path planning, obstacle avoidance, etc., and the whole process does not need manual intervention, thereby improving the intelligent level of the engineering machinery.

[0134] ​The embodiment of the present application further provides a computer readable storage medium, wherein computer instructions are stored in the computer readable storage medium, and the computer instructions are used to implement the point cloud registration method based on the deep collaborative attention network or the training method of the deep collaborative attention network model.

[0135] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as object-oriented programming language Java and interpreted scripting language JavaScript.

[0136] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flow or flows and / or block or blocks.

[0137] These computer program instructions can also be stored in a computer readable memory capable of directing the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flow or flows and / or block or blocks.

[0138] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flow or flows and / or block or blocks.

[0139] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.

[0140] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A point cloud registration method based on a deep co-attention network, characterized in that, The method comprises the following steps: obtaining first point cloud data and second point cloud data, wherein there is overlapping point cloud between the first point cloud data and the second point cloud data; inputting the first point cloud data and the second point cloud data into a pre-trained deep collaborative attention network model to obtain a point cloud registration result between the first point cloud data and the second point cloud data, wherein the deep collaborative attention network model comprises a feature extraction network and a registration network, the feature extraction network comprises at least one multi-scale aggregation encoding module, the multi-scale aggregation encoding module is used for extracting multi-scale point cloud features of the first point cloud data and the second point cloud data through different size pooling operations, and the registration network is used for obtaining a point cloud registration result according to the multi-scale point cloud features of the first point cloud data and the second point cloud data.

2. The point cloud registration method based on deep coattentive network according to claim 1, characterized in that, The multi-scale aggregation encoding module is specifically used for: extracting first features of input point cloud data through a convolution operation, wherein the input point cloud data comprises the first point cloud data and the second point cloud data; performing different size pooling operations on the first features to obtain second features of different scales; realizing dimension unification of the second features of different scales by using an up-sampling operation to obtain the multi-scale point cloud features.

3. The point cloud registration method based on deep coattentive network according to claim 2, characterized in that, The realization of the dimension unification of the second features of different scales by using the up-sampling operation to obtain the multi-scale point cloud features comprises: realizing dimension unification of the second features of different scales by using an up-sampling operation to obtain third features, wherein the scale of the third features is the same as the scale of the first features; adding the first features and the third features to obtain the multi-scale point cloud features.

4. The point cloud registration method based on deep coattentive network of claim 1, characterized in that, The feature extraction network comprises N multi-scale aggregation encoding modules, the N multi-scale aggregation encoding modules are connected in series, the output of an i-th multi-scale aggregation encoding module is used as the input of an (i+1)-th multi-scale aggregation encoding module, i is a positive integer less than or equal to N.

5. The point cloud registration method based on deep coattentive network according to claim 4, characterized in that, the output of the first multi-scale aggregation encoding module and the output of the second multi-scale aggregation encoding module are fused to serve as the input of the third multi-scale aggregation encoding module; in the case that i is greater than or equal to 3, the input of the i-th multi-scale aggregation encoding module and the output of the i-th multi-scale aggregation encoding module are fused to serve as the input of the (i+1)-th multi-scale aggregation encoding module.

6. The point cloud registration method based on deep coattentive network of claim 1, wherein, The registration network comprises an attention-based overlapping significant module; the attention-based overlapping significant module is used for obtaining interaction attention information between the first point cloud data and the second point cloud data based on the multi-scale point cloud features of the first point cloud data and the second point cloud data, and performing registration on point cloud data based on the interaction attention information.

7. The point cloud registration method based on deep coattentive network according to claim 6, characterized in that, The attention-based overlapping significant module is specifically used for: receiving first multi-scale point cloud features and second multi-scale point cloud features, wherein the first multi-scale point cloud features are multi-scale point cloud features extracted by the feature extraction network from the first point cloud data, and the second multi-scale point cloud features are multi-scale point cloud features extracted by the feature extraction network from the second point cloud data; obtaining an interaction attention weight between the first multi-scale point cloud feature and the second multi-scale point cloud feature, the interaction attention weight serving as interaction attention information between the first point cloud data and the second point cloud data; enhancing the first multi-scale point cloud feature and the second multi-scale point cloud feature based on the interaction attention weight respectively to obtain a first enhanced feature and a second enhanced feature; registering the first enhanced feature and the second enhanced feature to obtain the point cloud registration result.

8. The point cloud registration method based on deep coattentive network according to claim 7, characterized in that, Before the obtaining the interaction attention weight between the first multi-scale point cloud feature and the second multi-scale point cloud feature, the method further comprises: optimizing the first multi-scale point cloud feature and the second multi-scale point cloud feature into a first optimized feature and a second optimized feature respectively; the obtaining the interaction attention weight between the first multi-scale point cloud feature and the second multi-scale point cloud feature comprises: obtaining an interaction attention weight between the first optimized feature and the second optimized feature; the enhancing the first multi-scale point cloud feature and the second multi-scale point cloud feature based on the interaction attention weight respectively to obtain a first enhanced feature and a second enhanced feature comprises: enhancing the first optimized feature and the second optimized feature based on the interaction attention weight respectively to obtain a first enhanced feature and a second enhanced feature.

9. The point cloud registration method based on deep coattentive network according to claim 8, characterized in that, The optimizing the first multi-scale point cloud feature and the second multi-scale point cloud feature into a first optimized feature and a second optimized feature respectively comprises: performing convolution operation on the first multi-scale point cloud feature and the second multi-scale point cloud feature respectively; activating the feature after the convolution operation by using an activation function to obtain an activated feature; performing point multiplication operation on the activated feature of the first multi-scale point cloud feature and the first multi-scale point cloud feature to obtain the first optimized feature, and performing point multiplication operation on the activated feature of the second multi-scale point cloud feature and the second multi-scale point cloud feature to obtain the second optimized feature.

10. The point cloud registration method based on deep coattentive network according to claim 9, characterized in that, The performing convolution operation on the first multi-scale point cloud feature and the second multi-scale point cloud feature respectively comprises: obtaining a feature channel number of a convolution kernel for optimizing the multi-scale point cloud feature; determining a convolution kernel size according to the feature channel number; performing convolution operation on the first multi-scale point cloud feature and the second multi-scale point cloud feature based on the convolution kernel size.

11. The point cloud registration method based on deep coattentive network of claim 9, characterized in that, The obtaining the interaction attention weight between the first multi-scale point cloud feature and the second multi-scale point cloud feature comprises: concatenating the activated feature of the first multi-scale point cloud feature and the activated feature of the second multi-scale point cloud feature to obtain a first fusion feature; performing full connection and activation on the first fusion feature to obtain the interaction attention weight.

12. The point cloud registration method based on deep coattentive network according to claim 7, characterized in that, The registering the first enhanced feature and the second enhanced feature to obtain the point cloud registration result comprises: concatenating the first enhanced feature and the second enhanced feature to obtain a second fusion feature; perceiving the second fusion feature based on a multi-layer perception to obtain a rotation quaternion, and the point cloud registration result comprises the rotation quaternion.

13. A point cloud registration apparatus based on a deep co-attention network, characterized in that, comprises: A point cloud acquisition module is configured to acquire a plurality of point cloud data, the plurality of point cloud data comprising first point cloud data and second point cloud data, and there being overlapping point clouds between the first point cloud data and the second point cloud data; A deep collaborative attention network model is configured to obtain a point cloud registration result between the first point cloud data and the second point cloud data according to the input first point cloud data and second point cloud data. The deep collaborative attention network model comprises a feature extraction network and a registration network, the feature extraction network comprises at least one multi-scale aggregation encoding module, the multi-scale aggregation encoding module is configured to extract multi-scale point cloud features of the first point cloud data and the second point cloud data through different size pooling operations, and the registration network is configured to obtain a point cloud registration result according to the multi-scale point cloud features of the first point cloud data and the second point cloud data.

14. An intelligent perception system for an engineering machine, characterized by The intelligent perception system adopts the deep collaborative attention network-based point cloud registration method in any one of claims 1-12 to register the collected point cloud data.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed to implement the deep collaborative attention network-based point cloud registration method in any one of claims 1-12.

Images