Multi-modal three-dimensional point cloud registration method and system, terminal and storage medium

Abstract

The invention discloses a multi-modal three-dimensional point cloud registration method and system, a terminal and a storage medium, and the method comprises the steps: obtaining initial point cloud data, carrying out the point cloud feature extraction processing and point cloud image feature extraction processing, and obtaining point cloud features and point cloud image features; performing feature enhancement processing and feature fusion processing on the point cloud features and the point cloud image features to obtain mixed features; performing feature splicing processing and significance score calculation according to the mixed features to obtain significance scores, and performing feature selection processing according to the significance scores to obtain key points and key point features; and performing matching matrix construction processing and related weight calculation according to the key points and the key point features to obtain target rigid transformation parameters, and aligning the initial point cloud data according to the target rigid transformation parameters. According to the method, the optimal rigid transformation parameter is calculated, and the point cloud data is registered according to the optimal rigid transformation parameter, so that accurate alignment of the point cloud data can be quickly realized.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional point cloud technology, and in particular to a multi-modal three-dimensional point cloud registration method, system, terminal and computer-readable storage medium. Background Art

[0002] 3D point cloud registration is a key technology in the field of computer vision. Its core goal is to accurately align point cloud data from different positions and postures, so as to achieve data display, comparison and fusion in a unified coordinate system. With the rapid development of high-precision sensor technology, point cloud has gradually become the mainstream form of 3D data representation. Point cloud registration technology has been widely used in autonomous driving, augmented reality, 3D printing and other fields.

[0003] However, point cloud data naturally has characteristics such as partiality and sparsity, which leads to large accuracy deviations when estimating the rotation and translation of the point cloud at different perspectives, making it impossible to achieve accurate alignment of the point cloud.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal three-dimensional point cloud registration method, system, terminal and computer-readable storage medium, aiming to solve the problem that point cloud data in the prior art naturally has characteristics such as partiality and sparsity, which leads to large accuracy deviations when estimating the rotation and translation of the point cloud at different viewing angles, and the problem of being unable to achieve accurate alignment of the point cloud.

[0006] To achieve the above object, the present invention provides a multimodal three-dimensional point cloud registration method, which comprises the following steps: Acquire initial point cloud data, and perform point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features; Performing feature enhancement processing and feature fusion processing on the point cloud features and the point cloud image features to obtain mixed features; Performing feature concatenation processing and significance score calculation according to the mixed features to obtain a significance score, and performing feature selection processing according to the significance score to obtain key points and key point features; A matching matrix construction process and related weight calculation are performed according to the key points and the key point features to obtain target rigid transformation parameters, and the initial point cloud data are aligned according to the target rigid transformation parameters.

[0007] Optionally, in the multimodal three-dimensional point cloud registration method, the initial point cloud data includes source point cloud data and target point cloud data; the point cloud features include source point cloud features and target point cloud features; The step of acquiring initial point cloud data and performing point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features specifically includes: Acquire the source point cloud data and the target point cloud data, perform point cloud feature extraction processing on the source point cloud data and the target point cloud data, and obtain the source point cloud features and the target point cloud features; Point cloud image feature extraction processing is performed on the source point cloud data and the target point cloud data to obtain point cloud image features.

[0008] Optionally, the multimodal three-dimensional point cloud registration method, wherein the step of performing point cloud image feature extraction processing on the source point cloud data and the target point cloud data to obtain point cloud image features, specifically includes: Performing multi-view projection processing on the source point cloud data and the target point cloud data to obtain multiple two-dimensional image sequences; Using a CNN network to extract features from the plurality of two-dimensional image sequences to obtain a plurality of projection features; An aggregation function is used to fuse the plurality of projection features to obtain the point cloud image features.

[0009] Optionally, the multimodal three-dimensional point cloud registration method, wherein the initial point cloud data is obtained, and the initial point cloud data is subjected to point cloud feature extraction processing and point cloud image feature extraction processing to obtain point cloud features and point cloud image features, further comprising: Acquire overlapping point features and non-overlapping point features between the source point cloud data and the target point cloud data, construct a feature pair set according to the overlapping point features and the non-overlapping point features, and perform overlapping contrast learning on the source point cloud data and the target point cloud data according to the feature pair set; The source point cloud data and the target point cloud data are subjected to feature space projection processing and similarity maximization calculation to obtain similarity results, and multimodal comparative learning is performed on the source point cloud data and the target point cloud data according to the similarity results.

[0010] Optionally, in the multimodal three-dimensional point cloud registration method, the mixed features include source point cloud mixed features and target point cloud mixed features; The performing feature enhancement processing and feature fusion processing on the point cloud features and the point cloud image features to obtain mixed features specifically includes: An attention mechanism is used to perform feature interactive learning processing on the source point cloud features and the target point cloud features to obtain source point cloud enhanced features and target point cloud enhanced features; Inputting the source point cloud enhancement feature and the point cloud image feature into a multi-layer perceptron to obtain a first output dimension, and performing attention weight calculation and feature fusion processing according to the first output dimension to obtain a source point cloud mixed feature; The target point cloud enhancement features and the point cloud image features are input into the multi-layer perceptron to obtain a second output dimension, and attention weight calculation and feature fusion processing are performed according to the second output dimension to obtain target point cloud mixed features.

[0011] Optionally, the multimodal three-dimensional point cloud registration method, wherein the feature splicing processing and significance score calculation are performed according to the mixed features to obtain the significance score, and the feature selection processing is performed according to the significance score to obtain the key points and key point features, specifically includes: Performing maximum pooling processing on the source point cloud mixed features and the source point cloud data to obtain a global feature vector; Performing splicing processing on the global feature vector and the target point cloud mixed feature to obtain a splicing feature; The significance score corresponding to each feature in the splicing features is calculated through a one-dimensional convolutional network, and the key points and key point features are selected in order from high to low according to the significance scores to obtain a preset number of key points and key point features.

[0012] Optionally, the multimodal three-dimensional point cloud registration method, wherein the matching matrix construction processing and related weight calculation are performed according to the key points and the key point features to obtain target rigid transformation parameters, and the initial point cloud data are aligned according to the target rigid transformation parameters, specifically includes: Constructing a spatial coordinate combination and a mixed feature combination according to the key points and the key point features, and performing two-way parallel convolution processing on the spatial coordinate combination and the mixed feature combination to obtain a spatial coordinate matching matrix and a feature matching matrix; If the results of the spatial coordinate matching matrix and the feature matching matrix are inconsistent, performing concatenation, aggregation and convolution processing on the spatial coordinate matching matrix and the feature matching matrix to obtain a matching score; Calculating the relevant weight corresponding to the matching score, and solving the relevant weight by using the weighted singular value decomposition method to obtain the target rigid transformation parameter; The source point cloud data and the target point cloud data are aligned according to the target rigid transformation parameters.

[0013] In addition, to achieve the above-mentioned purpose, the present invention further provides a multimodal three-dimensional point cloud registration system, wherein the multimodal three-dimensional point cloud registration system comprises: A point cloud feature extraction module is used to obtain initial point cloud data, and perform point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features; A hybrid feature generation module, used for performing feature enhancement processing and feature fusion processing on the point cloud feature and the point cloud image feature to obtain a hybrid feature; A feature selection processing module is used to perform feature splicing processing and significance score calculation according to the mixed features to obtain the significance score, and perform feature selection processing according to the significance score to obtain key points and key point features; The target rigid transformation parameter generation module is used to perform matching matrix construction processing and related weight calculation according to the key points and the key point features to obtain target rigid transformation parameters, and align the initial point cloud data according to the target rigid transformation parameters.

[0014] In the present invention, initial point cloud data is obtained, and point cloud feature extraction and point cloud image feature extraction are performed on the initial point cloud data to obtain point cloud features and point cloud image features; feature enhancement and feature fusion are performed on the point cloud features and the point cloud image features to obtain mixed features; feature splicing and significance score calculation are performed based on the mixed features to obtain significance scores, and feature selection is performed based on the significance scores to obtain key points and key point features; matching matrix construction and related weight calculation are performed based on the key points and the key point features to obtain target rigid transformation parameters, and the initial point cloud data is aligned based on the target rigid transformation parameters. The present invention extracts features from the initial point cloud data to obtain point cloud features and point cloud image features, and constructs mixed features based on the point cloud features and the point cloud image features, and then determines key points and key point features based on the mixed features, and calculates the optimal rigid transformation parameters of the point cloud data based on the key points and the key point features, and finally the point cloud data can be aligned based on the optimal rigid transformation parameters, which significantly improves the accuracy and robustness of point cloud registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a preferred embodiment of the multi-modal three-dimensional point cloud registration method of the present invention; Figure 2 It is a schematic diagram of the overall structure implementation process of a preferred embodiment of the multi-modal three-dimensional point cloud registration method of the present invention; Figure 3 It is a structural schematic diagram of a correspondence relationship search module of a preferred embodiment of the multi-modal three-dimensional point cloud registration method of the present invention; Figure 4 It is a schematic diagram of the overall architecture of a preferred embodiment of the multi-modal three-dimensional point cloud registration method of the present invention; Figure 5 It is a structural diagram of a preferred embodiment of the multi-modal three-dimensional point cloud registration system of the present invention; Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] 3D point cloud registration is a key technology in the field of computer vision. Its core goal is to accurately align point cloud data from different positions and postures, so as to achieve data display, comparison and fusion in a unified coordinate system. With the rapid development of high-precision sensor technology, point cloud has gradually become the mainstream form of 3D data representation, and point cloud registration technology has been widely used in fields such as autonomous driving, augmented reality and 3D printing. However, point cloud data naturally has characteristics such as partiality and sparsity, which often leads to large accuracy deviations when estimating the rotation and translation of point clouds from different perspectives. Therefore, how to quickly and accurately achieve the registration of partially overlapping point clouds has always been an important technical problem that needs to be solved in this field.

[0018] With the breakthrough progress of deep learning in the field of computer vision, the three-dimensional point cloud registration method based on neural network has gradually become a research hotspot. The processing flow of deep point cloud registration mainly consists of three core links: feature extraction, matching search and registration optimization. Specifically, the neural network first extracts high-dimensional feature representations from the input source point cloud and target point cloud. These features need to have rotation invariance and local description capabilities; then the correspondence between point clouds is established based on the extracted features, and feature matching is usually achieved by using methods such as nearest neighbor search or attention mechanism; finally, the optimal rigid transformation parameters are obtained by optimizing the preset loss function (such as point-to-point distance, feature similarity, etc.), thereby completing the precise alignment of point clouds. Compared with traditional registration algorithms, deep learning methods can more comprehensively capture the geometric shape, spatial structure and semantic features of point clouds with their powerful feature learning capabilities, and have stronger environmental adaptability and robustness. This end-to-end learning framework not only simplifies the registration process, but also continuously improves model performance in a data-driven manner, providing a more reliable solution for practical applications.

[0019] To solve the above problems, the present invention proposes a multimodal 3D point cloud registration method, which achieves higher precision results by perceiving the global shape. At the same time, two contrastive learning strategies are introduced, where overlapping contrastive learning is used to emphasize overlapping point features, and cross-modal contrastive learning is used to construct the correspondence between 2D and 3D. In addition, a method for predicting point cloud masks is designed to extract key points and reduce computing resource consumption.

[0020] The multimodal three-dimensional point cloud registration method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the multimodal three-dimensional point cloud registration method includes the following steps: Step S10, obtaining initial point cloud data, and performing point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features. The initial point cloud data includes source point cloud data and target point cloud data; the point cloud features include source point cloud features and target point cloud features.

[0021] like Figure 2 As shown, the present invention proposes a three-dimensional point cloud registration method based on multimodal contrastive learning. The method mainly consists of four parts, namely feature extraction, multiple contrastive learning, attention fusion and mask prediction, and correspondence search.

[0022] Specifically, the source point cloud data and the target point cloud data are acquired, and point cloud feature extraction processing is performed on the source point cloud data and the target point cloud data to obtain the source point cloud features and the target point cloud features.

[0023] The specific process of feature extraction is as follows: the input is two sets of point clouds, including the source point cloud data (in, is the first source point cloud, For the Source point cloud, For the source point cloud) and target point cloud data (in, is the first target point cloud, For the target point cloud, For the The objective of the present invention is to calculate the optimal rigid transformation parameters, which include the rotation matrix and translation vectors , according to the optimal rigid transformation parameters to align the two sets of point clouds (alignment is done by finding the optimal rotation matrix and translation vectors , align the source point cloud to the target point cloud through rigid body transformation), where is the three-dimensional special orthogonal group, which describes the rotation transformation in three-dimensional space. is a three-dimensional real number space. It should be noted that the present invention does not require a strict one-to-one correspondence between the source point cloud and the target point cloud, so it can handle points with different numbers ( ,in, is the total number of source point clouds, is the total number of target point clouds).

[0024] The feature extraction process of the present invention is divided into two parallel branches: point cloud and image. In the point cloud branch, the point cloud is extracted. and Each point of is taken as the vertex of the graph, and the point-by-point features are extracted through the EdgeConv operation (EdgeConv extracts point-by-point features by dynamically constructing a local graph and aggregating edge features. The process includes: 1. Constructing a dynamic K nearest neighbor graph; 2. Calculating edge features; 3. Aggregating edge features), and using the KNN algorithm to dynamically construct a graph structure at each layer to expand the receptive field, and finally obtain point cloud features and The construction process of the graph structure is as follows: for each point, find the K nearest neighbor points with the Euclidean distance to form a K nearest neighbor graph, usually by calculating the distance matrix between points, and then select the points corresponding to the K smallest distances in each row as neighbor points.

[0025] The source point cloud data and the target point cloud data are subjected to multi-view projection processing to obtain a plurality of two-dimensional image sequences; a CNN network (Convolutional Neural Network) is used to extract features of the plurality of two-dimensional image sequences to obtain a plurality of projection features; an aggregation function is used to fuse the plurality of projection features to obtain the point cloud image features.

[0026] In the image branch, the 3D point cloud is projected onto Different perspectives are used to generate corresponding two-dimensional image sequences, and then image features are extracted. , image features The expression is: ;in For copy operation, is an aggregate function, A picture representing the projection, for One of the different perspectives.

[0027] Among them, image features The extraction process is as follows: By removing the 3D point cloud (including source point cloud and target point cloud) from VThe projected images are projected from different perspectives to obtain multiple two-dimensional images. The CNN network is then used to extract features from these projected images. Finally, the features of all perspectives are fused through the aggregation function to obtain the final image features. In essence, it is to convert 3D information into 2D features and then integrate them through multi-view projection + CNN.

[0028] Furthermore, overlapping point features and non-overlapping point features between the source point cloud data and the target point cloud data are obtained, a feature pair set is constructed based on the overlapping point features and the non-overlapping point features, and overlapping comparative learning is performed on the source point cloud data and the target point cloud data based on the feature pair set.

[0029] Multiple contrastive learning consists of two parts: overlapping contrastive learning and multimodal contrastive learning. Overlapping contrastive learning aims to enhance the feature expression of overlapping areas while suppressing the interference of non-overlapping areas. The specific implementation process is as follows: First, the source point cloud By the true value rigid transformation (referring to the rotation matrix and translation vectors ) and then based on the target point cloud The minimum distance threshold is used to determine the overlapping points (wherein the minimum distance is calculated as the Euclidean space distance of the three-dimensional coordinates. When the calculated Euclidean space distance is less than the preset threshold, the point cloud position at this time is determined to be an overlapping point); secondly, the overlapping point features and non-overlapping point features of the two groups of point clouds are extracted respectively through the overlapping selection mechanism (the extraction process is: the indexes of the overlapping points and non-overlapping points are determined by the threshold respectively, and then the selection is performed); then, a feature pair set is constructed (that is, the positive pair set + negative pair set 1 + negative pair set 2, which are used for overlapping contrast learning): the overlapping point feature pairs of the two groups of point clouds are defined as the positive pair set, and the point clouds are defined as Overlapping point features and point cloud The non-overlapping point feature pairs constitute negative pair set 1, and the point cloud Overlapping point features and point cloud The non-overlapping point feature pairs constitute the negative pair set 2.

[0030] The source point cloud data and the target point cloud data are subjected to feature space projection processing and similarity maximization calculation to obtain similarity results, and multimodal comparative learning is performed on the source point cloud data and the target point cloud data according to the similarity results.

[0031] It can be understood that the cross-modal contrastive learning module is responsible for constructing the corresponding mapping relationship between two-dimensional and three-dimensional features. The process includes: first, the point cloud features ( and ) and image features Projected to the unified feature space ( Represents the dimension of the point feature), and obtains the projection vector , and Then, by calculating and The average value of the point cloud modality is obtained by the unified projection representation In the invariant space, the goal is to maximize and similarity between them (for cross-modal contrastive learning), since they both correspond to the same object.

[0032] Step S20: Perform feature enhancement processing and feature fusion processing on the point cloud features and the point cloud image features to obtain mixed features, wherein the mixed features include source point cloud mixed features and target point cloud mixed features.

[0033] Specifically, an attention mechanism is used to perform feature interactive learning processing on the source point cloud features and the target point cloud features to obtain source point cloud enhanced features and target point cloud enhanced features; the source point cloud enhanced features and the point cloud image features are input into a multi-layer perceptron to obtain a first output dimension, and attention weight calculation and feature fusion processing are performed according to the first output dimension to obtain source point cloud mixed features; the target point cloud enhanced features and the point cloud image features are input into the multi-layer perceptron to obtain a second output dimension, and attention weight calculation and feature fusion processing are performed according to the second output dimension to obtain target point cloud mixed features.

[0034] The present invention adopts a two-layer attention fusion structure to enhance the context information extraction of features, where the first layer of attention mechanism focuses on point cloud features. and The information interaction between them generates enhanced point cloud features through feature interaction and ,This process automatically highlights the key area information in the point cloud.

[0035] Among them, the enhanced point cloud features are generated through feature interaction and The specific implementation process is: the original point cloud features and Through the attention mechanism, information exchange and complementarity between features are achieved to generate enhanced features. and In layman's terms, it is to let the two sets of features "pay attention" to each other's important information, learn and complement each other, and finally obtain a richer feature representation.

[0036] The second layer of attention mechanism aims to enhance the discriminability of point features by integrating global shape and texture information. Take the point cloud feature as an example. and image features Input MLP (Multi-Layer Perceptron) processing, The output of the query is an array ,in, is the output dimension of MLP, The output as key-value pairs (including and , K and V All belong to key-value pairs). By calculating the attention weight to capture global shape and texture information (where is the matrix transpose), and finally generates the source point cloud mixed features that integrate multimodal information : ; Similarly, the target point cloud mixed features can be obtained in the same way .

[0037] Step S30: performing feature concatenation processing and significance score calculation according to the mixed features to obtain significance scores, and performing feature selection processing according to the significance scores to obtain key points and key point features.

[0038] Specifically, the source point cloud mixed features and the source point cloud data are subjected to maximum pooling processing to obtain a global feature vector; the global feature vector and the target point cloud mixed features are spliced ​​to obtain a spliced ​​feature; the significance score corresponding to each feature in the spliced ​​feature is calculated through a one-dimensional convolutional network, and the significance scores are selected in descending order to obtain a preset number of key points and key point features.

[0039] The mask prediction module provided in the present invention optimizes feature expression by screening discriminative features and suppressing non-discriminative features. The specific process of mask prediction is as follows: first, the source point cloud mixed features And source point cloud data Perform the maximum pooling operation, copy the global feature vector obtained by pooling and mix it with the target point cloud feature splicing; then use a one-dimensional convolutional network to calculate the significance score for each feature. The higher the score, the stronger the discriminability of the feature, and the more conducive to subsequent matching; finally, select the top-K points based on the significance score, set their mask values ​​to 1, and the rest to 0, and thus filter out the key point coordinates of K key points and key point features , where the key point coordinates and key point features Used to guide the corresponding relationship search.

[0040] Step S40: performing matching matrix construction processing and relevant weight calculation according to the key points and the key point features to obtain target rigid transformation parameters, and aligning the initial point cloud data according to the target rigid transformation parameters.

[0041] The present invention sets up a corresponding relationship search module, which adopts a cascade framework of coarse registration and fine registration, and uses mixed features and spatial coordinates to perform independent corresponding point searches, such as Figure 3 By reasonably integrating the information of feature space and geometric space at different registration stages, the registration accuracy is effectively improved. This cascade registration strategy can not only make full use of information at different scales, but also gradually refine the registration results to avoid local optimal solutions.

[0042] Specifically, a spatial coordinate combination and a mixed feature combination are constructed according to the key points and the key point features, and two-way parallel convolution processing is performed on the spatial coordinate combination and the mixed feature combination to obtain a spatial coordinate matching matrix and a feature matching matrix; if the results of the spatial coordinate matching matrix and the feature matching matrix are inconsistent, the spatial coordinate matching matrix and the feature matching matrix are spliced, aggregated and convolved to obtain a matching score; the relevant weights corresponding to the matching scores are calculated, and the weighted singular value decomposition method is used to solve the relevant weights to obtain target rigid transformation parameters; the source point cloud data and the target point cloud data are aligned according to the target rigid transformation parameters.

[0043] According to the selected key point coordinates And its corresponding key point features , you can construct a spatial coordinate combination and hybrid feature combinations Among them, the spatial coordinate combination It contains the geometric position relationship between point pairs (i.e., between source point cloud and target point cloud), and the mixed feature combination The high-level semantic information between point pairs is encoded. The two combinations describe the similarities between point pairs from different perspectives and provide complementary information for subsequent matching.

[0044] like Figure 3 As shown ( Figure 3 K in is the number of key points, Represents the dimension of point features, M is the total number of target point clouds, is the initial rotation matrix, is the initial translation vector, is the rotation matrix after the nth iteration, is the translation vector after the nth iteration. The present invention processes the spatial coordinate combination respectively through two parallel convolution operations and hybrid feature combinations , compressing high-dimensional features into a one-dimensional matching matrix, including the spatial coordinate matching matrix and feature matching matrix . Among them, the spatial coordinate matching matrix Reflects the geometric similarity between point pairs, feature matching matrix It reflects the semantic similarity between point pairs. The larger the value of the matrix element, the higher the matching probability of the corresponding point pairs. This two-way feature extraction strategy can capture the local geometric structure and global semantic information of the point cloud at the same time.

[0045] The final matching matrix is ​​generated based on the spatial coordinate matching matrix and feature matching matrix When two matching matrices (i.e., spatial coordinate matching matrices and feature matching matrix ) are consistent, it can be regarded as a high-confidence match; when there is inconsistency, it is necessary to evaluate the degree of conflict to determine the matching credibility. Specifically, after splicing the two matching matrices, the convolution operation is used to map them to a high-dimensional feature space, and then the point is obtained through maximum aggregation and convolution operations. Match score Therefore, the The weight of a point pair is defined as: ;in, For the The weights of point pairs are defined as follows: represents the indicator function, Represents the matching score of all points The median is the median of all scores sorted. It is The matching score of the points is calculated by matching the spatial coordinates to the matrix and feature matching matrix After the relevant weights are obtained, the weighted singular value decomposition (SVD) is used to solve the final rigid transformation: ;in, is the rotation matrix; is the translation vector; To calculate the minimum and value; The source point cloud Point coordinates; According to the target point cloud The coordinates of the corresponding points found; the double vertical lines indicate the solution of the Euclidean distance.

[0046] In summary, the present invention introduces point cloud projection images as auxiliary information sources and designs a dual contrast learning strategy: overlapping contrast learning focuses on overlapping area feature extraction, and cross-modal contrast learning establishes a mapping relationship between two-dimensional and three-dimensional features. At the same time, the mask prediction module is innovatively introduced for key point recognition, and finally the rigid transformation is solved by corresponding relationship search. The overall architecture of the present invention is shown in the figure below. Figure 4 As shown ( Figure 4 In is the source point cloud feature, is the target point cloud feature, is the image feature, is the mixed feature of the source point cloud, is the mixed feature of the target point cloud, and SVD is the weighted singular value decomposition).

[0047] Beneficial effects of the present invention: The present invention proposes a three-dimensional point cloud registration method based on multimodal contrastive learning. The dual strategies of overlapping contrastive learning and cross-modal contrastive learning are introduced to enhance feature expression, a double-layer attention mechanism is combined to achieve efficient feature fusion, and a mask prediction module is designed to automatically identify key points. Finally, a cascaded registration framework is used to collaboratively optimize the feature space and geometric space information, which significantly improves the accuracy and robustness of point cloud registration and provides a more reliable solution for practical applications.

[0048] The three-dimensional point cloud registration method proposed in the present invention mainly includes four core innovations: 1. A feature learning strategy that combines feature extraction with multiple contrastive learning, in which overlapping contrastive learning enhances the discriminability of overlapping area features, and cross-modal contrastive learning realizes the fusion of two-dimensional and three-dimensional features; 2. A feature fusion method based on a double-layer attention mechanism enhances the expression of contextual information; 3. An innovative mask prediction module is used for key point recognition; 4. A cascaded correspondence search framework that combines feature space and geometric space information.

[0049] In addition, under the premise of keeping the overall framework unchanged, each module of the present invention can be replaced by other technical solutions: feature extraction can use deep learning architectures such as Transformer, contrastive learning can introduce other loss functions such as triplet loss, mask prediction can use traditional methods based on geometric features such as ISS (Intrinsic Shape Signatures), Harris3D (Harris 3D Feature Detector), etc., and correspondence search can also use traditional matching algorithms such as RAN and SAC as alternatives.

[0050] Furthermore, if Figure 5As shown, based on the above multimodal 3D point cloud registration method, the present invention also provides a multimodal 3D point cloud registration system, wherein the multimodal 3D point cloud registration system includes: The point cloud feature extraction module 51 is used to obtain initial point cloud data, and perform point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features; A mixed feature generation module 52, used for performing feature enhancement processing and feature fusion processing on the point cloud feature and the point cloud image feature to obtain a mixed feature; A feature selection processing module 53 is used to perform feature splicing processing and significance score calculation according to the mixed features to obtain the significance score, and perform feature selection processing according to the significance score to obtain key points and key point features; The target rigid transformation parameter generation module 54 is used to perform matching matrix construction processing and related weight calculation according to the key points and the key point features to obtain target rigid transformation parameters, and align the initial point cloud data according to the target rigid transformation parameters.

[0051] Furthermore, if Figure 6 As shown, based on the above-mentioned multimodal three-dimensional point cloud registration method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0052] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a multimodal three-dimensional point cloud registration program 40 is stored on the memory 20, and the multimodal three-dimensional point cloud registration program 40 can be executed by the processor 10, thereby realizing the multimodal three-dimensional point cloud registration method in the present application.

[0053] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multimodal three-dimensional point cloud registration method.

[0054] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0055] In one embodiment, when the processor 10 executes the multimodal three-dimensional point cloud registration program 40 in the memory 20 , the steps of the multimodal three-dimensional point cloud registration method are implemented.

[0056] In summary, the present invention provides a multimodal three-dimensional point cloud registration method, system and terminal, the method comprising: acquiring initial point cloud data, and performing point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features; performing feature enhancement processing and feature fusion processing on the point cloud features and the point cloud image features to obtain mixed features; performing feature splicing processing and significance score calculation based on the mixed features to obtain significance scores, and performing feature selection processing based on the significance scores to obtain key points and key point features; performing matching matrix construction processing and related weight calculation based on the key points and the key point features to obtain target rigid transformation parameters, and aligning the initial point cloud data according to the target rigid transformation parameters. The present invention obtains point cloud features and point cloud image features by performing feature extraction on initial point cloud data, and constructs hybrid features based on the point cloud features and point cloud image features, and then determines key points and key point features based on the hybrid features, and calculates the optimal rigid transformation parameters of the point cloud data based on the key points and key point features. Finally, the point cloud data can be aligned according to the optimal rigid transformation parameters, which significantly improves the accuracy and robustness of point cloud registration.

[0057] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0058] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0059] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal three-dimensional point cloud registration method, characterized in that: The multimodal three-dimensional point cloud registration method comprises: Acquire initial point cloud data, and perform point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features; Performing feature enhancement processing and feature fusion processing on the point cloud features and the point cloud image features to obtain mixed features; Performing feature concatenation processing and significance score calculation according to the mixed features to obtain a significance score, and performing feature selection processing according to the significance score to obtain key points and key point features; A matching matrix construction process and related weight calculation are performed according to the key points and the key point features to obtain target rigid transformation parameters, and the initial point cloud data are aligned according to the target rigid transformation parameters.

2. The multimodal three-dimensional point cloud registration method according to claim 1, characterized in that: The initial point cloud data includes source point cloud data and target point cloud data; the point cloud features include source point cloud features and target point cloud features; The step of acquiring initial point cloud data and performing point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features specifically includes: Acquire the source point cloud data and the target point cloud data, perform point cloud feature extraction processing on the source point cloud data and the target point cloud data, and obtain the source point cloud features and the target point cloud features; Point cloud image feature extraction processing is performed on the source point cloud data and the target point cloud data to obtain point cloud image features.

3. The multimodal three-dimensional point cloud registration method according to claim 2, characterized in that: The performing point cloud image feature extraction processing on the source point cloud data and the target point cloud data to obtain point cloud image features specifically includes: Performing multi-view projection processing on the source point cloud data and the target point cloud data to obtain multiple two-dimensional image sequences; Using a CNN network to extract features from the plurality of two-dimensional image sequences to obtain a plurality of projection features; An aggregation function is used to fuse the plurality of projection features to obtain the point cloud image features.

4. The multimodal three-dimensional point cloud registration method according to claim 2, characterized in that: The initial point cloud data is obtained, and point cloud feature extraction processing and point cloud image feature extraction processing are performed on the initial point cloud data to obtain point cloud features and point cloud image features, and then the following further comprises: Acquire overlapping point features and non-overlapping point features between the source point cloud data and the target point cloud data, construct a feature pair set according to the overlapping point features and the non-overlapping point features, and perform overlapping contrast learning on the source point cloud data and the target point cloud data according to the feature pair set; The source point cloud data and the target point cloud data are subjected to feature space projection processing and similarity maximization calculation to obtain similarity results, and multimodal comparative learning is performed on the source point cloud data and the target point cloud data according to the similarity results.

5. The multimodal three-dimensional point cloud registration method according to claim 2, characterized in that: The mixed features include source point cloud mixed features and target point cloud mixed features; The performing feature enhancement processing and feature fusion processing on the point cloud features and the point cloud image features to obtain mixed features specifically includes: An attention mechanism is used to perform feature interactive learning processing on the source point cloud features and the target point cloud features to obtain source point cloud enhanced features and target point cloud enhanced features; Inputting the source point cloud enhancement feature and the point cloud image feature into a multi-layer perceptron to obtain a first output dimension, and performing attention weight calculation and feature fusion processing according to the first output dimension to obtain a source point cloud mixed feature; The target point cloud enhancement features and the point cloud image features are input into the multi-layer perceptron to obtain a second output dimension, and attention weight calculation and feature fusion processing are performed according to the second output dimension to obtain target point cloud mixed features.

6. The multimodal three-dimensional point cloud registration method according to claim 5, characterized in that: The performing feature concatenation processing and significance score calculation according to the mixed features to obtain the significance score, and performing feature selection processing according to the significance score to obtain key points and key point features specifically includes: Performing maximum pooling processing on the source point cloud mixed features and the source point cloud data to obtain a global feature vector; Performing splicing processing on the global feature vector and the target point cloud mixed feature to obtain a splicing feature; The significance score corresponding to each feature in the splicing features is calculated through a one-dimensional convolutional network, and the key points and key point features are selected in order from high to low according to the significance scores to obtain a preset number of key points and key point features.

7. The multimodal three-dimensional point cloud registration method according to claim 2, characterized in that: The matching matrix construction process and related weight calculation are performed according to the key points and the key point features to obtain target rigid transformation parameters, and the initial point cloud data is aligned according to the target rigid transformation parameters, specifically including: Constructing a spatial coordinate combination and a mixed feature combination according to the key points and the key point features, and performing two-way parallel convolution processing on the spatial coordinate combination and the mixed feature combination to obtain a spatial coordinate matching matrix and a feature matching matrix; If the results of the spatial coordinate matching matrix and the feature matching matrix are inconsistent, performing concatenation, aggregation and convolution processing on the spatial coordinate matching matrix and the feature matching matrix to obtain a matching score; Calculating the relevant weight corresponding to the matching score, and solving the relevant weight by using the weighted singular value decomposition method to obtain the target rigid transformation parameter; The source point cloud data and the target point cloud data are aligned according to the target rigid transformation parameters.

8. A multimodal three-dimensional point cloud registration system, characterized in that: The multimodal three-dimensional point cloud registration system comprises: A point cloud feature extraction module is used to obtain initial point cloud data, and perform point cloud feature extraction processing and point cloud image feature extraction processing on the initial point cloud data to obtain point cloud features and point cloud image features; A hybrid feature generation module, used for performing feature enhancement processing and feature fusion processing on the point cloud feature and the point cloud image feature to obtain a hybrid feature; A feature selection processing module is used to perform feature splicing processing and significance score calculation according to the mixed features to obtain the significance score, and perform feature selection processing according to the significance score to obtain key points and key point features; The target rigid transformation parameter generation module is used to perform matching matrix construction processing and related weight calculation according to the key points and the key point features to obtain target rigid transformation parameters, and align the initial point cloud data according to the target rigid transformation parameters.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a multimodal three-dimensional point cloud registration program stored in the memory and executable on the processor. When the multimodal three-dimensional point cloud registration program is executed by the processor, the steps of the multimodal three-dimensional point cloud registration method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal three-dimensional point cloud registration program, and when the multimodal three-dimensional point cloud registration program is executed by a processor, the steps of the multimodal three-dimensional point cloud registration method according to any one of claims 1 to 7 are implemented.

Images