Multi-modal three-dimensional point cloud registration method and equipment for industrial detection

Abstract

The invention discloses a multi-modal three-dimensional point cloud registration method and equipment for industrial detection. The method comprises the following steps: acquiring first point cloud data containing a plurality of first feature points and second point cloud data containing a plurality of second feature points; acquiring a plurality of first images of different specific view angles of the first point cloud data, and acquiring a plurality of second images of different specific view angles of the second point cloud data; extracting 3D features of the first point cloud data and the second point cloud data; extracting 2D features of each first image and each second image; generating a multi-modal multi-view feature of each first feature point according to the 3D feature of the first point cloud data and the 2D feature of each first image, and similarly, generating a multi-modal multi-view feature of each second feature point; and determining a rigid transformation matrix based on the first feature point and the second feature point with the multi-modal and multi-view features, and performing registration on the first point cloud data and the second point cloud data by adopting the rigid transformation matrix. According to the invention, the registration precision can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a multimodal three-dimensional point cloud registration method and device for industrial inspection. Background Art

[0002] In the field of industrial inspection, high-precision three-dimensional point cloud registration is a core technology to ensure equipment quality and production safety. It is widely used in scenarios such as aircraft engine blade defect detection and automobile weld seam integrity analysis.

[0003] There are many existing methods for point cloud registration, which can be roughly divided into two categories: traditional methods and deep learning methods. For example, the iterative closest point (ICP) algorithm is one of the most famous traditional point cloud registration algorithms and has been widely used to solve registration problems in the early days. In recent years, due to the powerful learning capabilities of deep learning, many methods have used neural networks to extract deep features and then adopted the singular value decomposition algorithm to calculate the transformation matrix. Inspired by ICP, the deep closest point (DCP) algorithm proposes a point cloud feature extraction network to extract deep features to replace the manual features in ICP. To address the problem of single point cloud features, the UR&R algorithm proposes a differentiable renderer for the RGB-D point cloud registration framework. It uses a differentiable rendering technique to create projections of the transformed point cloud and calculate the photometric and depth losses of the registration target.

[0004] However, most existing point cloud registration methods are limited to extracting spatial information of the point cloud modality. The limited information of a single modality often leads to the loss of detailed information such as semantics and texture, which affects the registration accuracy and ultimately leads to problems such as missing synthetic scene details and texture fracture in the synthetic scene. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a multimodal three-dimensional point cloud registration method and device for industrial inspection.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] The present invention provides a multimodal three-dimensional point cloud registration method for industrial inspection, comprising:

[0008] Acquire first point cloud data and second point cloud data, wherein the first point cloud data includes a plurality of first feature points, and the second point cloud data includes a plurality of second feature points;

[0009] Acquire a plurality of first images of the first point cloud data at different specific perspectives, and acquire a plurality of second images of the second point cloud data at different specific perspectives;

[0010] extracting 3D features of the first point cloud data and the second point cloud data respectively;

[0011] Extracting 2D features of each first image and 2D features of each second image respectively;

[0012] generating a multimodal, multi-view feature for each first feature point based on the 3D features of the first point cloud data and the 2D features of each first image, and generating a multimodal, multi-view feature for each second feature point based on the 3D features of the second point cloud data and the 2D features of each second image;

[0013] Determining a rigid transformation matrix based on the first feature points and the second feature points having multimodal and multi-view features;

[0014] The first point cloud data and the second point cloud data are registered using the determined rigid transformation matrix.

[0015] The present invention also provides a multimodal three-dimensional point cloud registration device for industrial inspection, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0016] The memory is used to store computer programs;

[0017] The processor is used to implement the steps of the above-mentioned multimodal three-dimensional point cloud registration method for industrial inspection when executing the program stored in the memory.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention renders multi-view images with the help of camera internal and external parameters, and jointly learns 2D domain knowledge from multi-view images to enrich 3D information. Multi-view 2D images are obtained by rendering the point cloud, and joint learning is performed from the multi-view images to supplement the information that is occluded in other perspectives. Through this multi-view, multi-modal joint learning, the image information can be fully utilized, thereby alleviating the lack of 2D domain information such as color and texture in the point cloud, promoting the subsequent accurate estimation of the rigid transformation matrix, and ultimately improving the accuracy of RGB-D point cloud registration. The present invention can be applied to scenarios such as surface defect detection of industrial equipment, assembly error analysis of precision parts, and real-time quality monitoring of production lines.

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a multimodal 3D point cloud registration method for industrial inspection provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of a partial registration visualization sample of ScanNet provided by an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of visualization of registration of different methods provided by embodiments of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0025] The present invention designs a multimodal three-dimensional point cloud registration method for industrial inspection. The method aims to fully extract RGB-D data information, combine the semantic information of RGB images with the spatial information of 3D point clouds, and form a joint feature to solve the problems of model breakage and texture dislocation caused by insufficient point cloud registration accuracy in virtual indoor scene synthesis. The method achieves semantic consistency synthesis and establishes correct correspondence by fusing RGB semantic information with point cloud spatial features, thereby achieving registration. The algorithm framework proposed by this method is novel and has high accuracy. It is particularly effective in extracting multimodal consistency information and removing non-overlapping redundant information for efficient registration, especially for point cloud registration problems in RGB-D scenes. The present invention can process a pair of point clouds scanned from different angles by an RGB-D sensor, and can achieve accurate registration of RGB-D point clouds in large-scale scenes.

[0026] Figure 1 This is a flow chart of a multimodal 3D point cloud registration method for industrial inspection provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0027] S101. Acquire first point cloud data and second point cloud data, wherein the first point cloud data includes a plurality of first feature points, and the second point cloud data includes a plurality of second feature points.

[0028] Here, the first point cloud data and the second point cloud data are a pair of point cloud data to be registered, and each of the first point cloud data and the second point cloud data has an RGB-D image. For example, the first point cloud data is represented as a point cloud Among them, s represents the point cloud The number of feature points (i.e., the first feature points) in R s×3 Represents a tensor of dimension s×3; the second point cloud data is represented as a point cloud Among them, s represents the point cloud The number of feature points (i.e., the first feature point) in the point cloud, m represents the ... The number of feature points (ie, second feature points) in .

[0029] S102: Acquire a plurality of first images of the first point cloud data at different specific perspectives, and acquire a plurality of second images of the second point cloud data at different specific perspectives.

[0030] Exemplarily, the multiple different specific perspectives include: multiple or all of top, bottom, left, right, front, and back.

[0031] S103 , extracting 3D features of the first point cloud data and the second point cloud data respectively.

[0032] S104 , extracting 2D features of each first image and 2D features of each second image respectively.

[0033] S105. Generate a multimodal, multi-view feature for each first feature point based on the 3D features of the first point cloud data and the 2D features of each first image, and generate a multimodal, multi-view feature for each second feature point based on the 3D features of the second point cloud data and the 2D features of each second image.

[0034] S106 : Determine a rigid transformation matrix based on the first feature point and the second feature point having multi-modal and multi-view features.

[0035] S107 , registering the first point cloud data and the second point cloud data using the determined rigid transformation matrix.

[0036] In some embodiments, when the first point cloud data and the second point cloud data do not have multiple images with different specific perspectives, the above S102 can be implemented through steps S1021 to S1022:

[0037] S1021. Render the first point cloud data from a plurality of different specific perspectives respectively, and obtain corresponding first images from a plurality of different specific perspectives.

[0038] S1022: Render the second point cloud data from a plurality of different specific perspectives respectively, and obtain a plurality of second images from different specific perspectives.

[0039] For example, the first image or the second image of multiple different specific perspectives is represented as Among them, n represents n different specific perspectives, mv ranges from 1 to n, I mv Represents the first image or the second image of the mvth specific view angle, where H and W are the height and width of the image respectively.

[0040] In some embodiments, the above S103 can be specifically implemented as follows: using a 3D initial encoder to extract 3D features of the first point cloud data and the second point cloud data respectively. Exemplarily, the 3D initial encoder is a KP convolution block (proposed by the paper Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, Francois Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. IEEE International Conference on Computer Vision, 2019: 6411-6420). For example, the first point cloud data The 3D features can be expressed as Second point cloud data The 3D features can be expressed as and, C2 represents the dimension of the 3D feature of each feature point, E g (·) represents the 3D initial encoder.

[0041] In some embodiments, the above S104 can be specifically implemented as follows: using a 2D initial encoder to extract the 2D features of each first image and the 2D features of each second image. For example, the 2D initial encoder is a ResNet network. For example, the first image or the second image I of the mvth specific perspective mv The 2D feature representation is F mv , F mv =E v (I mv ),and, C1 represents the dimension of the 2D feature of each pixel, E v (·) represents the 2D initial encoder.

[0042] In some embodiments, the step of “generating a multimodal, multi-view feature of each first feature point based on the 3D feature of the first point cloud data and the 2D feature of each first image” in S105 may be specifically implemented as S1051 to S1055:

[0043] S1051, for the first feature point Select the first feature point from the 3D features of the first point cloud data 3D features The value of i ranges from 1 to s.

[0044] S1052, from the first point cloud data In the example, determine the first feature point k nearest neighbor points at each specific viewpoint among multiple different specific viewpoints.

[0045] Here, at each specific viewing angle, the Euclidean distance can be used to calculate the distance from the first point cloud data. Search k nearest neighbor points, among which the feature point With a feature point The Euclidean distance between them is calculated as: Where D(·,·) represents the calculation formula of Euclidean distance, and ‖·‖ represents the calculation of the second norm.

[0046] S1053. Calculate the pixel position coordinates of each neighboring point at each specific viewing angle using the camera's intrinsic parameter matrix and extrinsic parameter matrix.

[0047] Here, each neighboring point at each specific viewing angle has spatial coordinates. The pixel position coordinates of each neighboring point at each specific viewing angle are calculated based on the spatial coordinates of each neighboring point at each specific viewing angle using the camera's intrinsic and extrinsic matrix. Specifically, for each neighboring point, the formula for calculating the pixel position coordinates of the neighboring point based on the spatial coordinates of the neighboring point is: Among them, the K in ∈R 3×4 is the camera’s own internal parameter matrix, R ex ∈R 4×4 It is the camera's own external parameter matrix, (u,v) is the projected pixel position coordinate, and (x,y,z) is the spatial coordinate.

[0048] S1054. For each specific viewing angle, select the 2D features of each neighboring point at the specific viewing angle from the 2D features of the first image at the specific viewing angle according to the pixel position coordinates of each neighboring point at the specific viewing angle.

[0049] For example, for the mvth specific viewing angle, according to the pixel position coordinates of each neighboring point under the mvth specific viewing angle, the first image I of the mvth specific viewing angle is obtained. mv 2D features F mv In the example, we select the 2D features of each neighbor point under the mvth specific viewpoint and get 2D features of the k nearest neighbor points at the mvth specific viewpoint

[0050] S1055, the first feature point 3D features and the first feature point The 2D features of k neighboring points under multiple different specific perspectives are fused to obtain the first feature point Multimodal and multi-view features

[0051] Specifically, the maximum pooling function is used to pool the first feature point The 2D features P of the k nearest neighbors at each specific viewpoint mv Aggregate and obtain the aggregated 2D feature max(P mv ); Then, the first feature point The aggregated 2D features of multiple different specific perspectives are spliced ​​to obtain the spliced ​​multi-perspective feature cat mv＝1,2,…,n (max(P mv )) and the spliced ​​multi-view feature cat mv＝1,2,…,n (max(P mv )) is encoded to obtain the multi-view 2D feature P′, where P′=MLP(cat mv＝1,2,…,n (max(P mv ))), where MLP(·), cat(·) and max(·) represent multi-layer perceptron, concatenation operation and maximum pooling function respectively; then, the first feature point 3D features and multi-view 2D features P′ and then encoded, and the encoded features are combined with the first feature point through the residual structure 3D features Fusion is performed to obtain the first feature point Multimodal and multi-view features Right now Here, information fusion can be enhanced through the residual structure.

[0052] In some embodiments, the principle of "generating a multimodal multi-view feature of each second feature point according to the 3D feature of the second point cloud data and the 2D feature of each second image" in S105 is the same as that of S1051 to S1055 above, and the second feature point can be obtained by executing the same principle as S1051 to S1055 above. Multimodal and multi-view features Right now Among them, the value of i' ranges from 1 to m, Represents the second feature point 3D features, P′′ represents the 3D features generated Multimodal and multi-view features In the process, the multi-view 2D feature P′ is generated by executing the same principle as the above S1051 to S1055.

[0053] In some embodiments, the above S106 is implemented through steps S1061 to S1062:

[0054] S1061. Select a preset number of correspondences with the largest weights from the first feature points and the second feature points with multimodal and multi-view features, where each correspondence includes a first feature point and a second feature point that is most similar to the first feature point.

[0055] Specifically, for the first feature point From the second point cloud data Find the first feature point The first similarity to the second feature point A second feature point similar to the second Next, according to the first feature point Multimodal and multi-view features The second characteristic point Multimodal and multi-view features and the second feature point Multimodal and multi-view features Calculate the first feature point and the second feature point A corresponding relationship The weight w i ; After traversing the first point cloud data After the first feature point in the image is obtained, multiple correspondences and the weight of each correspondence are obtained; based on the weights, K correspondences with the largest weights are selected from the multiple correspondences. Here, K is a preset number, and the value of K can be set according to actual needs, for example, K = 400.

[0056] Specifically, the calculation is based on the first feature point and the second feature point A corresponding relationship The weight w i The method is: according to the first feature point Multimodal and multi-view features and the second feature point Multimodal and multi-view features Calculating Rockwell Ratio 1 and The difference between them, as the corresponding relationship The weight w i ,Right now For example, Among them, D(·,·) represents the calculation formula of Euclidean distance, Represents the second feature point Multimodal and multi-view features Represents the second feature point Multimodal and multi-view features

[0057] S1062: Determine a rigid transformation matrix using a RANSAC algorithm according to a preset number of corresponding relationships and a weight of each corresponding relationship.

[0058] Specifically, the selected preset number of correspondences and the weight of each correspondence are used as the input of the RANSAC algorithm. The RANSAC method estimates the rigid transformation matrix by minimizing the error E(C, T) between the corresponding feature points. The calculation formula of the error E(C, T) is: Where C=K, the value of o ranges from 1 to C. Indicates the oth corresponding relationship among C corresponding relationships The first feature point in Indicates the oth corresponding relationship The second characteristic point in o Indicates the oth corresponding relationship Here, the weights of the rigid transformation function are calculated using a weighted singular value decomposition algorithm to calculate the corresponding relationships and their weights, resulting in a rigid transformation matrix. The rigid transformation matrix consists of the rotation matrix R and the translation vector t.

[0059] The above S107 is specifically implemented as follows: after multiplying the rotation matrix R and the first point cloud data, it is added to the translation vector t to obtain the first point cloud data after registration. Specifically, The calculation formula is:

[0060] In some embodiments, the 3D features of the first point cloud data and the second point cloud data are extracted by a 3D initial encoder, the 2D features of each first image and the 2D features of each second image are extracted by a 2D initial encoder, the multimodal multi-view features of the first feature point and the multimodal multi-view features of the second feature point are generated by a cross-modal multi-view fusion module, the rigid transformation matrix is ​​determined by a rigid transformation matrix estimation module, the registration of the first point cloud data and the second point cloud data is performed by a registration module, and the 3D initial encoder, the 2D initial encoder, the cross-modal multi-view fusion module, the rigid transformation matrix estimation module and the registration module constitute a three-dimensional point cloud registration network. The three-dimensional point cloud registration network is trained using the following loss function:

[0061] L=L geo +L dep +λE(C,T);

[0062] Among them, each point cloud data to be registered corresponds to an RGB-D image, L geo Represents the rendering loss of RGB image, L dep Represents the rendering loss of the depth map, and λ is a preset coefficient, for example, λ = 0.1.

[0063] Specifically, L geo and L dep The calculation formulas are: in, Represents the point cloud data obtained by rigidly transforming one of the point cloud data to be registered. Express The RGB image obtained by rendering, Express The depth image obtained by rendering, Represents the point cloud data obtained by performing a rigid inverse transformation on another point cloud data in a pair of point cloud data to be registered. Express The RGB image obtained by rendering,

[0064] Express The depth image obtained by rendering, ‖·‖ represents the calculation of the bi-norm.

[0065] In response to the problem of insufficient utilization of image information in multimodal learning, the present invention proposes an RGB-D point cloud registration method based on multi-view and multi-modality. This method renders multi-view images with the help of camera internal and external parameters, and jointly learns 2D domain knowledge from multi-view images to enrich 3D information. Multi-view 2D images are obtained by rendering the point cloud, and information occluded in other perspectives is supplemented by joint learning from multi-view images. Through this multi-view and multi-modal joint learning, the image information can be fully utilized, thereby alleviating the lack of 2D domain information such as color and texture in the point cloud, promoting the subsequent establishment of correspondences and rigid transformation estimation, and realizing accurate registration of RGB-D point clouds. The method proposed in the present invention can effectively extract information from 2D and 3D modalities, and realize accurate registration of RGB-D point clouds based on the extracted information. This invention responds to the key technology research direction of "multimodal perception and intelligent quality inspection" in the "Industrial Internet Innovation and Development Action Plan (2021-2023)". By integrating the sensor's RGB-D multimodal information through an independent algorithm, it can achieve millimeter-level alignment accuracy and can help update and replace domestic industrial inspection equipment.

[0066] The present invention also provides a multimodal three-dimensional point cloud registration device for industrial inspection, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement the steps of the above-mentioned multimodal three-dimensional point cloud registration method for industrial inspection when executing the program stored in the memory.

[0067] The effectiveness of the present invention is verified through specific experiments below.

[0068] Experiment 1

[0069] ScanNet is a dataset of RGB-D scans of real indoor environments, containing 2.5 million RGB-D data points from 1513 scans covering 707 different spaces. The dataset includes RGB-D images, camera intrinsic parameters, and the actual camera pose. The official data is divided into 1594k scenes for training, 12.6k scenes for validation, and 26k scenes for testing. The experiment uses three evaluation metrics to evaluate RGB-D point cloud registration: rotation error (RE), translation error (TE), and Chamfer distance (CD). Based on these metrics, the experiment reports rotation accuracy (RA), translation accuracy (TA), and Chamfer accuracy (CA). We conducted registration experiments on the ScanNet dataset, and the results are shown in the following table, where bold indicates the best results.

[0070] Table 1. Registration results of RGB-D point cloud pairs (mm represents millimeters)

[0071] Model training set RA (%) RE(%) TA (%) TE (%) CA (%) CD(mm) ICP - 31.7 10.4 7.5 22.4 8.4 32.9 SIFT - 55.2 18.6 17.7 26.5 38.1 42.6 FCGF - 70.2 9.5 27.5 23.6 52.0 24.4 UR&R ScanNet 92.7 3.4 77.2 7.3 86.0 5.9 BYOC ScanNet 94.1 2.6 78.4 5.9 87.3 5.0 LLT ScanNet 95.5 2.5 80.4 5.5 88.9 4.6 PointMBF ScanNet 96.0 2.5 83.9 5.6 92.8 4.7 Ours ScanNet 96.8 2.4 84.4 5.5 93.3 4.6

[0072] As can be seen from Table 1, the present invention (i.e., "Ours" in Table 1) has achieved new optimal performance in almost all indicators. Compared with the latest state-of-the-art point cloud registration network PointMBF (Mingzhi Yuan, Kexue Fu, Zhihao Li, Yucong Meng, and Manning Wang. Pointmbf: A multi-scale bidirectional fusion network for unsupervised RGB-D point cloud registration. IEEE International Conference on Computer Vision, 2023: 17694-17705), the present invention has achieved significant improvement in almost all performance indicators. In addition, we also show some registration visualization samples of ScanNet, such as Figure 2 As shown. Figure 2As can be seen from the visualization results shown, the rendered X and Y are almost the same, and the black area represents the area in the camera field of view that is not covered by the point cloud. It can also be seen that the error of the transformation matrix predicted by the present invention is extremely small, which also shows that the alignment effect of the present invention is almost the same as the real alignment.

[0073] Experiment 2

[0074] The 3DMatch dataset is an RGBD reconstruction dataset that combines data from earlier sources, including 7Scenes, SUN3D, and RGB-D Scenes v.2. The official data set is divided into 122k scenes for training, 1.5k scenes for validation, and 1.5k scenes for testing. To evaluate generalization performance, our proposed method and the comparison method were trained on 3DMatch and evaluated on ScanNet. The experimental results are shown in the table below.

[0075] Table 2. Registration results of RGB-D point cloud pairs (mm represents millimeters)

[0076]

[0077]

[0078] As can be seen from Table 2, when trained using the 3DMatch dataset, the present invention (i.e., “Ours” in Table 2) performs best among all methods. It is worth noting that the performance of the present invention trained using the 3DMatch dataset even exceeds the performance of other algorithms trained on the ScanNet dataset. It can be seen that the present invention has the ability to learn across datasets because it can effectively transfer knowledge from the smaller 3DMatch dataset to the ScanNet dataset, thereby showing good performance on the larger dataset. In addition, we also show the registration visualization of different registration methods, such as Figure 3 As shown, from Figure 3 As shown in the registration visualization diagram, the present invention (i.e. Figure 3 “Ours” in ) surpasses the current best method PointMBF.

[0079] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0080] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0081] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0082] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A multimodal 3D point cloud registration method for industrial inspection, characterized by: include: Acquire first point cloud data and second point cloud data, wherein the first point cloud data includes a plurality of first feature points, and the second point cloud data includes a plurality of second feature points; Acquire a plurality of first images of the first point cloud data at different specific perspectives, and acquire a plurality of second images of the second point cloud data at different specific perspectives; extracting 3D features of the first point cloud data and the second point cloud data respectively; Extracting 2D features of each first image and 2D features of each second image respectively; generating a multimodal, multi-view feature for each first feature point based on the 3D features of the first point cloud data and the 2D features of each first image, and generating a multimodal, multi-view feature for each second feature point based on the 3D features of the second point cloud data and the 2D features of each second image; Determining a rigid transformation matrix based on the first feature points and the second feature points having multimodal and multi-view features; The first point cloud data and the second point cloud data are registered using the determined rigid transformation matrix.

2. The multimodal 3D point cloud registration method for industrial inspection according to claim 1, characterized in that: The determining of a rigid transformation matrix based on the first feature points and the second feature points having multimodal and multi-view features includes: Selecting a preset number of correspondences with the largest weights from the first feature points and the second feature points having multimodal and multi-view features, wherein each correspondence includes a first feature point and a second feature point that is most similar to the first feature point; According to the preset number of corresponding relationships and the weight of each corresponding relationship, a rigid transformation matrix is ​​determined using a RANSAC algorithm.

3. The multimodal 3D point cloud registration method for industrial inspection according to claim 2, characterized in that: The selecting a preset number of correspondences with the largest weights from the first feature points and the second feature points having multimodal and multi-view features includes: For each first feature point, find a second feature point that is first similar to the first feature point and a second feature point that is second similar to the first feature point from the second point cloud data; Calculating a weight of a correspondence between the first feature point and the second feature point that is first similar to the first feature point based on the multimodal multi-view feature of the first feature point, the multimodal multi-view feature of a second feature point that is first similar to the first feature point, and the second feature point that is second similar to the first feature point; After traversing all first feature points in the first point cloud data, obtaining a plurality of corresponding relationships and a weight of each corresponding relationship; According to the weights, a preset number of corresponding relationships with the largest weights are selected from the multiple corresponding relationships.

4. The multimodal 3D point cloud registration method for industrial inspection according to claim 3, characterized in that: The calculating, based on the multimodal multi-view features of the first feature point, the multimodal multi-view features of a second feature point that is first similar to the first feature point, and the multimodal multi-view features of a second feature point that is secondarily similar to the first feature point, a weight of a correspondence formed by the first feature point and the second feature point that is first similar to the first feature point includes: Calculating a Rockwell ratio based on the multimodal multi-view feature of the first feature point, a second feature point that is first similar to the first feature point, and a second feature point that is second similar to the first feature point; The difference between 1 and the Rockwell ratio is used as the weight of a corresponding relationship formed by the first feature point and a second feature point that is first similar to the first feature point.

5. The multimodal 3D point cloud registration method for industrial inspection according to claim 1, characterized in that: Generating a multimodal multi-view feature of each first feature point according to the 3D feature of the first point cloud data and the 2D feature of each first image includes: For each first feature point, select a 3D feature of the first feature point from the 3D features of the first point cloud data; Determining, from the first point cloud data, k neighboring points of the first feature point at each of the multiple different specific perspectives; Calculate the pixel position coordinates of each neighboring point at each specific viewing angle through the camera's intrinsic and extrinsic matrix; For each specific viewing angle, select the 2D features of each neighboring point at the specific viewing angle from the 2D features of the first image at the specific viewing angle according to the pixel position coordinates of each neighboring point at the specific viewing angle; The 3D feature of the first feature point and the 2D features of the k neighboring points of the first feature point under the multiple different specific perspectives are fused to obtain a multimodal multi-perspective feature of the first feature point.

6. The multimodal 3D point cloud registration method for industrial inspection according to claim 5, characterized in that: The fusing of the 3D feature of the first feature point with the 2D features of the k neighboring points of the first feature point under the multiple different specific perspectives to obtain the multimodal multi-perspective feature of the first feature point includes: The maximum pooling function is used to aggregate the 2D features of the k neighboring points of the first feature point at each specific perspective to obtain the aggregated 2D features of each specific perspective; splicing the aggregated 2D features of the first feature point at the multiple different specific perspectives to obtain a spliced ​​multi-perspective feature, and encoding the spliced ​​multi-perspective feature to obtain a multi-perspective 2D feature; The 3D feature of the first feature point and the multi-view 2D feature are spliced ​​and then encoded, and the encoded feature is fused with the 3D feature of the first feature point through a residual structure to obtain a multimodal multi-view feature of the first feature point.

7. The multimodal 3D point cloud registration method for industrial inspection according to claim 1, characterized in that: The acquiring of a plurality of first images of the first point cloud data at different specific perspectives, and the acquiring of a plurality of second images of the second point cloud data at different specific perspectives, comprises: Rendering the first point cloud data from a plurality of different specific perspectives respectively, and correspondingly obtaining a plurality of first images from different specific perspectives; The second point cloud data is rendered from a plurality of different specific perspectives respectively, and a plurality of second images from different specific perspectives are correspondingly obtained.

8. The multimodal 3D point cloud registration method for industrial inspection according to claim 1, characterized in that: The 3D features of the first point cloud data and the second point cloud data are extracted by a 3D initial encoder, the 2D features of each first image and the 2D features of each second image are extracted by a 2D initial encoder, the multimodal multi-view features of the first feature point and the multimodal multi-view features of the second feature point are generated by a cross-modal multi-view fusion module, the rigid transformation matrix is ​​determined by a rigid transformation matrix estimation module, the registration of the first point cloud data and the second point cloud data is performed by a registration module, and the 3D initial encoder, the 2D initial encoder, the cross-modal multi-view fusion module, the rigid transformation matrix estimation module and the registration module constitute a three-dimensional point cloud registration network.

9. The multimodal 3D point cloud registration method for industrial inspection according to claim 8, characterized in that: The 3D point cloud registration network is trained using the following loss function: L=L geo +L dep +λE(C,T); Among them, each point cloud data to be registered corresponds to an RGB-D image, L geo Represents the rendering loss of RGB image, L dep represents the rendering loss of the depth map, λ is a preset coefficient, and E(C,T) represents the error between corresponding feature points when estimating the rigid transformation matrix using the RANSAC method.

10. A multimodal 3D point cloud registration device for industrial inspection, comprising a processor, a communication interface, a memory, and a communication bus, characterized in that: The processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is configured to implement the method steps described in any one of claims 1 to 9 when executing a program stored in the memory.

Images