Non-rigid registration method, device, equipment and storage medium for point cloud processing

Abstract

The present application provides a non-rigid registration method, device, equipment and storage medium for point cloud processing, which relates to the field of three-dimensional reconstruction. The method includes: reconstructing the point cloud of the depth map through a pinhole camera model to generate a virtual data set; training the model through a preset training loss function to obtain a hierarchical matching fusion network; fine-tuning the hierarchical matching fusion network through a preset inference loss function; inputting the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to align the overlapping areas; performing rotational interpolation processing on the non-overlapping areas to align the non-overlapping areas. After determining the non-overlapping areas of the source point cloud and the target point cloud based on the overlapping areas, the present application performs rotational interpolation processing on the non-overlapping areas to align the non-overlapping areas; the overlapping area registration and the non-overlapping area registration correspond to different stages, avoiding the influence of the non-overlapping area registration on the overlapping area registration results, and improving the accuracy of point cloud registration.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a non-rigid registration method, device, equipment and storage medium for point cloud processing. Background Art

[0002] In the field of computer vision and 3D reconstruction, point cloud is an important form of data representation. Point cloud is constructed by collecting the spatial coordinates of each sampling point on the surface of an object. Through point cloud data, the 3D shape of the object can be accurately reconstructed to perform quantitative analysis and measurement of the object. Point cloud data can provide robots with accurate environmental perception capabilities to perform tasks. Through sensors such as lidar, robots can obtain dense point cloud data of the surrounding environment and then perform tasks such as path planning and target recognition. In addition, point cloud technology can also achieve autonomous navigation and obstacle recognition of vehicles through real-time perception of the vehicle's surrounding environment and accurate 3D reconstruction.

[0003] In the related technology, when performing three-dimensional reconstruction, in order to synthesize the multi-view point cloud data collected by scanning, it is necessary to reconstruct and align the point cloud data at each viewpoint; in the process of point cloud reconstruction and registration, rigid registration and non-rigid registration can be used; if only rigid transformation matrix is ​​used to align the point cloud data of different frames, the registration result will be poor; non-rigid registration technology can align the reconstructed model of the previous frame to the reconstructed model of the current frame, and can also bring the annotation information of the previous frame into the current coordinate system, thereby obtaining more comprehensive and accurate image information.

[0004] However, the current non-rigid registration method is only applicable to scenes with small non-rigid deformations. For the non-rigid registration algorithm of partially overlapping point clouds, it relies on the most rigid terms to constrain the registration of non-overlapping areas, which will cause the registration results of non-overlapping areas to affect the registration results of overlapping areas. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a non-rigid registration method, device, equipment and storage medium for point cloud processing, which solves the problem that the non-rigid registration algorithm relies on as rigid terms as possible to constrain the registration process of non-overlapping areas, resulting in inaccurate registration results.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a non-rigid registration method for point cloud processing, which includes: analyzing the collected original image based on a preset depth estimation algorithm to generate a depth map; reconstructing the depth map into a point cloud through a preset pinhole camera model to generate a virtual data set, the virtual data set including a source point cloud and a target point cloud; based on the virtual data set, training the model through a preset training loss function to obtain a hierarchical matching fusion network; fine-tuning the hierarchical matching fusion network through a preset inference loss function; inputting the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlapping matching and downsampling fusion, determining predicted overlapping points, overlapping areas and multiple first target soft matching points corresponding to the source point cloud to align the overlapping areas; determining the non-overlapping area between the source point cloud and the target point cloud based on the overlapping area, and performing rotation interpolation processing on the non-overlapping area to align the non-overlapping area.

[0008] According to a first aspect of an embodiment of the present application, the virtual data set also includes a non-rigid deformation corresponding to the target point cloud; the depth map is reconstructed by a preset pinhole camera model to generate a virtual data set, including: based on the preset pinhole camera model, the images on the same side of the depth map are converted into point cloud information, and the number of points in the point cloud information is reduced by voxel downsampling to obtain an initial point cloud; the initial point cloud is subjected to a random rigid transformation to generate an overlap probability of the initial point cloud; based on the overlap probability and the initial point cloud in the spatial coordinate system Axis coordinates, crop the initial point cloud and generate partially overlapping source point cloud and target point cloud; randomly sample a point in the central area of ​​the target point cloud as the center point; based on the center point, use The polynomial generates the target point cloud in Non-rigid deformation on the axis.

[0009] According to the first aspect of the embodiment of the present application, the source point cloud and the target point cloud are input into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlapping matching and downsampling fusion, and determine the predicted overlapping points, overlapping areas and multiple first target soft matching points corresponding to the source point cloud to align the overlapping areas, including: based on the farthest downsampling algorithm, sampling and processing the source point cloud and the target point cloud respectively to generate a first point cloud and a second point cloud; The algorithm extracts the features of the points in the first point cloud and the second point cloud based on different sampling levels to generate two sets of target sampling features; the two sets of target sampling features are analyzed by a similarity calculation method to determine the point-to-point similarity between the first point cloud and the second point cloud, and a similarity matrix is ​​constructed; The similarity matrix is ​​analyzed row by row and column by column in turn, the first probability and the second probability that each sampling point belongs to the overlapping area are calculated, and the first similarity probability and the second similarity probability corresponding to the first point cloud and the second point cloud are determined; matrix multiplication operation is performed on the first similarity probability, the second similarity probability and the similarity matrix to obtain a fused similarity matrix; the index weight is determined based on the data information corresponding to the row vector of the fused similarity matrix, the index weight and the point coordinates in the second point cloud are weighted, and the initial soft matching points corresponding to different downsampling levels are determined.

[0010] According to the first aspect of the embodiment of the present application, after determining the index weight based on the data information corresponding to the row vector of the fused similarity matrix, weighting the index weight and the point coordinates in the second point cloud, and determining the initial soft matching points corresponding to different downsampling levels, the non-rigid registration method for point cloud processing also includes: determining a baseline level from multiple downsampling levels, and determining a baseline soft matching point corresponding to the baseline level from multiple initial soft matching points; determining that the first similarity probability and the second similarity probability corresponding to the baseline soft matching point are the first baseline probability and the second baseline probability, respectively; based on a preset interpolation algorithm, the first similarity probability at different downsampling levels is determined. The point cloud is analyzed to determine multiple interpolation virtual matching points and difference matching probabilities corresponding to the interpolation virtual matching points; through the preset output splicing unit and the fully connected layer, the multiple interpolation virtual matching points and the benchmark soft matching points are fused in turn to obtain multiple first target soft matching points; based on the output splicing unit and the fully connected layer, the multiple difference matching probabilities are fused with the first benchmark probability and the second benchmark probability respectively to obtain multiple first target probabilities and multiple second target probabilities; the first target probability and the second target probability greater than the preset threshold are determined, and the corresponding points in the source point cloud and the target point cloud are predicted overlapping points, and the predicted overlapping points constitute the overlapping area.

[0011] According to the first aspect of the embodiment of the present application, a plurality of first target soft matching points constitute a predicted point cloud; the aforementioned rotation interpolation processing is performed on the non-overlapping areas to align the non-overlapping areas, including: estimating the rigid transformation matrix between the source point cloud and the target point cloud using a random sampling consensus algorithm based on the predicted overlapping points and their corresponding first target soft matching points; transforming the source point cloud using the rigid transformation matrix to obtain a rotated source point cloud; calculating the displacement vector field between the rotated source point cloud and the predicted point cloud, as well as the smoothed displacement vector field after smoothing; determining the displacement vector corresponding to the overlapping area from the smoothed displacement vector field; and based on the displacement vector, using a trilinear interpolation algorithm to calculate the displacement amount of the points in the non-overlapping area to determine the second target soft matching points corresponding to each point in the non-overlapping area to complete the non-rigid alignment between the source point cloud and the target point cloud.

[0012] According to a first aspect of an embodiment of the present application, the training loss function includes a true value loss function, an overlap prediction loss function and a smoothing loss function; the training loss function satisfies the expression:

[0013]

[0014] in, , and is a hyperparameter, is the true value loss function, and the true value generated by the virtual data set is used to supervise the prediction of soft matching points in the overlapping area. is the smoothing loss function, and is the overlapping prediction loss function, and both are for the overlapping area of ​​the source point cloud and the overlapping area of ​​the target point cloud respectively.

[0015] The inference loss function includes the chamfer distance loss function, the pseudo-truth loss function and the smoothing loss function. The inference loss function satisfies the expression:

[0016]

[0017] in, is the inference loss function, , and is a hyperparameter, and Negatively correlated with the number of iterations to reduce constraints on local neighborhood changes; is the chamfer distance loss function, is the pseudo-truth loss function.

[0018] According to the first aspect of the embodiment of the present application, the overlap prediction loss function and Both Classification loss function; truth loss function Satisfies the expression:

[0019]

[0020] in, is the number of points in the source point cloud, Yes The predicted soft matching points are Yes The true soft matching point of Yes The true value of the overlap probability is at point In the case of overlapping areas, is 1, at point When not in the overlapping area, is 0;

[0021] Smooth loss function Satisfies the expression:

[0022]

[0023] in, Yes The displacement vector satisfies: ; And indicates point The neighboring points of Indicate point The set of neighboring points of It is a neighbor point The displacement vector satisfies: ;

[0024] Chamfer distance loss function Satisfies the expression:

[0025]

[0026] in, is the target point cloud Middle distance point The nearest point, is the distance point in the predicted point cloud The nearest point, is the number of points in the target point cloud;

[0027] Pseudo-truth loss function Satisfies the expression:

[0028]

[0029] in, Yes Pseudo matching points, is the number of pseudo matching point pairs.

[0030] In a second aspect, an embodiment of the present application provides a non-rigid registration device for point cloud processing, which includes an acquisition and analysis module, a reconstruction module, a training module, a fine-tuning module and a registration module; the acquisition and analysis module is used to analyze the acquired original image based on a preset depth estimation algorithm to generate a depth map; the reconstruction module is used to perform point cloud reconstruction on the depth map through a preset pinhole camera model to generate a virtual data set, and the virtual data set includes a source point cloud and a target point cloud; the training module is used to perform model training based on the virtual data set through a preset training loss function to obtain a hierarchical matching fusion network; the fine-tuning module is used to fine-tune the hierarchical matching fusion network through a preset inference loss function; the registration module is used to input the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlapping matching and downsampling fusion, determine the predicted overlapping points, overlapping areas and multiple first target soft matching points corresponding to the source point cloud, so as to align the overlapping areas; the registration module is also used to determine the non-overlapping area between the source point cloud and the target point cloud according to the overlapping area, and perform rotation interpolation processing on the non-overlapping area to align the non-overlapping area.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the non-rigid registration method for point cloud processing in the aforementioned first aspect is implemented.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the non-rigid registration method for point cloud processing in the aforementioned first aspect is implemented.

[0033] The present invention provides a non-rigid registration method, device, equipment and storage medium for point cloud processing. Compared with the prior art, it has the following beneficial effects:

[0034] The present application converts an original image into a depth map through a depth estimation algorithm, reconstructs a point cloud based on the depth map to generate a virtual data set, and trains a hierarchical matching fusion network on the virtual data set. During use, the hierarchical matching fusion network is fine-tuned to improve the generalization of the method; the source point cloud and the target point cloud in the virtual data set are subjected to feature extraction, overlapping matching and downsampling fusion through the hierarchical matching fusion network to determine the predicted overlapping points, overlapping areas and a plurality of first target soft matching points corresponding to each point in the source point cloud of the target point cloud to realize the recognition and registration of the overlapping areas. The present application integrates the tasks of the recognition and registration of the overlapping areas into one network to realize dense matching point estimation; after determining the non-overlapping areas of the source point cloud and the target point cloud based on the overlapping areas, the non-overlapping areas are subjected to rotation interpolation processing to register the non-overlapping areas; the overlapping area registration and the non-overlapping area registration correspond to different stages to avoid the influence of the non-overlapping area registration on the overlapping area registration results, thereby improving the accuracy of the point cloud registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flowchart of a non-rigid registration method for point cloud processing provided in an embodiment of the present application;

[0037] Figure 2 yes Figure 1 An exemplary flow chart of S160;

[0038] Figure 3 It is a structural schematic diagram of a non-rigid registration device for point cloud processing provided in an embodiment of the present application;

[0039] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0042] The embodiments of the present application provide a non-rigid registration method, apparatus, device and storage medium for point cloud processing, thereby solving the problem that the non-rigid registration algorithm relies on as rigid terms as possible to constrain the registration process of non-overlapping areas, resulting in inaccurate registration results, and achieving high-precision registration of point clouds.

[0043] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0044] Point cloud is a data format that describes a three-dimensional model and is widely used in the fields of computer vision and three-dimensional reconstruction. Point cloud is composed of a set of discrete points in space, which represent the surface information of an object. Compared with traditional two-dimensional images, point cloud can provide more accurate three-dimensional geometric shape information. Point cloud is constructed by collecting the spatial coordinates of each sampling point on the surface of the object. The spatial coordinates of the point cloud correspond to the three directions in the three-dimensional space. In addition to the coordinate information, each point in the point cloud may also contain additional information such as color and intensity. Point cloud comes from a variety of sources, mainly including laser scanning, photogrammetry, and radar. Among them, through laser scanners, dense point cloud data on the surface of an object can be obtained, which can accurately represent the geometric shape and position information of the object.

[0045] Point cloud data can be used to accurately reconstruct the three-dimensional shape of an object to perform quantitative analysis and measurement of the object; point cloud data can provide robots with accurate environmental perception capabilities to perform tasks. Through sensors such as lidar, robots can obtain dense point cloud data of the surrounding environment and then perform tasks such as path planning and target recognition. In addition, point cloud technology can also achieve autonomous navigation and obstacle recognition of the vehicle through real-time perception and accurate three-dimensional reconstruction of the vehicle's surrounding environment.

[0046] In the related technology, when performing three-dimensional reconstruction, in order to synthesize the multi-view point cloud data collected by scanning, it is necessary to reconstruct and align the point cloud data under each view. In the process of point cloud reconstruction and registration, rigid registration and non-rigid registration can be used. If only the rigid transformation matrix is ​​used to align the point cloud data of different frames, the registration result will be poor. The non-rigid registration technology can align the reconstructed model of the previous frame to the reconstructed model of the current frame, and can also bring the annotation information of the previous frame into the current coordinate system, thereby obtaining more comprehensive and accurate image information. However, the current non-rigid registration method is only applicable to scenes with small non-rigid deformations. For the non-rigid registration algorithm of partially overlapping point clouds, it relies on as rigid terms as possible to constrain the registration of non-overlapping areas, which will cause the registration results of non-overlapping areas to affect the registration results of overlapping areas.

[0047] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The following first introduces a non-rigid registration method for point cloud processing provided in an embodiment of the present application.

[0049] A flowchart of a non-rigid registration method for point cloud processing provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the non-rigid registration method for point cloud processing may include the following steps S110 - S160 .

[0050] S110 , analyzing the collected original image based on a preset depth estimation algorithm to generate a depth map.

[0051] S120 , reconstructing the point cloud of the depth map by using a preset pinhole camera model to generate a virtual data set, where the virtual data set includes a source point cloud and a target point cloud.

[0052] S130, based on the virtual data set, the model is trained by a preset training loss function to obtain a hierarchical matching fusion network.

[0053] S140, fine-tuning the hierarchical matching fusion network through a preset inference loss function.

[0054] S150, input the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network, perform feature extraction, overlap matching and downsampling fusion, determine the predicted overlap points, overlap areas and multiple first target soft matching points corresponding to the source point cloud, so as to align the overlap areas.

[0055] S160 . Determine a non-overlapping area between the source point cloud and the target point cloud according to the overlapping area, and perform rotation interpolation processing on the non-overlapping area to align the non-overlapping area.

[0056] In an embodiment of the present application, in order to facilitate image analysis, the captured original image can be converted into a depth map based on a depth estimation algorithm, and point cloud reconstruction can be performed to obtain source point cloud and target point cloud. It can be understood that each pixel value of the depth map can represent the distance between a point in the scene and the camera.

[0057] In addition, in order to improve the generalization of the method, the present application fine-tunes the hierarchical matching fusion network through an inference loss function. After the fine-tuning, the overlapping areas are determined and registered in the first stage through a series of means such as feature extraction, overlapping matching and downsampling fusion; in the second stage of registering the non-overlapping areas, the determined non-overlapping areas are subjected to rotational interpolation processing; the present application aligns the overlapping areas and non-overlapping areas in two different stages to avoid interference of the non-overlapping area registration on the overlapping area registration results.

[0058] In an example, the aforementioned depth estimation algorithm may be a binocular depth estimation algorithm or a monocular depth estimation algorithm.

[0059] The above is a specific implementation method of the non-rigid registration method for point cloud processing provided in an embodiment of the present application. The present application trains a hierarchical matching fusion network on a virtual data set. During use, the hierarchical matching fusion network is fine-tuned to improve the generalization of the method. The present application integrates the tasks of overlapping area recognition and registration into one network to achieve dense matching point estimation. After determining the non-overlapping areas of the source point cloud and the target point cloud based on the overlapping areas, the non-overlapping areas are rotated and interpolated to align the non-overlapping areas. The overlapping area registration and the non-overlapping area registration correspond to different stages, which avoids the influence of the non-overlapping area registration on the overlapping area registration results, thereby improving the accuracy of point cloud registration.

[0060] In some embodiments, the virtual data set also includes a non-rigid deformation corresponding to the target point cloud; the aforementioned point cloud reconstruction of the depth map by a preset pinhole camera model to generate a virtual data set, that is, S120 may specifically include the following steps:

[0061] S210, based on a preset pinhole camera model, converting images on the same side of the depth map into point cloud information, and reducing the number of points in the point cloud information by voxel downsampling to obtain an initial point cloud.

[0062] S220 , performing a random rigid transformation on the initial point cloud to generate an overlapping probability of the initial point cloud.

[0063] S230, based on the overlap probability and the initial point cloud in the spatial coordinate system Axis coordinates, crop the initial point cloud and generate partially overlapping source point cloud and target point cloud.

[0064] S240: Randomly sample a point in the central area of ​​the target point cloud as the central point.

[0065] S250, based on the center point, using The polynomial generates the target point cloud in Non-rigid deformation on the axis.

[0066] In the embodiment of the present application, it can be understood that after obtaining the depth map, the pinhole camera model can be used to extract the point cloud information from the depth map; since the point cloud information is directly converted from the depth map, the data density in the point cloud information is relatively large and the features are relatively complex. In order to avoid information redundancy, the point cloud information is processed by voxel downsampling, which can reduce the density of data in the point cloud information, reduce the amount of data and computational complexity, and retain the key features of the point cloud information for subsequent processing and analysis. Furthermore, after the initial point cloud is subjected to a random rigid transformation to obtain the overlap probability, the initial point cloud can be cropped to generate partially overlapping source point clouds and target point clouds, and the non-rigid deformations in the scene can be quantified.

[0067] In some embodiments, the source point cloud and the target point cloud are input into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlap matching and downsampling fusion, and predict overlap points, overlap areas and a plurality of first target soft matching points corresponding to the source point cloud to register the overlap areas, that is, S150 may specifically include the following steps:

[0068] S310 , based on the farthest end downsampling algorithm, sampling the source point cloud and the target point cloud respectively to generate a first point cloud and a second point cloud.

[0069] S320, through The algorithm extracts features of points in the first point cloud and the second point cloud based on different sampling levels and generates two sets of target sampling features.

[0070] In the embodiment of the present application, it can be understood that since the features contained in the source point cloud and the target point cloud are relatively complex, in the process of processing and analyzing the source point cloud and the target point cloud, the farthest end downsampling algorithm and The algorithm performs feature extraction.

[0071] In one example, the first point cloud can be represented as: , the second point cloud can be expressed as: ,in, represents the downsampling level;

[0072] First point cloud midpoint The characteristics can be expressed as , the second point cloud midpoint The characteristics can be expressed as .

[0073] In another example, when extracting features of points in the first point cloud and the second point cloud, in addition to using Algorithm, other neural network point cloud feature extraction methods can also be used.

[0074] S330, analyzing the two groups of target sampling features by a similarity calculation method, determining the point-to-point similarity between the first point cloud and the second point cloud, and constructing a similarity matrix.

[0075] In one example, in the process of extracting the similarity between the first point cloud and the second point cloud, a similarity calculation method such as the reciprocal of the Euclidean distance can be used to calculate the point-to-point similarity between the first point cloud and the second point cloud based on the features of the points. , Indicate point With point The similarity value between the first point cloud and the second point cloud The similarity matrix between can be expressed as .

[0076] S340, through The similarity matrix is ​​analyzed row by row and column by column in turn, a first probability and a second probability that each sampling point belongs to the overlapping area are calculated, and a first similarity probability and a second similarity probability corresponding to the first point cloud and the second point cloud are determined.

[0077] In one example, by analyzing the similarity matrix row by row, the first probability that each sampling point belongs to the overlapping area can be determined. , the calculation formula is as follows:

[0078]

[0079] in, Represents the similarity matrix OK, The function is used to normalize the similarity matrix. OK, The operation selects the maximum value from each row of the similarity matrix Similarity values; The function is a multilayer perceptron and Combination of functions, The input of the function is Operation Selected Similarity values, The output of the function is a point The probability of being in the overlapping area .

[0080] Accordingly, the operation on each row of the similarity matrix is ​​converted into the operation on each column, and the sampling points are calculated. The second probability of belonging to the overlapping area , the calculation formula is:

[0081]

[0082] in, Represents the similarity matrix No. List.

[0083] In getting multiple first probabilities and the second probability After that, the first similarity probability and the second similarity probability corresponding to the first point cloud and the second point cloud can be determined. The first similarity probability can be expressed as , the second similarity probability can be expressed as: .

[0084] S350, performing matrix multiplication operation on the first similarity probability, the second similarity probability and the similarity matrix to obtain a fused similarity matrix.

[0085] In one example, the fused similarity matrix ,in, for The transposed matrix of .

[0086] S360, determining index weights based on data information corresponding to row vectors of the fused similarity matrix, weighting the index weights and point coordinates in the second point cloud, and determining initial soft matching points corresponding to different downsampling levels.

[0087] In one example, the first point cloud Points in The initial soft matching point The midpoint of the second point cloud The coordinates of are calculated, The calculation formula is as follows:

[0088]

[0089] in, Represents the fusion similarity matrix No. Row, function Output fusion similarity matrix The largest The index corresponding to the similarity value; soft matching point By weight The coordinates of the midpoint of the second point cloud Perform weighted summation to obtain; when the index In the function When the index is output, the index weight Equal to the value in the fused similarity matrix , when the index Not in function When the index is output, the index weight Set to 0; Indicates the corresponding First point cloud at downsampling level The coordinates of the initial soft matching points.

[0090] In some embodiments, after determining the index weight based on the data information corresponding to the row vector of the fused similarity matrix, weighting the index weight and the point coordinates in the second point cloud, and determining the initial soft matching points corresponding to different downsampling levels, that is, after S360, the non-rigid registration method for point cloud processing further includes:

[0091] S371. Determine a reference level from multiple downsampling levels, and determine a reference soft matching point corresponding to the reference level from multiple initial soft matching points.

[0092] S372: Determine that the first similarity probability and the second similarity probability corresponding to the reference soft matching point are respectively the first reference probability and the second reference probability.

[0093] S373. Based on a preset interpolation algorithm, analyze the first point cloud at different downsampling levels to determine a plurality of interpolation virtual matching points and difference matching probabilities corresponding to the interpolation virtual matching points.

[0094] S374. By using a preset output concatenation unit and a fully connected layer, a plurality of interpolated virtual matching points and reference soft matching points are sequentially merged to obtain a plurality of first target soft matching points.

[0095] S375. Based on the output concatenation unit and the fully connected layer, the plurality of difference matching probabilities are respectively fused with the first reference probability and the second reference probability to obtain a plurality of first target probabilities and a plurality of second target probabilities.

[0096] S376. Determine a first target probability and a second target probability that are greater than a preset threshold, and the corresponding points in the source point cloud and the target point cloud are predicted overlapping points, and the predicted overlapping points constitute an overlapping area.

[0097] In the embodiment of the present application, it can be understood that due to different downsampling levels, the downsampling levels corresponding to the multiple initial soft matching points are different. In the process of analyzing the overlapping area and the non-overlapping area between the source point cloud and the target point cloud, for data analysis, one level can be selected from the multiple downsampling levels as the reference level, and the corresponding reference soft matching point can be determined;

[0098] Accordingly, after determining the benchmark level, the first similarity probability and the second similarity probability corresponding to the benchmark soft matching point can be used as benchmark probabilities and participate in the next step of analysis. Specifically, the first similarity probability and the second similarity probability corresponding to the benchmark soft matching point are the first benchmark probability and the second benchmark probability, respectively.

[0099] Furthermore, after obtaining the interpolated virtual matching points and the difference matching probabilities, the interpolated virtual matching points can be fused and spliced ​​with the reference soft matching points, and then the spliced ​​matching points can be fused using the fully connected layer to obtain multiple first target soft matching points. Accordingly, from the perspective of probability, the predicted overlapping probabilities at different downsampling levels can be fused through the same operation; specifically, the difference matching probabilities corresponding to the interpolated virtual matching points can be fused and spliced ​​with the first reference probability and the second reference probability, respectively, to obtain the first target probability and the second target probability. In the process of reasoning, the points in the first target probability and the second target probability that are greater than the preset threshold are regarded as predicted overlapping points, and multiple predicted overlapping points constitute the overlapping area.

[0100] In one example, in order to merge and refine the results of different downsampling levels, they can be unified to the same sampling level through an interpolation algorithm; the interpolation algorithm can be defined as:

[0101]

[0102] in, Yes The interpolation virtual matching points, It corresponds to First point cloud at downsampling level point, It is First point cloud at downsampling level point, and also point of Nearest neighbor point; Yes The predicted displacement vector of is the weight used for interpolation, the denominator is used to normalize weights, weights It can be calculated by similarity measurement methods such as the inverse of the Euclidean distance.

[0103] In some embodiments, please refer to Figure 2 , the aforementioned multiple first target soft matching points constitute a predicted point cloud; the aforementioned rotation interpolation processing is performed on the non-overlapping area to align the non-overlapping area, which may specifically include the following steps:

[0104] S410 , estimating a rigid transformation matrix between a source point cloud and a target point cloud using a random sampling consensus algorithm according to the predicted overlapping points and their corresponding first target soft matching points.

[0105] S420, transform the source point cloud using a rigid transformation matrix to obtain a rotated source point cloud.

[0106] S430, calculating the displacement vector field between the rotation source point cloud and the prediction point cloud, and the smoothed displacement vector field.

[0107] S440: Determine a displacement vector corresponding to the overlapping area from the smoothed displacement vector field.

[0108] S450, based on the displacement vector, using a trilinear interpolation algorithm to calculate the displacement of the points in the non-overlapping area to determine the second target soft matching points corresponding to each point in the non-overlapping area to complete the non-rigid registration between the source point cloud and the target point cloud.

[0109] In the embodiment of the present application, it can be understood that when the hierarchical matching fusion network is used on the reconstructed point cloud, the hierarchical matching fusion network predicts the overlapping points of the source point cloud and the corresponding multiple first target soft matching points to obtain the registration result of the overlapping area. Further, based on the predicted overlapping points and the first target soft matching points, the second target soft matching points corresponding to the non-overlapping points are predicted, and finally the non-rigid registration is completed.

[0110] Based on this, the present application provides a two-stage registration framework to register overlapping areas and non-overlapping areas respectively; the process of determining non-overlapping points in non-overlapping areas needs to be based on the registration of overlapping areas, and the registration of non-overlapping areas occurs after the registration of overlapping areas, and the registration of non-overlapping areas will not affect the registration of overlapping areas.

[0111] In one example, the non-rigid registration method for point cloud processing provided in this application can be applied to minimally invasive surgery; minimally invasive surgery has the advantages of less trauma and faster recovery, and is being widely adopted in the field of surgery today; however, in minimally invasive surgery, doctors cannot directly observe tissue structures outside the current screen range, and the lack of intraoperative information will weaken the judgment and decision-making ability of the surgeon, affecting the quality of minimally invasive surgery. Non-rigid registration technology can register the intraoperative reconstruction model of the previous frame to the intraoperative reconstruction model of the current frame, thereby expanding the doctor's field of view and enabling the doctor to understand the situation of a larger range of surgical areas; at the same time, non-rigid registration technology can also bring the annotation information of the previous frame into the current coordinate system, allowing the doctor to obtain annotation information beyond the field of view of the endoscope. Therefore, introducing non-rigid registration technology into the field of minimally invasive surgery can provide doctors with more intraoperative information and assist doctors in making clinical decisions.

[0112] In one example, the training loss function includes a true value loss function, an overlap prediction loss function, and a smoothing loss function; the training loss function satisfies the expression:

[0113]

[0114] in, , and is a hyperparameter, is the true value loss function, and the true value generated by the virtual data set is used to supervise the prediction of soft matching points in the overlapping area. is the smoothing loss function, and is the overlapping prediction loss function, and both are for the overlapping area of ​​the source point cloud and the overlapping area of ​​the target point cloud respectively.

[0115] The inference loss function includes the chamfer distance loss function, the pseudo-truth loss function and the smoothing loss function. The inference loss function satisfies the expression:

[0116]

[0117] in, is the inference loss function, , and is a hyperparameter, and Negatively correlated with the number of iterations to reduce constraints on local neighborhood changes; is the chamfer distance loss function, is the pseudo-truth loss function.

[0118] Overlap prediction loss function and Both Classification loss function; truth loss function Satisfies the expression:

[0119]

[0120] in, is the number of points in the source point cloud, Yes The predicted soft matching points are Yes The true soft matching point of Yes The true value of the overlap probability is at point In the case of overlapping areas, is 1, at point When not in the overlapping area, is 0.

[0121] Smooth loss function Satisfies the expression:

[0122]

[0123] in, Yes The displacement vector satisfies: ; And indicates point The neighboring points of Indicate point The set of neighboring points of It is a neighbor point The displacement vector satisfies: .

[0124] Chamfer distance loss function Satisfies the expression:

[0125]

[0126] in, is the target point cloud Middle distance point The nearest point, is the distance point in the predicted point cloud The nearest point, is the number of points in the target point cloud.

[0127] Pseudo-truth loss function Satisfies the expression:

[0128]

[0129] in, Yes Pseudo matching points, is the number of pseudo matching point pairs.

[0130] In some embodiments, please refer to Figure 3 The present application also provides a non-rigid registration device 500 for point cloud processing, and the non-rigid registration device 500 may include the following modules:

[0131] The acquisition and analysis module 510 is used to analyze the acquired original image based on a preset depth estimation algorithm to generate a depth map.

[0132] The reconstruction module 520 is used to reconstruct the point cloud of the depth map through a preset pinhole camera model to generate a virtual data set, where the virtual data set includes a source point cloud and a target point cloud.

[0133] The training module 530 is used to perform model training based on the virtual data set through a preset training loss function to obtain a hierarchical matching fusion network.

[0134] The fine-tuning module 540 is used to fine-tune the hierarchical matching fusion network through a preset inference loss function.

[0135] The registration module 550 is used to input the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlapping matching and downsampling fusion, determine the predicted overlapping points, overlapping areas and multiple first target soft matching points corresponding to the source point cloud to align the overlapping areas.

[0136] The registration module 550 is further used to determine the non-overlapping area between the source point cloud and the target point cloud according to the overlapping area, and perform rotation interpolation processing on the non-overlapping area to register the non-overlapping area.

[0137] According to an embodiment of the present application, any multiple modules among the acquisition and analysis module 510, the reconstruction module 520, the training module 530, the fine-tuning module 540 and the registration module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0138] In some embodiments, the virtual data set also includes a non-rigid deformation corresponding to the target point cloud; the reconstruction module 520 may specifically include:

[0139] The conversion unit 521 is used to convert the images on the same side of the depth map into point cloud information based on a preset pinhole camera model, and reduce the number of points in the point cloud information by voxel downsampling to obtain an initial point cloud.

[0140] The rigid transformation unit 522 is used to perform a random rigid transformation on the initial point cloud to generate an overlapping probability of the initial point cloud.

[0141] The clipping generation unit 523 is used to clip the initial point cloud and generate partially overlapping source point cloud and target point cloud based on the overlap probability and the X-axis coordinate of the initial point cloud in the spatial coordinate system.

[0142] The random sampling unit 524 is used to randomly sample a point in the central area of ​​the target point cloud as the central point.

[0143] The deformation generating unit 525 is used to generate a deformation based on the center point using The polynomial generates a non-rigid deformation of the target point cloud on the Z axis.

[0144] In some embodiments, the registration module 550 may specifically include:

[0145] The sampling unit 551 is used to perform sampling processing on the source point cloud and the target point cloud respectively based on the farthest downsampling algorithm to generate a first point cloud and a second point cloud.

[0146] The feature generation unit 552 is used to generate The algorithm extracts features of points in the first point cloud and the second point cloud based on different sampling levels and generates two sets of target sampling features.

[0147] The similarity determination unit 553 is used to analyze the two groups of target sampling features by a similarity calculation method, determine the point-to-point similarity between the first point cloud and the second point cloud, and construct a similarity matrix.

[0148] Matrix analysis unit 554, used to The similarity matrix is ​​analyzed row by row and column by column in turn, a first probability and a second probability that each sampling point belongs to the overlapping area are calculated, and a first similarity probability and a second similarity probability corresponding to the first point cloud and the second point cloud are determined.

[0149] The matrix fusion unit 555 is used to perform matrix multiplication operation on the first similarity probability, the second similarity probability and the similarity matrix to obtain a fused similarity matrix.

[0150] The weighting unit 556 is used to determine the index weight based on the data information corresponding to the row vector of the fused similarity matrix, weight the index weight and the point coordinates in the second point cloud, and determine the initial soft matching points corresponding to different downsampling levels.

[0151] In some embodiments, the aforementioned multiple first target soft matching points constitute a predicted point cloud; the aforementioned registration module 550 may further include a non-rigid registration unit 557, and the non-rigid registration unit 557 may specifically include:

[0152] The estimation subunit 5571 is used to estimate the rigid transformation matrix between the source point cloud and the target point cloud using a random sampling consensus algorithm based on the predicted overlapping points and their corresponding first target soft matching points.

[0153] The transformation subunit 5572 is used to transform the source point cloud through a rigid transformation matrix to obtain a rotated source point cloud.

[0154] The calculation subunit 5573 is used to calculate the displacement vector field between the rotation source point cloud and the prediction point cloud, and the smoothed displacement vector field after smoothing.

[0155] The displacement vector determination subunit 5574 is used to determine the displacement vector corresponding to the overlapping area from the smooth displacement vector field.

[0156] The matching point determination subunit 5575 is used to calculate the displacement of the non-overlapping area points based on the displacement vector using a trilinear interpolation algorithm to determine the second target soft matching points corresponding to each point in the non-overlapping area to complete the non-rigid alignment between the source point cloud and the target point cloud.

[0157] In some embodiments, the non-rigid registration apparatus 500 may further include an overlapping region determination module 560, which may specifically include:

[0158] The first determining unit 561 is used to determine a reference level from a plurality of downsampling levels, and to determine a reference soft matching point corresponding to the reference level from a plurality of initial soft matching points.

[0159] The second determining unit 562 is configured to determine that the first similarity probability and the second similarity probability corresponding to the reference soft matching point are respectively the first reference probability and the second reference probability.

[0160] The interpolation unit 563 is used to analyze the first point cloud at different downsampling levels based on a preset interpolation algorithm, and determine a plurality of interpolation virtual matching points and difference matching probabilities corresponding to the interpolation virtual matching points.

[0161] A first fusion unit 564 is used to sequentially fuse multiple interpolation virtual matching points and reference soft matching points through a preset output concatenation unit and a fully connected layer to obtain multiple first target soft matching points;

[0162] The second fusion unit 565 is used to fuse multiple difference matching probabilities with the first reference probability and the second reference probability based on the output splicing unit and the fully connected layer, so as to obtain multiple first target probabilities and multiple second target probabilities.

[0163] The third determination unit 566 is used to determine the first target probability and the second target probability that are greater than a preset threshold value, and the corresponding points in the source point cloud and the target point cloud are predicted overlapping points, and the predicted overlapping points constitute an overlapping area.

[0164] Figure 3 Each module in the device shown has the function of implementing each step in the aforementioned non-rigid registration method for point cloud processing and can achieve its corresponding technical effect. For the sake of concise description, it will not be repeated here.

[0165] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 4 shown.

[0166] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0167] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0168] The memory 620 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 620 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 620 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 620 is a non-volatile solid-state memory.

[0169] The memory 620 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the non-rigid registration methods for point cloud processing in the above embodiments.

[0170] The processor 610 implements any one of the non-rigid registration methods for point cloud processing in the above embodiments by reading and executing computer program instructions stored in the memory 620 .

[0171] In one example, the electronic device may further include a communication interface 630 and a bus 600. Figure 4 As shown, the processor 610, the memory 620, and the communication interface 630 are connected via a bus 600 and communicate with each other.

[0172] The communication interface 630 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0173] Bus 600 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 600 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0174] In addition, in combination with the non-rigid registration method for point cloud processing in the above embodiments, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the non-rigid registration methods for point cloud processing in the above embodiments is implemented.

[0175] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0176] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0177] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0178] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0179] In summary, compared with the prior art, this application has the following beneficial effects:

[0180] 1. After determining the non-overlapping areas of the source point cloud and the target point cloud based on the overlapping areas, the present application performs rotational interpolation processing on the non-overlapping areas to align the non-overlapping areas; the overlapping area registration and the non-overlapping area registration correspond to different stages, which can reduce the impact of the non-overlapping area registration on the overlapping area registration results and improve the accuracy of point cloud registration.

[0181] 2. This application designs an inference loss function. During inference, an unsupervised inference loss function is used to fine-tune the hierarchical matching fusion network, thereby improving the generalization of the method.

[0182] 3. The hierarchical matching fusion network uses matrix multiplication operations to couple the prediction and registration of overlapping areas, integrating the tasks of point cloud overlapping area recognition and overlapping area registration into one neural network, and realizing dense matching point estimation.

[0183] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-rigid registration method for point cloud processing, characterized in that: include: Analyze the collected original image based on a preset depth estimation algorithm to generate a depth map; Reconstructing the depth map into a point cloud by using a preset pinhole camera model to generate a virtual data set, wherein the virtual data set includes a source point cloud and a target point cloud; Based on the virtual data set, model training is performed through a preset training loss function to obtain a hierarchical matching fusion network; Fine-tuning the hierarchical matching fusion network through a preset inference loss function; Input the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlap matching and downsampling fusion, determine predicted overlap points, overlap areas and a plurality of first target soft matching points corresponding to the source point cloud, so as to register the overlap areas; Determine a non-overlapping area between the source point cloud and the target point cloud according to the overlapping area, and perform rotation interpolation processing on the non-overlapping area to register the non-overlapping area; The step of inputting the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlap matching and downsampling fusion, and determining predicted overlap points, overlap areas and a plurality of first target soft matching points corresponding to the source point cloud to register the overlap areas includes: Based on the farthest downsampling algorithm, the source point cloud and the target point cloud are sampled and processed respectively to generate a first point cloud and a second point cloud; pass An algorithm is used to extract features of points in the first point cloud and the second point cloud based on different sampling levels to generate two sets of target sampling features; Analyzing the two groups of target sampling features by a similarity calculation method, determining the point-to-point similarity between the first point cloud and the second point cloud, and constructing a similarity matrix; pass Performing row-by-row analysis and column-by-column analysis on the similarity matrix in turn, calculating a first probability and a second probability that each sampling point belongs to the overlapping area, and determining a first similarity probability and a second similarity probability corresponding to the first point cloud and the second point cloud; Performing a matrix multiplication operation on the first similarity probability, the second similarity probability, and the similarity matrix to obtain a fused similarity matrix; An index weight is determined based on data information corresponding to a row vector of the fused similarity matrix, and the index weight and the point coordinates in the second point cloud are weighted to determine initial soft matching points corresponding to different downsampling levels.

2. The non-rigid registration method for point cloud processing according to claim 1, characterized in that: The virtual data set also includes a non-rigid deformation corresponding to the target point cloud; The point cloud reconstruction of the depth map is performed by using a preset pinhole camera model to generate a virtual data set, including: Based on a preset pinhole camera model, converting images on the same side of the depth map into point cloud information, and reducing the number of points in the point cloud information by voxel downsampling to obtain an initial point cloud; Performing a random rigid transformation on the initial point cloud to generate an overlapping probability of the initial point cloud; Based on the overlap probability and the initial point cloud in the spatial coordinate system Axis coordinates, cropping the initial point cloud and generating partially overlapping source point cloud and target point cloud; In the central area of ​​the target point cloud, a point is randomly sampled as the central point; Based on the center point, use The target point cloud is generated by the polynomial The non-rigid deformation on the axis.

3. The non-rigid registration method for point cloud processing according to claim 1, characterized in that: After determining the index weight based on the data information corresponding to the row vector of the fused similarity matrix, weighting the index weight and the point coordinates in the second point cloud, and determining the initial soft matching points corresponding to different downsampling levels, the non-rigid registration method for point cloud processing further includes: Determine a reference level from a plurality of downsampling levels, and determine a reference soft matching point corresponding to the reference level from a plurality of the initial soft matching points; Determine that the first similarity probability and the second similarity probability corresponding to the reference soft matching point are respectively a first reference probability and a second reference probability; Based on a preset interpolation algorithm, the first point cloud at different downsampling levels is analyzed to determine a plurality of interpolation virtual matching points and difference matching probabilities corresponding to the interpolation virtual matching points; By using a preset output concatenation unit and a fully connected layer, the plurality of interpolated virtual matching points and the reference soft matching points are sequentially merged to obtain a plurality of first target soft matching points; Based on the output concatenation unit and the fully connected layer, a plurality of the difference matching probabilities are respectively fused with the first reference probability and the second reference probability to obtain a plurality of first target probabilities and a plurality of second target probabilities; The first target probability and the second target probability that are greater than a preset threshold are determined, and corresponding points in the source point cloud and the target point cloud are predicted overlapping points, and the predicted overlapping points constitute an overlapping area.

4. The non-rigid registration method for point cloud processing according to claim 1 or 2, characterized in that: The plurality of first target soft matching points constitute a predicted point cloud; and the non-overlapping area is subjected to rotational interpolation processing to register the non-overlapping area, including: estimating a rigid transformation matrix between the source point cloud and the target point cloud using a random sampling consensus algorithm according to the predicted overlapping points and the first target soft matching points corresponding thereto; Transforming the source point cloud by the rigid transformation matrix to obtain a rotated source point cloud; Calculating a displacement vector field between the rotation source point cloud and the prediction point cloud, and a smoothed displacement vector field; determining a displacement vector corresponding to the overlapping area from the smoothed displacement vector field; Based on the displacement vector, a trilinear interpolation algorithm is used to calculate the displacement of points in the non-overlapping area to determine the second target soft matching points corresponding to each point in the non-overlapping area to complete the non-rigid registration between the source point cloud and the target point cloud.

5. The non-rigid registration method for point cloud processing according to claim 1 or 2, characterized in that: The training loss function includes a true value loss function, an overlap prediction loss function and a smoothing loss function; the training loss function satisfies the expression: in, , and is a hyperparameter, is the truth loss function, and the prediction of soft matching points in the overlapping area is supervised by the truth value generated by the virtual data set, is the smooth loss function, and is the overlap prediction loss function, and both are for the overlap area of ​​the source point cloud and the overlap area of ​​the target point cloud respectively; The inference loss function includes a chamfer distance loss function, a pseudo-truth loss function and the smoothing loss function, and the inference loss function satisfies the expression: in, is the inference loss function, , and is a hyperparameter, and Negatively correlated with the number of iterations to reduce constraints on local neighborhood changes; is the chamfer distance loss function, is the pseudo-truth loss function.

6. The non-rigid registration method for point cloud processing according to claim 5, characterized in that: The overlap prediction loss function and Both Classification loss function; the truth loss function Satisfies the expression: in, is the number of points in the source point cloud, Yes The predicted soft matching points are Yes The true soft matching point of Yes The true value of the overlap probability is at point In the case of overlapping areas, is 1, at point When not in the overlapping area, is 0; The smooth loss function Satisfies the expression: in, Yes The displacement vector satisfies: ; And indicates point The neighboring points of Indicate point The set of neighboring points of It is a neighbor point The displacement vector satisfies: ; The chamfer distance loss function Satisfies the expression: in, is the target point cloud Middle distance point The nearest point, is the distance point in the predicted point cloud The nearest point, is the number of points in the target point cloud; The pseudo-truth loss function Satisfies the expression: in, Yes Pseudo matching points, is the number of pseudo matching point pairs.

7. A non-rigid registration device for point cloud processing, characterized in that: include: The acquisition and analysis module is used to analyze the acquired original image based on a preset depth estimation algorithm to generate a depth map; A reconstruction module, used to perform point cloud reconstruction on the depth map through a preset pinhole camera model to generate a virtual data set, wherein the virtual data set includes a source point cloud and a target point cloud; A training module, used to perform model training based on the virtual data set through a preset training loss function to obtain a hierarchical matching fusion network; A fine-tuning module, used for fine-tuning the hierarchical matching fusion network through a preset inference loss function; A registration module is used to input the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlap matching and downsampling fusion, determine predicted overlap points, overlap areas and a plurality of first target soft matching points corresponding to the source point cloud, so as to register the overlap areas; The registration module is further used to determine a non-overlapping area between the source point cloud and the target point cloud according to the overlapping area, and perform rotation interpolation processing on the non-overlapping area to register the non-overlapping area; The step of inputting the source point cloud and the target point cloud into the fine-tuned hierarchical matching fusion network to perform feature extraction, overlap matching and downsampling fusion, and determining predicted overlap points, overlap areas and a plurality of first target soft matching points corresponding to the source point cloud to register the overlap areas includes: Based on the farthest downsampling algorithm, the source point cloud and the target point cloud are sampled and processed respectively to generate a first point cloud and a second point cloud; pass An algorithm is used to extract features of points in the first point cloud and the second point cloud based on different sampling levels to generate two sets of target sampling features; Analyzing the two groups of target sampling features by a similarity calculation method, determining the point-to-point similarity between the first point cloud and the second point cloud, and constructing a similarity matrix; pass Performing row-by-row analysis and column-by-column analysis on the similarity matrix in turn, calculating a first probability and a second probability that each sampling point belongs to the overlapping area, and determining a first similarity probability and a second similarity probability corresponding to the first point cloud and the second point cloud; Performing a matrix multiplication operation on the first similarity probability, the second similarity probability, and the similarity matrix to obtain a fused similarity matrix; An index weight is determined based on data information corresponding to a row vector of the fused similarity matrix, and the index weight and the point coordinates in the second point cloud are weighted to determine initial soft matching points corresponding to different downsampling levels.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the non-rigid registration method for point cloud processing as claimed in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the non-rigid registration method for point cloud processing according to any one of claims 1 to 6 is implemented.

Images