A liver point cloud registration method and device based on Transformer architecture

Abstract

The present invention discloses a liver point cloud registration method and device based on the Transformer architecture, which relates to the technical field of three-dimensional point cloud registration, and is used to solve the technical problem that the existing registration technology fails to comprehensively consider the mechanical properties of various liver tissues and the geometric properties of point clouds in the process of calculating rigid and non-rigid body transformations, ignores the differences in liver parenchyma and vascular system, and the interference of factors such as body posture, gravity, and pneumoperitoneum pressure, resulting in poor registration effect. The method includes obtaining a first point cloud and a second point cloud, pre-processing the first point cloud through a preset finite element analysis solver and preset deformation conditions to generate a third point cloud, and processing the first point cloud, the second point cloud, and the third point cloud in combination with a liver point cloud registration model based on the Transformer architecture to generate a liver point cloud registration result. The present invention adopts a point cloud Transformer architecture suitable for fusing different modal features, which can capture the potential matching relationship between each modality, thereby improving the registration effect.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud registration, and in particular to a liver point cloud registration method and device based on a Transformer architecture. Background Art

[0002] Liver surgery requires precise pre-operative planning due to the complexity of the organ and the high risk of bleeding. Therefore, three-dimensional visualization technology based on CT (Computed Tomography) / MRI (Magnetic Resonance Imaging) and laparoscopic augmented reality navigation have become research hotspots in recent years and have been initially applied in the field of hepatobiliary surgery. The precise positioning of the laparoscopic augmented reality navigation system is of great clinical significance for ensuring the complete resection of lesions, reducing the risk of accidental tissue injury and bleeding, and postoperative healing. At the same time, the navigation system integrates multimodal information into one view, which can reduce the cognitive burden of doctors, reduce their reliance on experience, and shorten the training time of doctors.

[0003] For surgical navigation systems, preoperative and intraoperative point cloud registration is one of the key issues. In clinical practice, preoperative liver surface point cloud data can be obtained through CT scan reconstruction, and intraoperative liver surface point cloud data can be obtained through various point cloud collectors. The above two types of data have different spatial coordinates and modal differences. The goal of point cloud registration technology is to achieve precise alignment of the two types of data, thereby providing accurate surgical navigation information.

[0004] In recent years, point cloud registration algorithms based on deep learning have shown better performance than traditional algorithms, further improving the registration effect. Among them, models based on the Transformer architecture have shown potential in modeling remote contextual dependencies of point cloud data. Due to the permutation equivariant operation and good global feature learning capabilities, the Transformer model architecture is suitable for point cloud processing and analysis, but the application of Transformer to liver point cloud registration also requires optimization for surgical scenarios.

[0005] Most existing liver point cloud registration technologies use learning overlapping masks to filter out non-overlapping areas to achieve liver point cloud registration. However, filtering non-overlapping areas will weaken the model's focus on global information, and because there is a significant difference in the coverage of preoperative and intraoperative point clouds, the preoperative point cloud reconstructed by CT scans can cover the entire liver surface, but the point cloud collected under laparoscopy during surgery only covers part of the liver surface, resulting in partial loss of perspective. In addition, the existing registration technology fails to comprehensively consider the mechanical properties of various liver tissues and the geometric properties of point clouds in the process of calculating rigid and non-rigid body transformations, ignoring the differences in liver parenchyma and vascular system, and the interference of factors such as body shape, gravity, and pneumoperitoneum pressure, resulting in poor registration results. Summary of the invention

[0006] The present invention provides a liver point cloud registration method and device based on Transformer architecture, which are used to solve the technical problem that the existing liver point cloud registration technology based on Transformer architecture leads to poor registration effect.

[0007] The first aspect of the present invention provides a liver point cloud registration method based on a Transformer architecture, comprising:

[0008] Acquire a first point cloud and a second point cloud, and preprocess the first point cloud based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud;

[0009] Inputting the first point cloud, the second point cloud and the third point cloud into a liver point cloud registration model based on a Transformer architecture, wherein the liver point cloud registration model based on a Transformer architecture comprises a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module;

[0010] Using the feature extraction module to perform feature extraction based on the first point cloud and the second point cloud to generate a first point cloud matrix and a second point cloud matrix;

[0011] Performing feature fusion matching according to the first point cloud matrix and the second point cloud matrix through the Transformer feature fusion matching module, and outputting a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask;

[0012] Taking the first point cloud, the third point cloud, the predicted mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of the transformation matrix estimation module, and outputting a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds;

[0013] The deformation field prediction module is used to perform registration according to the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result.

[0014] Optionally, preprocessing the first point cloud based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud includes:

[0015] Based on the preset deformation condition, the preset finite element analysis solver is used to perform deformation processing on the first point cloud to generate a third point cloud.

[0016] Optionally, the first point cloud matrix includes a first target matrix, a first restoration matrix and a first splicing matrix; the second point cloud matrix includes a second target matrix, a second restoration matrix and a second splicing matrix; the feature extraction module includes an encoder, a decoder and a geometric embedder; the feature extraction module is used to extract features according to the first point cloud and the second point cloud to generate the first point cloud matrix and the second point cloud matrix, including:

[0017] Using an encoder to downsample the first point cloud and the second point cloud and extract point-by-point visual features to generate a first target matrix and a second target matrix;

[0018] Performing feature restoration on the first target matrix and the second target matrix respectively through a decoder, and outputting a first restoration matrix and a second restoration matrix;

[0019] Using the first target coordinate matrix in the first target matrix and the second target coordinate matrix in the second target matrix as inputs of a geometric embedder, respectively, and outputting a first geometric embedding matrix and a second geometric embedding matrix;

[0020] splicing the first geometric embedding matrix and the first target high-level visual feature matrix in the first target matrix to generate a first splicing matrix;

[0021] The second geometric embedding matrix and the second target high-level visual feature matrix in the second target matrix are concatenated to generate a second concatenated matrix.

[0022] Optionally, the using the first target coordinate matrix in the first target matrix and the second target coordinate matrix in the second target matrix as inputs of a geometric embedder respectively, and outputting a first geometric embedding matrix and a second geometric embedding matrix, comprises:

[0023] Using a K nearest neighbor algorithm to collect neighboring points of a plurality of first matrix elements in the first target coordinate matrix and a plurality of second matrix elements in the second target coordinate matrix, respectively, to generate a first nearest neighbor point set corresponding to each of the first matrix elements and a second nearest neighbor point set corresponding to each of the second matrix elements;

[0024] Constructing distance geometric features for the multiple first neighboring points in each of the first nearest neighboring point sets and the multiple second neighboring points in each of the second nearest neighboring point sets, respectively, and outputting multiple positional relationships and multiple Euclidean distances corresponding to the first neighboring points in each of the first nearest neighboring point sets, and multiple positional relationships and multiple Euclidean distances corresponding to the second neighboring points in each of the second nearest neighboring point sets;

[0025] Performing matrix covariance calculation on each of the first nearest neighbor point sets and each of the second nearest neighbor point sets respectively, and outputting feature data corresponding to each of the first nearest neighbor point sets and feature data corresponding to each of the second nearest neighbor point sets;

[0026] Perform feature connection based on multiple positional relationships and multiple Euclidean distances corresponding to each first neighbor point in each of the first nearest neighbor point sets and feature data corresponding to each of the first nearest neighbor point sets, and determine connection features corresponding to each of the first neighbor points in each of the first nearest neighbor point sets;

[0027] Performing feature splicing on connection features corresponding to each first neighbor point in each of the first nearest neighbor point sets respectively, to generate first neighbor point splicing features corresponding to each of the first nearest neighbor point sets;

[0028] Inputting the first neighbor point concatenation features corresponding to each of the first nearest neighbor point sets into a multi-layer perceptron to generate first geometric embedding features corresponding to each of the first nearest neighbor point sets;

[0029] Using first geometric embedding features corresponding to each of the first nearest neighbor point sets to construct a first geometric embedding matrix;

[0030] Perform feature connection based on multiple positional relationships and multiple Euclidean distances corresponding to each second neighbor point in each second nearest neighbor point set and feature data corresponding to each second nearest neighbor point set, and determine connection features corresponding to each second neighbor point in each second nearest neighbor point set;

[0031] Performing feature splicing on connection features corresponding to each second neighbor point in each of the second nearest neighbor point sets respectively, to generate second neighbor point splicing features corresponding to each of the second nearest neighbor point sets;

[0032] Inputting the second neighbor point concatenation features corresponding to each of the second nearest neighbor point sets into a multi-layer perceptron to generate second geometric embedding features corresponding to each of the second nearest neighbor point sets;

[0033] A second geometric embedding matrix is ​​constructed using the second geometric embedding features corresponding to each of the second nearest neighbor point sets.

[0034] Optionally, the Transformer feature fusion matching module includes a mask predictor and a Transformer feature fuser; performing feature fusion matching according to the first point cloud matrix and the second point cloud matrix by the Transformer feature fusion matching module to output a first target fusion feature matrix, a second target fusion feature matrix and a predicted mask includes:

[0035] Using a Transformer feature fuser to generate a first target fused feature matrix and a second target fused feature matrix according to the first splicing matrix and the second splicing matrix;

[0036] A mask predictor is used to perform mask prediction based on a first target high-level visual feature matrix in the first target matrix, a second target high-level visual feature matrix in the second target matrix, and a first restored low-level point feature matrix in the first restored matrix to generate a predicted mask.

[0037] Optionally, the using a Transformer feature fuser to generate a first target fused feature matrix and a second target fused feature matrix according to the first splicing matrix and the second splicing matrix includes:

[0038] Using a self-attention mechanism to perform calculations based on the first splicing matrix and the second splicing matrix, respectively, to determine a first attention feature matrix and a second attention feature matrix;

[0039] Performing linear projection on the high-level visual feature matrix in the first stitching matrix and the high-level visual feature matrix in the second stitching matrix respectively to generate a first query matrix corresponding to the first stitching matrix and a second query matrix corresponding to the second stitching matrix;

[0040] Using a multi-layer perceptron to generate a first initial fused feature matrix according to the first query matrix, the first attention feature matrix, and the high-level visual feature matrix in the first concatenated matrix;

[0041] Using a multi-layer perceptron to generate a second initial fused feature matrix according to the second query matrix, the second attention feature matrix, and the high-level visual feature matrix in the second concatenated matrix;

[0042] Based on the cross attention mechanism, a first target fusion feature matrix and a second target fusion feature matrix are determined according to the first initial fusion feature matrix and the second initial fusion feature matrix.

[0043] Optionally, the mask predictor includes a plurality of convolutional layers and a Softmax activation function layer; the mask predictor performs mask prediction according to a first target high-level visual feature matrix in the first target matrix, a second target high-level visual feature matrix in the second target matrix, and a first restored low-level point feature matrix in the first restored matrix to generate a predicted mask, comprising:

[0044] performing maximum pooling on the first target high-level visual feature matrix in the first target matrix and the second target high-level visual feature matrix in the second target matrix respectively, and outputting a first global feature vector and a second global feature vector;

[0045] Splicing a plurality of rows of feature vectors in the first restored low-level point feature matrix in the first restored matrix with the first global feature vector and the second global feature vector to generate a plurality of matrix splicing vectors;

[0046] A multi-layer perceptron is used to fuse a plurality of matrix splicing vectors, and output a splicing vector fusion matrix;

[0047] Using multiple convolution layers to perform convolution operation on the splicing vector fusion matrix to determine the splicing vector convolution matrix;

[0048] The concatenated vector convolution matrix is ​​nonlinearly mapped through a Softmax activation function layer to generate a prediction mask.

[0049] Optionally, the step of taking the first point cloud, the third point cloud, the predicted mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of the transformation matrix estimation module, and outputting a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds, comprises:

[0050] Generate a confidence matrix and a Gaussian correlation matrix according to the first target fusion feature matrix, the second target fusion feature matrix, the first restored low-level point feature matrix in the first restored matrix, and the second restored low-level point feature matrix in the second restored matrix;

[0051] Based on a preset topk strategy, the prediction mask is used to screen the matrix elements in the confidence matrix and the matrix elements in the Gaussian correlation matrix to determine the point relationship between the first and second point clouds;

[0052] Based on the singular value decomposition method, a preset optimization objective function is optimized and solved by using the point relationship between the first and second point clouds to determine the optimal rigid transformation matrix;

[0053] Using the optimal rigid transformation matrix to transform the first point cloud and the second point cloud respectively, to generate a first transformed point cloud and a second transformed point cloud;

[0054] According to the point relationships between the first and second point clouds, a point correspondence relationship sequence between the first and second point clouds is constructed.

[0055] Optionally, the deformation field prediction module is used to perform registration according to the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result, including:

[0056] Input the first transformed point cloud, the third transformed point cloud and the second point cloud into a multi-layer perceptron respectively, and output a first increased dimension transformed point cloud matrix, a third increased dimension transformed point cloud matrix and a second increased dimension point cloud matrix;

[0057] Based on the point correspondence relationship sequence between the first and second point clouds, segmenting the matrix elements in the third dimension-enlarged transformed point cloud matrix to determine a plurality of unmatched relationship matrix elements;

[0058] Adding a plurality of the unmatched relationship matrix elements to the second increased dimension point cloud matrix to determine a second supplementary point cloud matrix;

[0059] Fusing the second supplementary point cloud matrix and the third dimension-enlarged transformed point cloud matrix, and outputting second and third fused point cloud matrices;

[0060] Using a multi-layer perceptron pair to generate a second and a third dimension enlarged feature matrix and a first dimension enlarged feature matrix according to the second and third fused point cloud matrices and the first dimension enlarged transformed point cloud matrix respectively;

[0061] Performing maximum pooling on the second and third dimension-enlarged feature matrices and the first dimension-enlarged feature matrix respectively to generate second and third target global features and the first target global feature;

[0062] Respectively splicing a plurality of rows of feature vectors in the first dimension-increased transformed point cloud matrix with the second and third target global features and the first target global feature to generate a plurality of matrix global splicing vectors;

[0063] fusing a plurality of matrix global splicing vectors to output a predicted deformation field;

[0064] generating a target point cloud matrix according to the predicted deformation field and the first transformed point cloud;

[0065] Transforming the second supplementary point cloud matrix to determine a second supplementary point cloud coordinate matrix;

[0066] Performing error calculation according to the target point cloud matrix and the second supplementary point cloud coordinate matrix to determine a target registration error, and judging whether the target registration error is less than or equal to a preset error threshold;

[0067] If yes, the target point cloud matrix and the second supplementary point cloud coordinate matrix are used as the liver point cloud registration result.

[0068] The second aspect of the present invention provides a liver point cloud registration device based on a Transformer architecture, comprising:

[0069] An acquisition module, used for acquiring a first point cloud and a second point cloud, and preprocessing the first point cloud based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud;

[0070] An input module, used to input the first point cloud, the second point cloud and the third point cloud into a liver point cloud registration model based on a Transformer architecture, wherein the liver point cloud registration model based on a Transformer architecture includes a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module;

[0071] A feature extraction module, configured to extract features based on the first point cloud and the second point cloud using the feature extraction module to generate a first point cloud matrix and a second point cloud matrix;

[0072] A feature fusion matching module, used to perform feature fusion matching according to the first point cloud matrix and the second point cloud matrix through the Transformer feature fusion matching module, and output a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask;

[0073] a transformation module, configured to use the first point cloud, the third point cloud, the prediction mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of the transformation matrix estimation module, and output a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds;

[0074] A registration module is used to use the deformation field prediction module to perform registration according to the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result.

[0075] It can be seen from the above technical solutions that the present invention has the following advantages:

[0076] The above technical scheme of the present invention provides a liver point cloud registration method based on the Transformer architecture. First, a first point cloud and a second point cloud are obtained, and the first point cloud is preprocessed based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud; then, the first point cloud, the second point cloud and the third point cloud are input into a liver point cloud registration model based on the Transformer architecture. The liver point cloud registration model based on the Transformer architecture includes a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module; the feature extraction module is used to perform feature extraction based on the first point cloud and the second point cloud to generate a first point cloud matrix and a second point cloud matrix; the Transformer feature fusion matching module is used to perform feature fusion matching based on the first point cloud matrix and the second point cloud matrix, and a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask are output; ... The three point clouds, the predicted mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix are used as the input of the transformation matrix estimation module, and the point correspondence relationship sequence between the first transformed point cloud, the third transformed point cloud and the first and second point clouds is output; finally, the deformation field prediction module is used to align the first transformed point cloud, the third transformed point cloud, the point correspondence relationship sequence between the first and second point clouds and the second point cloud to generate a liver point cloud registration result; based on the above scheme, the first point cloud and the second point cloud are obtained, the first point cloud is preprocessed by a preset finite element analysis solver and a preset deformation condition to generate a third point cloud, and the first point cloud, the second point cloud and the third point cloud are processed in combination with the liver point cloud registration model based on the Transformer architecture to generate a process of generating a liver point cloud registration result. The present invention adopts a point cloud Transformer architecture suitable for fusing different modal features, which can capture the potential matching relationship between each modality, thereby improving the registration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0078] Figure 1 A flowchart of the steps of a liver point cloud registration method based on a Transformer architecture provided in the first embodiment of the present invention;

[0079] Figure 2 An overall framework diagram for generating a first point cloud, a second point cloud, and a third point cloud provided in the first embodiment of the present invention;

[0080] Figure 3 A schematic diagram of the structure of a liver point cloud registration model based on a Transformer architecture provided in the first embodiment of the present invention;

[0081] Figure 4 A schematic diagram of the structure of a deformation field prediction module provided in the first embodiment of the present invention;

[0082] Figure 5 A schematic diagram of the structure of an application example of a liver point cloud registration model based on a Transformer architecture provided in the first embodiment of the present invention;

[0083] Figure 6 This is a structural block diagram of a liver point cloud registration device based on the Transformer architecture provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0084] The embodiment of the present invention provides a liver point cloud registration method and device based on the Transformer architecture, which is used to solve the technical problem that the existing liver point cloud registration technology based on the Transformer architecture leads to poor registration effect.

[0085] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] See also Figure 1 , Figure 1 A flowchart of the steps of a liver point cloud registration method based on a Transformer architecture provided in Example 1 of the present invention.

[0087] The present invention provides a liver point cloud registration method based on Transformer architecture, comprising:

[0088] Step 101: Acquire a first point cloud and a second point cloud, and pre-process the first point cloud based on a preset finite element analysis solver and preset deformation conditions to generate a third point cloud.

[0089] It should be noted that in clinical applications, the preoperative image reconstruction point cloud is obtained as the first point cloud (first clinical point cloud), the second point cloud (second clinical point cloud) is obtained under laparoscopic conditions through the point cloud collector, and the first clinical point cloud is preprocessed by the preset finite element analysis solver and the preset deformation conditions to obtain the third clinical point cloud (third point cloud); wherein, the preset finite element analysis solver can perform deformation simulation analysis on the liver contour and internal and external tissues separately based on reasonable assumptions about the biomechanical properties of each tissue, thereby ensuring the accurate positioning of the liver's internal tissues while registering the liver surface. The preset finite element analysis solver participates in the generation and enhancement of the original training data on the one hand, and provides supplementary data for the second point cloud that lacks the perspective of the surgical process on the other hand, and depends on the doctor's experience for selection.

[0090] Furthermore, the finite element simulation module (pre-set finite element analysis solver) performs calculations based on gridded data, so the module also provides the function of mutual conversion between point cloud data and gridded data to ensure that the number of points remains unchanged before and after deformation, and the point matching relationship remains unchanged, so as to obtain the deformation field for model learning.

[0091] Specifically, based on the preset deformation condition, the first point cloud is deformed by using the preset finite element analysis solver to generate the third point cloud; wherein the preset deformation condition is to apply a deformation field u to the first point cloud under the biomechanical constraint. FEM The deformed point cloud is obtained and saved as the third point cloud P.

[0092] It is worth mentioning that in the model training stage, since it is difficult to obtain high-quality training data in the clinical medical field, the present invention generates a synthetic data set based on 50 real liver samples for model learning. The 50 real liver samples are subjected to different degrees of deformation by a preset finite element analysis solver, and 2000 original CT point cloud samples are simulated and saved in the database; the preprocessing process of the real liver samples in the model training stage is specifically as follows:

[0093] A preset finite element analysis solver is used to generate an original CT point cloud sample based on a real liver sample; a preset transformation matrix is ​​used to perform a rigid body transformation on the original CT point cloud sample to output a first point cloud; based on a first deformation condition, the original CT point cloud sample is deformed to generate a second point cloud; based on a second deformation condition, a preset finite element analysis solver is used to deform the first point cloud to generate a third point cloud.

[0094] Please note that Figure 2First, the acquired real liver sample is deformed by the preset finite element analysis solver to simulate the original CT point cloud sample. Then, the rigid body transformation is applied to the original CT point cloud sample by the randomly generated transformation matrix {R, t} (preset transformation matrix) and saved as the first point cloud S, where R is the rotation matrix and t is the translation vector. Based on the first deformation condition, the original CT point cloud sample is deformed to generate the second point cloud, where the first deformation condition is to apply the deformation field u to the original CT point cloud sample under the biomechanical constraint. tar The deformed point cloud is obtained, and the point cloud data sampled under the simulated view loss is referenced to the real laparoscopic field of view during surgery and saved as the second point cloud T; ​​finally, based on the second deformation condition, the preset finite element analysis solver is used to deform the first point cloud to generate the third point cloud, where the second deformation condition is to apply a deformation field u to the first point cloud under biomechanical constraints. FEM The deformed point cloud is obtained and saved as the third point cloud P.

[0095] Furthermore, in the practical application of the present invention, the deformation field u tar and u FEM The difference is that u tar It is mainly determined by physical factors such as pneumoperitoneum pressure, body posture, and breathing. FEM It is mainly determined by the doctor's experience and judgment.

[0096] In this embodiment, the synthetic data of the present invention following specific biomechanical laws and doctors' clinical experience can not only make the model more sensitive to the alignment relationship before and after deformation, but also promote the model to make deformation field predictions that are more in line with physical reality.

[0097] Step 102: input the first point cloud, the second point cloud and the third point cloud into a liver point cloud registration model based on a Transformer architecture, wherein the liver point cloud registration model based on a Transformer architecture includes a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module.

[0098] Please note that Figure 3-Figure 4 The liver point cloud registration model based on Transformer architecture proposed in the present invention is composed of a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module. The feature extraction module includes a mask predictor and a Transformer feature fuser; the Transformer feature fusion matching module includes a mask predictor and a Transformer feature fuser; the mask predictor includes multiple convolutional layers and a Softmax activation function layer.

[0099] Step 103: Use a feature extraction module to perform feature extraction based on the first point cloud and the second point cloud to generate a first point cloud matrix and a second point cloud matrix.

[0100] The first point cloud matrix includes a first target matrix, a first restoration matrix and a first splicing matrix.

[0101] The second point cloud matrix includes a second target matrix, a second restoration matrix and a second stitching matrix.

[0102] The feature extraction module includes an encoder, a decoder, and a geometric embedder.

[0103] It should be noted that the feature extraction module proposed in the present invention includes an encoder, a decoder, and a geometric embedder. At the same time, both the encoder and the decoder use KPConv (Kernel Point Convolution) as the backbone network for feature extraction.

[0104] Further, step 102 may include the following sub-steps S21-S25:

[0105] Step S21, using an encoder to downsample the first point cloud and the second point cloud and extract point-by-point visual features to generate a first target matrix and a second target matrix;

[0106] Step S22, performing feature restoration on the first target matrix and the second target matrix respectively through a decoder, and outputting a first restoration matrix and a second restoration matrix;

[0107] It should be noted that the encoder is responsible for downsampling the first point cloud S and the second point cloud T respectively and extracting visual features point by point, so as to obtain the coordinate matrices of the two point cloud subsets, that is, the first target coordinate matrix in the first target matrix , the second target coordinate matrix in the second target matrix , and the corresponding high-level visual feature matrix, that is, the first target high-level visual feature matrix in the first target matrix , the second target high-level visual feature matrix in the second target matrix .

[0108] Furthermore, the decoder is responsible for converting the feature matrix extracted in the previous downsampling stage, i.e., the first target matrix , the second target matrix Learn and restore to obtain the restored feature matrix, that is, the first restored matrix , the second reduction matrix , is the first reduced coordinate matrix in the first reduced matrix, is the first reduced low-level point feature matrix in the first reduced matrix, is the second reduced coordinate matrix in the second reduced matrix, is the second restored low-level point feature matrix in the second restored matrix. The present invention uses skip connections to enable the features restored in the upsampling stage to fuse more low-level information, ensuring that the result is more refined.

[0109] Step S23, using the first target coordinate matrix in the first target matrix and the second target coordinate matrix in the second target matrix as inputs of the geometric embedder respectively, and outputting the first geometric embedding matrix and the second geometric embedding matrix;

[0110] It should be noted that the point cloud input to the geometry embedder is downsampled to obtain a point cloud subset, namely the first target coordinate matrix in the first target matrix , the second target coordinate matrix in the second target matrix ,The geometric embedder is responsible for computing the geometric embedding of a subset of the point cloud to capture the ,geometric structure inside the point cloud and the geometric consistency between point clouds.

[0111] Further, step S23 may include the following sub-steps S231-S2311:

[0112] Step S231, using the K nearest neighbor algorithm to collect neighboring points of multiple first matrix elements in the first target coordinate matrix and multiple second matrix elements in the second target coordinate matrix, to generate a first nearest neighbor point set corresponding to each first matrix element and a second nearest neighbor point set corresponding to each second matrix element;

[0113] Step S232, respectively constructing distance geometric features for multiple first neighboring points in each first nearest neighboring point set and multiple second neighboring points in each second nearest neighboring point set, and outputting multiple positional relationships and multiple Euclidean distances corresponding to each first neighboring point in each first nearest neighboring point set, and multiple positional relationships and multiple Euclidean distances corresponding to each second neighboring point in each second nearest neighboring point set;

[0114] Step S233, performing matrix covariance calculation on each first nearest neighbor point set and each second nearest neighbor point set, and outputting feature data corresponding to each first nearest neighbor point set and feature data corresponding to each second nearest neighbor point set;

[0115] Step S234, performing feature connection according to multiple positional relationships and multiple Euclidean distances corresponding to each first neighbor point in each first nearest neighbor point set and feature data corresponding to each first nearest neighbor point set, to determine connection features corresponding to each first neighbor point in each first nearest neighbor point set;

[0116] Step S235, performing feature splicing on connection features corresponding to each first neighbor point in each first nearest neighbor point set, to generate first neighbor point splicing features corresponding to each first nearest neighbor point set;

[0117] Step S236, inputting the first neighbor point concatenation features corresponding to each first nearest neighbor point set into a multi-layer perceptron to generate first geometric embedding features corresponding to each first nearest neighbor point set;

[0118] Step S237, constructing a first geometric embedding matrix using the first geometric embedding features corresponding to each first nearest neighbor point set;

[0119] Step S238, performing feature connection according to multiple positional relationships and multiple Euclidean distances corresponding to each second neighbor point in each second nearest neighbor point set and feature data corresponding to each second nearest neighbor point set, to determine connection features corresponding to each second neighbor point in each second nearest neighbor point set;

[0120] Step S239, performing feature splicing on connection features corresponding to each second neighbor point in each second nearest neighbor point set, to generate second neighbor point splicing features corresponding to each second nearest neighbor point set;

[0121] Step S2310: input the second neighbor point concatenation features corresponding to each second nearest neighbor point set into a multi-layer perceptron to generate second geometric embedding features corresponding to each second nearest neighbor point set;

[0122] Step S2311: construct a second geometric embedding matrix using the second geometric embedding features corresponding to each second nearest neighbor point set.

[0123] Step S24, concatenating the first geometric embedding matrix and the first target high-level visual feature matrix in the first target matrix to generate a first concatenated matrix;

[0124] Step S25: splicing the second geometric embedding matrix and the second target high-level visual feature matrix in the second target matrix to generate a second splicing matrix.

[0125] Please note that Figure 3 , for the first target coordinate matrix input Each matrix element in , using the K nearest neighbor algorithm to collect The nearest K points As the nearest neighbor point set, for The jth neighbor of the nearest neighbor point; for each neighbor point in the above nearest neighbor point set , construct the distance geometric features of the point. The features include point and Point The spatial coordinates of the two points, the relative position relationship between , the Euclidean distance between two points .

[0126] Furthermore, the nearest neighbor point set collected above is used as the feature matrix to calculate the eigenvalues ​​of the covariance matrix , , ,in , and calculate the linear features , planarity characteristics , divergence characteristics For example, if K is 3, then for the point There are three nearest neighbors , , , construct the characteristic matrix of the nearest neighbor point and calculate the covariance matrix eigenvalues ​​and linear characteristics of the matrix , planarity characteristics , divergence characteristics , that is, a nearest neighbor point set corresponds to a linear feature , a planar feature , a divergence feature .

[0127] Further, the point The feature construction is carried out to obtain the point The connection characteristics of ,in is a feature connection operation. Finally, the features of the K points are concatenated and input into the MLP layer (multi-layer perceptron) to obtain the point The first geometric embedding feature ,in , is the feature dimension after geometric embedding. For example, if K is 3, then for point There are three nearest neighbors , , , build point Connection characteristics , similarly construct point , Features , , and finally merge to get the point The first geometric embedding feature Based on the above steps, the first target coordinate matrix Each matrix element in is calculated one by one and forms a matrix form, namely the first geometric embedding matrix , and finally concatenated with the visual feature matrix of N points, that is, the first target high-level visual feature matrix in the first target matrix, to obtain the first concatenated matrix , is the high-level visual feature matrix in the first concatenated matrix.

[0128] It is worth mentioning that based on the above-mentioned steps and principles of the first splicing matrix, the second splicing matrix can be obtained in the same way , It is a high-level visual feature matrix in the second splicing matrix, and the present invention will not go into details any more.

[0129] Step 104: Perform feature fusion matching based on the first point cloud matrix and the second point cloud matrix through the Transformer feature fusion matching module, and output a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask.

[0130] It should be noted that due to the lack of acquisition angle in actual applications, there is a non-overlapping area between the first point cloud and the second point cloud. The present invention uses a mask predictor to generate a predicted mask. ,in , is the number of points in the point cloud S. The mask can reduce the interference of non-overlapping areas on the registration. The Transformer architecture is usually used to extract global context information. In the point cloud registration task, the general application strategy is to introduce the geometric structure features of the point cloud and learn point-related features and geometric structure features at the same time. For example, the existing geometric Transformer architecture (GeometricTransformer) not only encodes high-level point features, but also explicitly encodes the geometric structure within the point and the geometric consistency between points. The present invention follows this strategy and adjusts the attention mechanism therein on its basis.

[0131] Specifically, step 104 may include the following sub-steps S41-S42:

[0132] Step S41, using a Transformer feature fuser to generate a first target fused feature matrix and a second target fused feature matrix according to the first splicing matrix and the second splicing matrix;

[0133] Further, step S41 may include the following sub-steps S411-S415:

[0134] Step S411, using a self-attention mechanism to perform calculations based on the first splicing matrix and the second splicing matrix respectively, to determine a first attention feature matrix and a second attention feature matrix;

[0135] Step S412, linearly projecting the high-level visual feature matrix in the first stitching matrix and the high-level visual feature matrix in the second stitching matrix respectively to generate a first query matrix corresponding to the first stitching matrix and a second query matrix corresponding to the second stitching matrix;

[0136] Step S413, using a multi-layer perceptron to generate a first initial fusion feature matrix according to the first query matrix, the first attention feature matrix, and the high-level visual feature matrix in the first splicing matrix;

[0137] Step S414, using a multi-layer perceptron to generate a second initial fusion feature matrix according to the second query matrix, the second attention feature matrix, and the high-level visual feature matrix in the second splicing matrix;

[0138] Step S415: Determine a first target fusion feature matrix and a second target fusion feature matrix based on the first initial fusion feature matrix and the second initial fusion feature matrix based on the cross attention mechanism.

[0139] It should be noted that the feature matrix obtained based on the above steps, that is, the first concatenation matrix and the second stitching matrix , , d t For the dimension of a point feature, input feature matrix , after calculation by the self-attention mechanism, the first attention feature matrix is ​​obtained , the calculation formula is as follows:

[0140] ;

[0141] in, is the first attention feature matrix; is the high-level visual feature matrix in the first concatenated matrix; , , , are the projection matrices for query, key, value, and geometric embedding respectively; is the first geometric embedding matrix.

[0142] Furthermore, the feature matrix output above is With the query matrix After splicing, calculate and obtain the first initial fusion feature matrix , the process can be expressed as:

[0143] ;

[0144] in, It is a three-layer fully connected network with normalization and ReLU activation function, that is, a multilayer perceptron.

[0145] Further, based on the above calculation, the first initial fusion feature matrix The principle of the steps is based on the second splicing matrix Similarly, the second initial fusion feature matrix can be obtained At this time, the knowledge of the potential overlapping area can only be obtained by transferring the information of the two point clouds. Therefore, the present invention combines the cross attention mechanism to mix the information of the two point clouds, that is, As the query matrix, Calculate the first target fusion feature matrix as a matrix of keys and values ,by As the query matrix, The matrix of keys and values ​​is used to calculate and output the second target fusion feature matrix in the same way. The first target fusion feature matrix The calculation process can be expressed as:

[0146] ;

[0147] ;

[0148] in, for right The result after the cross attention mechanism calculation; is the final feature matrix representing the fusion, that is, the first target fusion feature matrix; Norm is the numerical normalization operation.

[0149] Step S42, using a mask predictor to perform mask prediction based on a first target high-level visual feature matrix in the first target matrix, a second target high-level visual feature matrix in the second target matrix, and a first restored low-level point feature matrix in the first restored matrix to generate a predicted mask.

[0150] Further, step S42 may include the following sub-steps S421-S425:

[0151] Step S421, performing maximum pooling on the first target high-level visual feature matrix in the first target matrix and the second target high-level visual feature matrix in the second target matrix, respectively, and outputting a first global feature vector and a second global feature vector;

[0152] Step S422, concatenating multiple rows of feature vectors in the first restored low-level point feature matrix in the first restored matrix with the first global feature vector and the second global feature vector to generate multiple matrix concatenated vectors;

[0153] Step S423, using a multi-layer perceptron to fuse multiple matrix splicing vectors, and outputting a splicing vector fusion matrix;

[0154] Step S424: Use multiple convolution layers to perform convolution operation on the splicing vector fusion matrix to determine the splicing vector convolution matrix;

[0155] Step S425: Perform nonlinear mapping on the concatenated vector convolution matrix through a Softmax activation function layer to generate a prediction mask.

[0156] It should be noted that the present invention adopts a mask predictor whose input includes high-level point features of two surface point clouds, namely, the first target high-level visual feature matrix in the first target matrix , the second target high-level visual feature matrix in the second target matrix , and low-level point features, that is, the first reduced low-level point feature matrix in the first reduced matrix , whose output is the prediction mask Specifically: high-level point features Input the maximum pooling layer, at this time the maximum value of each column of the feature matrix is ​​retained, and the global feature vectors of the two point clouds are obtained, that is, the first global feature vector and the second global eigenvector Then the global eigenvector , , spliced ​​in Each point feature vector in (each row feature vector ) to form the connected vector (matrix concatenation vector) And fuse the above matrix splicing vectors through a multi-layer perceptron to obtain a splicing vector fusion matrix. Finally, multiple convolutional layers and a softmax layer are used as the prediction function Output prediction mask ,in The value range of each row is [0,1].

[0157] Step 105: Use the first point cloud, the third point cloud, the prediction mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of a transformation matrix estimation module, and output a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds.

[0158] Specifically, step 105 may include the following sub-steps S51-S55:

[0159] Step S51, generating a confidence matrix and a Gaussian correlation matrix according to the first target fusion feature matrix, the second target fusion feature matrix, the first restored low-level point feature matrix in the first restored matrix, and the second restored low-level point feature matrix in the second restored matrix;

[0160] Step S52: Based on the preset topk strategy, the matrix elements in the confidence matrix and the matrix elements in the Gaussian correlation matrix are screened using the prediction mask to determine the point relationship between the first and second point clouds;

[0161] Step S53, based on the singular value decomposition method, the point relationship between the first and second point clouds is used to optimize and solve the preset optimization objective function to determine the optimal rigid transformation matrix;

[0162] Step S54, using the optimal rigid transformation matrix to transform the first point cloud and the third point cloud respectively, to generate a first transformed point cloud and a third transformed point cloud;

[0163] Step S55: construct a point correspondence relationship sequence between the first and second point clouds according to the point relationships between the first and second point clouds.

[0164] It should be noted that the transformation matrix estimation module proposed in the present invention calls the feature fusion result of the Transformer feature fusion module, that is, , ,as well as , , and calculate the confidence matrix and the Gaussian correlation matrix , the matrix elements in the confidence matrix and the matrix elements in the Gaussian correlation matrix are calculated as follows:

[0165] ;

[0166] ;

[0167] ;

[0168] in, is the matrix element in the i-th row and j-th column of the confidence matrix, indicating the confidence; is the matrix element in the i-th row and j-th column of the Gaussian correlation matrix; is the similarity score matrix, which reflects the similarity of the two point cloud features, and the confidence The value should be greater than or equal to and The value in; d is and Feature matrix dimensions; For the matrix The i-th row element in ; For the matrix The j-th column element in ; for The i-th row element of ; for The j-th row element of ; is the square of the L2 norm.

[0169] Furthermore, using the prediction mask To reduce the interference of non-overlapping areas on point registration, the topk strategy (preset topk strategy) is used to select the point relationships that can maintain the top K rankings in both rows and columns from the confidence matrix and Gaussian correlation matrix one by one, that is, the point relationships between multiple first and second point clouds. Add to the point correspondence to obtain the point correspondence sequence between the first and second point clouds .

[0170] Further, according to the point correspondence sequence between the first and second point clouds , combined with the singular value decomposition (SVD) decomposition method, the rigid transformation matrix required to align the first point cloud with the second point cloud can be obtained. Finally, with the goal of minimizing the registration error between the first point cloud and the second point cloud, a selection is made among all possible rigid transformation matrices to solve the optimal rigid transformation matrix {R, t}, where the optimization objective function (preset optimization objective function) is as follows:

[0171] ;

[0172] Where R is the rotation matrix; t is the translation vector; is the coordinate value of the point element in the first point cloud; is the coordinate value of the point element in the second point cloud, and Points and A pair of points is a matching pair of points.

[0173] Furthermore, the optimal rigid transformation matrix is ​​finally applied to the first point cloud S and the third point cloud P, and the transformed point clouds are output, namely the first transformed point cloud S' and the second transformed point cloud P'. It is worth noting that the point correspondence relationship is It remains unchanged before and after the rigid body transformation, that is, .

[0174] Step 106: Use the deformation field prediction module to perform registration according to the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result.

[0175] It should be noted that the present invention uses the deformation field prediction module to call the point cloud T and the point clouds S' and P' after rigid body transformation as input to predict a deformation field As output, is the number of points in point cloud S, and the deformation field is applied to point cloud S' to obtain the final precise registration result. It is worth noting that point cloud S' and point cloud P' are equal in the number of points, and there is a one-to-one correspondence between the points. Therefore, the point correspondence between point cloud S' and T is It is equivalent to the point correspondence between point clouds P' and T, that is, Based on this prior knowledge, this module completes the missing parts of the point cloud perspective.

[0176] Specifically, step 106 may include the following sub-steps S61-S612:

[0177] Step S61, inputting the first transformed point cloud, the third transformed point cloud and the second point cloud into a multi-layer perceptron respectively, and outputting a first increased dimension transformed point cloud matrix, a third increased dimension transformed point cloud matrix and a second increased dimension point cloud matrix;

[0178] Step S62: based on the point correspondence sequence between the first and second point clouds, segment the matrix elements in the third dimension-enlarged transformed point cloud matrix to determine a plurality of unmatched matrix elements;

[0179] Step S63, adding a plurality of unmatched relationship matrix elements to the second increased dimension point cloud matrix to determine a second supplementary point cloud matrix;

[0180] Step S64, fusing the second supplementary point cloud matrix and the third dimension-enlarged transformed point cloud matrix, and outputting the second and third fused point cloud matrices;

[0181] Step S65, using a multi-layer perceptron to generate a second and a third dimension enlarged feature matrix and a first dimension enlarged feature matrix according to the second and the third fused point cloud matrix and the first dimension enlarged transformation point cloud matrix respectively;

[0182] Step S66, performing maximum pooling on the second and third dimension enlarged feature matrices and the first dimension enlarged feature matrix respectively to generate second and third target global features and the first target global feature;

[0183] Step S67, respectively splicing the multiple rows of feature vectors in the first dimension-enlarged transformed point cloud matrix with the second and third target global features and the first target global features to generate multiple matrix global splicing vectors;

[0184] Step S68, fusing multiple matrix global splicing vectors and outputting a predicted deformation field;

[0185] Step S69, generating a target point cloud matrix according to the predicted deformation field and the first transformed point cloud;

[0186] Step S610, transforming the second supplementary point cloud matrix to determine the second supplementary point cloud coordinate matrix;

[0187] Step S611, performing error calculation according to the target point cloud matrix and the second supplementary point cloud coordinate matrix, determining the target registration error, and judging whether the target registration error is less than or equal to a preset error threshold;

[0188] Step S612: If yes, the target point cloud matrix and the second supplementary point cloud coordinate matrix are used as the liver point cloud registration result.

[0189] It should be noted that the multi-layer perceptron is used to extract the features of point clouds S', P' and T respectively and increase the feature dimension to 128 dimensions to obtain the first increased dimension transformation point cloud matrix, the third increased dimension transformation point cloud matrix and the second increased dimension point cloud matrix; according to the element correspondence between P' and T , segment the point cloud matrix elements that do not obtain matching relationships from the third increased dimension transformation point cloud matrix, that is, multiple unmatched relationship matrix elements, and add them to the second increased dimension point cloud matrix to obtain the completed point cloud matrix (that is, the second supplementary point cloud matrix), ensure that the number of elements in the second supplementary point cloud matrix is ​​consistent with the first increased dimension transformation point cloud matrix, and finally obtain the corresponding relationship of the point cloud matrix elements In order to more clearly illustrate the fusion of features, the first dimension-enhanced transformation point cloud matrix and the second supplementary point cloud matrix can be obtained by and The third dimension-enhanced transformation point cloud matrix contains prior information about the deformation. This fusion strategy can complement the missing perspectives while allowing the model to learn the potential laws of liver deformation, thereby improving the prediction performance.

[0190] Furthermore, the feature matrix and The dimension is further increased to 1024 through a multi-layer perceptron, and then the global features are obtained through maximum pooling, namely the second and third target global features. and the first target global feature , and then the global features , Concatenate in feature matrix After filtering each row of elements (each row of feature vectors), a multi-layer perceptron is used to fuse the features and output the predicted deformation field U.

[0191] Furthermore, the final reconstructed target point cloud matrix is Then, the dimension of the second supplementary point cloud matrix is ​​converted from 128 dimensions to 3 dimensions, and according to this correspondence: part of the eigenvector in the second supplementary point cloud matrix corresponds to the point coordinate vector in the second point cloud, and part corresponds to the point coordinate vector in the third transformed point cloud, and the second supplementary point cloud coordinate matrix is ​​obtained. Finally, the target point cloud matrix is ​​calculated. The overall difference with the second supplementary point cloud coordinate matrix, that is, the final registration error (target registration error RE), is calculated as follows:

[0192] ;

[0193] Among them, RE is the target registration error; The midpoint of the second supplementary point cloud coordinate matrix The coordinate value of The midpoint of the target point cloud matrix The coordinate value of is the point matching relationship between the target point cloud matrix and the second supplementary point cloud coordinate matrix; It is the registered element in the target point cloud matrix; is the registered element in the second supplementary point cloud coordinate matrix, and There is a corresponding relationship.

[0194] Furthermore, if the target registration error is greater than a preset error threshold, the model parameters of the preset finite element analysis solver are updated based on the doctor's experience, the updated preset finite element analysis solver is used to update the second point cloud to obtain a new second point cloud, the new second point cloud is transformed using the optimal rigid transformation matrix to obtain a new second transformed point cloud, and the process jumps to step S61 until the target registration error is less than or equal to the preset error threshold. The target point cloud matrix and the second supplementary point cloud coordinate matrix determined when the target registration error is less than or equal to the preset error threshold are used as the liver point cloud registration result.

[0195] For example, see Figure 5For the intraoperative laparoscopic video acquisition unit, the present invention adopts the STORZ 3D ultra-high-definition laparoscope as a device for collecting intraoperative laparoscopic videos, which has the advantages of large field of view, high resolution, strong brightness, clear imaging, etc., and can be used alternately between 3D and 2D. For the preoperative CT reconstruction point cloud storage unit, the present invention realizes the spatial positioning of the instrument through an optical positioning system, obtains the position parameters of the camera, and the positioning accuracy reaches 0.25mm. The preoperative liver morphology is three-dimensionally reconstructed by three-dimensional visualization technology such as CT or MRI, and a three-dimensional model including vascular distribution, liver surface, and liver parenchyma is generated by medical image three-dimensional visualization software, and preprocessed into a grid model and a point cloud model (i.e., a liver point cloud registration model based on the Transformer architecture). The unit saves the model and related parameters in the storage unit, and compresses the data through the octree spatial index structure. For the finite element simulation unit, the present invention uses different reasonable assumption models to analyze the liver deformation. The present invention uses a mass spring model to analyze the liver surface deformation, and sets parameters according to the biomechanical properties of the liver tissue. Use OpenGL to complete virtual scene rendering, and use graphics processing unit (GPU) parallel computing to improve simulation speed. Establish a respiratory motion compensation model, and predict the organ motion trajectory through a periodic displacement field. For the point cloud registration unit, a variety of registration methods can be performed, including manual registration mode, coarse registration mode based on the improved ICP-Robust registration algorithm, and the multimodal fusion registration algorithm provided by the present invention. When the registration error is greater than the threshold, the finite element model update is automatically triggered for the doctor to choose calibration. For the three-dimensional visualization fusion display unit, the data of the remaining units can be called at the same time and rendered to the display in real time for the doctor to make intraoperative judgments and choices. At the same time, the unit will plan the vascular area and the liver parenchyma area, calculate and display the real-time distance between the current robot scalpel and each tissue, and provide the doctor with accurate, safe, and predictable navigation information for surgical cutting.

[0196] As a comparison of technical effects, we can refer to existing technologies. In recent years, point cloud registration algorithms based on deep learning have shown better performance than traditional algorithms, further improving the registration effect. Among them, the model based on the Transformer architecture has shown potential in modeling remote contextual dependencies of point cloud data. Due to the permutation equivariant operation and good global feature learning capabilities, the Transformer model architecture is suitable for point cloud processing and analysis, but the application of Transformer to liver point cloud registration also requires optimization for surgical scenarios.

[0197] First, the preoperative point cloud reconstructed by CT scan can cover the entire liver surface, while the point cloud collected by laparoscopy during surgery only covers part of the liver surface. The loss of some perspectives will affect the overall registration effect. Current registration techniques usually use learned overlapping masks to filter out non-overlapping areas, but this will weaken the model's attention to global information.

[0198] Second, affected by factors such as breathing, pneumoperitoneum pressure, instruments, and body posture during surgery, the liver as a soft tissue will deform. Therefore, the point cloud collected during surgery has non-rigid deformation compared to the point cloud reconstructed before surgery.

[0199] In the registration task, usually only the liver surface is used as the registration object and the internal structure of the liver is ignored. However, the liver parenchyma, intrahepatic vascular system, liver ligaments and other structures have different biomechanical properties, so the liver surface deformation cannot directly reflect the position and state of the liver's internal tissues. In clinical applications, after the liver surface is registered, the problem of the internal blood vessels of the liver cannot be registered often occurs, which in turn affects the accuracy of surgical navigation.

[0200] In response to the above problems, the present invention proposes a liver point cloud registration method based on the Transformer architecture, which applies the point cloud registration technology based on the Transformer architecture to obtain the matching relationship between the preoperative and intraoperative models, and combines finite element analysis to obtain a deformation point cloud that conforms to the laws of biomechanics to complete the missing perspective and predict the deformation field, and more accurately reflect the trend of various tissues inside the liver. Specifically, through three-dimensional imaging, finite element analysis and multimodal fusion technology, an accurate registration effect is obtained to improve the planning and execution of laparoscopic liver surgery. The main feature of this application is the fusion of multimodal data for registration and alignment: one is the preoperative liver point cloud of CT scan, the second is the intraoperative liver surface point cloud captured and reconstructed by a real-time binocular camera, and the third is the liver point cloud model obtained by simulating the deformation of the preoperative liver point cloud under biomechanical constraints. This application integrates multimodal data into a deep learning framework, uses a Transformer architecture suitable for point clouds to fuse features of different modalities, captures potential matching relationships between modalities, and ultimately calculates the rigid transformation matrix and predicts the non-rigid deformation field through point matching relationships to improve the registration accuracy, solving problems such as intraoperative tissue deformation, intraoperative loss of perspective, and decreased preoperative planning accuracy.

[0201] In an embodiment of the present invention, the present invention provides a liver point cloud registration method based on the Transformer architecture. First, a first point cloud and a second point cloud are obtained, and the first point cloud is preprocessed based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud; then, the first point cloud, the second point cloud and the third point cloud are input into a liver point cloud registration model based on the Transformer architecture. The liver point cloud registration model based on the Transformer architecture includes a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module; the feature extraction module is used to perform feature extraction based on the first point cloud and the second point cloud to generate a first point cloud matrix and a second point cloud matrix; the Transformer feature fusion matching module is used to perform feature fusion matching based on the first point cloud matrix and the second point cloud matrix, and a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask are output; the first point cloud, The third point cloud, the predicted mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix are used as inputs of the transformation matrix estimation module, and the point correspondence relationship sequence between the first transformed point cloud, the third transformed point cloud and the first and second point clouds is output; finally, the deformation field prediction module is used to align the first transformed point cloud, the third transformed point cloud, the point correspondence relationship sequence between the first and second point clouds and the second point cloud to generate a liver point cloud registration result; based on the above scheme, the first point cloud and the second point cloud are obtained, the first point cloud is preprocessed by a preset finite element analysis solver and a preset deformation condition to generate a third point cloud, and the first point cloud, the second point cloud and the third point cloud are processed in combination with the liver point cloud registration model based on the Transformer architecture to generate a process of generating a liver point cloud registration result. The present invention adopts a point cloud Transformer architecture suitable for fusing different modal features, which can capture the potential matching relationship between each modality, thereby improving the registration effect.

[0202] See also Figure 6 , Figure 6 This is a structural block diagram of a liver point cloud registration device based on the Transformer architecture provided in the second embodiment of the present invention.

[0203] The present invention provides a liver point cloud registration device based on Transformer architecture, comprising:

[0204] An acquisition module 601 is used to acquire a first point cloud and a second point cloud, and pre-process the first point cloud based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud;

[0205] An input module 602 is used to input the first point cloud, the second point cloud and the third point cloud into a liver point cloud registration model based on a Transformer architecture, wherein the liver point cloud registration model based on a Transformer architecture includes a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module;

[0206] A feature extraction module 603, configured to extract features based on the first point cloud and the second point cloud using a feature extraction module to generate a first point cloud matrix and a second point cloud matrix;

[0207] A feature fusion matching module 604 is used to perform feature fusion matching according to the first point cloud matrix and the second point cloud matrix through a Transformer feature fusion matching module, and output a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask;

[0208] The transformation module 605 is used to use the first point cloud, the third point cloud, the prediction mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of the transformation matrix estimation module, and output a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds;

[0209] The registration module 606 is used to use the deformation field prediction module to perform registration according to the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result.

[0210] Furthermore, the acquisition module 601 is specifically used for:

[0211] Based on the preset deformation condition, a preset finite element analysis solver is used to perform deformation processing on the first point cloud to generate a third point cloud.

[0212] Further, the first point cloud matrix includes a first target matrix, a first restoration matrix and a first splicing matrix; the second point cloud matrix includes a second target matrix, a second restoration matrix and a second splicing matrix; the feature extraction module includes an encoder, a decoder and a geometric embedder; the feature extraction module 603 includes:

[0213] A first submodule is used to use an encoder to downsample the first point cloud and extract point-by-point visual features from the second point cloud, respectively, to generate a first target matrix and a second target matrix;

[0214] The second submodule is used to perform feature restoration on the first target matrix and the second target matrix respectively through a decoder, and output a first restoration matrix and a second restoration matrix;

[0215] A third submodule is used to use the first target coordinate matrix in the first target matrix and the second target coordinate matrix in the second target matrix as inputs of the geometric embedder respectively, and output the first geometric embedding matrix and the second geometric embedding matrix;

[0216] A fourth submodule is used to splice the first geometric embedding matrix and the first target high-level visual feature matrix in the first target matrix to generate a first splicing matrix;

[0217] The fifth submodule is used to splice the second geometric embedding matrix and the second target high-level visual feature matrix in the second target matrix to generate a second splicing matrix.

[0218] Furthermore, the third submodule is specifically used for:

[0219] Use the K nearest neighbor algorithm to collect neighboring points of multiple first matrix elements in the first target coordinate matrix and multiple second matrix elements in the second target coordinate matrix, respectively, to generate a first nearest neighbor point set corresponding to each first matrix element and a second nearest neighbor point set corresponding to each second matrix element;

[0220] Constructing distance geometric features for multiple first neighboring points in each first nearest neighboring point set and multiple second neighboring points in each second nearest neighboring point set, and outputting multiple positional relationships and multiple Euclidean distances corresponding to each first neighboring point in each first nearest neighboring point set, and multiple positional relationships and multiple Euclidean distances corresponding to each second neighboring point in each second nearest neighboring point set;

[0221] Perform matrix covariance calculation on each first nearest neighbor point set and each second nearest neighbor point set respectively, and output feature data corresponding to each first nearest neighbor point set and feature data corresponding to each second nearest neighbor point set;

[0222] Perform feature connection based on multiple positional relationships and multiple Euclidean distances corresponding to each first neighbor point in each first nearest neighbor point set and feature data corresponding to each first nearest neighbor point set, and determine connection features corresponding to each first neighbor point in each first nearest neighbor point set;

[0223] Perform feature splicing on the connection features corresponding to each first neighbor point in each first nearest neighbor point set, and generate first neighbor point splicing features corresponding to each first nearest neighbor point set;

[0224] Inputting the first neighbor point concatenation features corresponding to each first nearest neighbor point set into a multi-layer perceptron to generate the first geometric embedding features corresponding to each first nearest neighbor point set;

[0225] Using the first geometric embedding features corresponding to each first nearest neighbor point set to construct a first geometric embedding matrix;

[0226] Perform feature connection based on multiple positional relationships and multiple Euclidean distances corresponding to each second neighbor point in each second nearest neighbor point set and feature data corresponding to each second nearest neighbor point set, and determine connection features corresponding to each second neighbor point in each second nearest neighbor point set;

[0227] Perform feature splicing on the connection features corresponding to each second neighbor point in each second nearest neighbor point set, and generate second neighbor point splicing features corresponding to each second nearest neighbor point set;

[0228] Inputting the second neighbor point concatenation features corresponding to each second nearest neighbor point set into a multi-layer perceptron to generate second geometric embedding features corresponding to each second nearest neighbor point set;

[0229] A second geometric embedding matrix is ​​constructed by using the second geometric embedding features corresponding to each second nearest neighbor point set.

[0230] Furthermore, the Transformer feature fusion matching module includes a mask predictor and a Transformer feature fuser; the feature fusion matching module 604 includes:

[0231] A sixth submodule is used to generate a first target fused feature matrix and a second target fused feature matrix according to the first splicing matrix and the second splicing matrix using a Transformer feature fuser;

[0232] The seventh submodule is used to perform mask prediction based on the first target high-level visual feature matrix in the first target matrix, the second target high-level visual feature matrix in the second target matrix, and the first restored low-level point feature matrix in the first restored matrix through a mask predictor to generate a predicted mask.

[0233] Furthermore, the sixth submodule is specifically used for:

[0234] A self-attention mechanism is used to perform calculations according to the first splicing matrix and the second splicing matrix respectively to determine a first attention feature matrix and a second attention feature matrix;

[0235] Performing linear projection on the high-level visual feature matrix in the first splicing matrix and the high-level visual feature matrix in the second splicing matrix respectively, to generate a first query matrix corresponding to the first splicing matrix and a second query matrix corresponding to the second splicing matrix;

[0236] A multi-layer perceptron is used to generate a first initial fusion feature matrix according to the first query matrix, the first attention feature matrix, and the high-level visual feature matrix in the first splicing matrix;

[0237] A multi-layer perceptron is used to generate a second initial fused feature matrix according to the second query matrix, the second attention feature matrix, and the high-level visual feature matrix in the second concatenation matrix;

[0238] Based on the cross attention mechanism, a first target fusion feature matrix and a second target fusion feature matrix are determined according to the first initial fusion feature matrix and the second initial fusion feature matrix.

[0239] Furthermore, the mask predictor includes a plurality of convolutional layers and a Softmax activation function layer; the seventh submodule is specifically used for:

[0240] Performing maximum pooling on the first target high-level visual feature matrix in the first target matrix and the second target high-level visual feature matrix in the second target matrix respectively, outputting a first global feature vector and a second global feature vector;

[0241] Splicing multiple rows of feature vectors in the first restored low-level point feature matrix in the first restored matrix with the first global feature vector and the second global feature vector to generate multiple matrix splicing vectors;

[0242] A multi-layer perceptron is used to fuse multiple matrix splicing vectors and output a splicing vector fusion matrix;

[0243] Using multiple convolution layers to perform convolution operations on the splicing vector fusion matrix to determine the splicing vector convolution matrix;

[0244] The concatenated vector convolution matrix is ​​nonlinearly mapped through the Softmax activation function layer to generate a prediction mask.

[0245] Furthermore, the transformation module 605 is specifically used for:

[0246] Generate a confidence matrix and a Gaussian correlation matrix according to the first target fusion feature matrix, the second target fusion feature matrix, the first restored low-level point feature matrix in the first restored matrix, and the second restored low-level point feature matrix in the second restored matrix;

[0247] Based on the preset topk strategy, the matrix elements in the confidence matrix and the matrix elements in the Gaussian correlation matrix are screened using the prediction mask to determine the point relationship between the multiple first and second point clouds;

[0248] Based on the singular value decomposition method, the preset optimization objective function is optimized and solved by using the point relationship between the first and second point clouds to determine the optimal rigid transformation matrix;

[0249] The first point cloud and the third point cloud are transformed respectively by using an optimal rigid transformation matrix to generate a first transformed point cloud and a third transformed point cloud;

[0250] According to the point relationships between the first and second point clouds, a point correspondence relationship sequence between the first and second point clouds is constructed.

[0251] Furthermore, the registration module 606 is specifically configured to:

[0252] Input the first transformed point cloud, the third transformed point cloud and the second point cloud into a multi-layer perceptron respectively, and output a first increased dimension transformed point cloud matrix, a third increased dimension transformed point cloud matrix and a second increased dimension point cloud matrix;

[0253] Based on the point correspondence relationship sequence between the first and second point clouds, segment the matrix elements in the third increased dimension transformation point cloud matrix to determine a plurality of unmatched relationship matrix elements;

[0254] adding a plurality of unmatched relationship matrix elements to the second increased dimension point cloud matrix to determine a second supplementary point cloud matrix;

[0255] Fusing the second supplementary point cloud matrix and the third dimension-enlarged transformed point cloud matrix, and outputting second and third fused point cloud matrices;

[0256] A multi-layer perceptron is used to generate a second and a third dimension enlarged feature matrix and a first dimension enlarged feature matrix according to the second and the third fused point cloud matrix and the first dimension enlarged transformation point cloud matrix respectively;

[0257] Performing maximum pooling on the second and third dimension enlarged feature matrices and the first dimension enlarged feature matrix respectively to generate the second and third target global features and the first target global features;

[0258] Respectively splicing a plurality of rows of feature vectors in the first dimension-enlarged transformed point cloud matrix with the second and third target global features and the first target global features to generate a plurality of matrix global splicing vectors;

[0259] Fusion of multiple matrix global splicing vectors to output predicted deformation field;

[0260] Generate a target point cloud matrix according to the predicted deformation field and the first transformed point cloud;

[0261] Transforming the second supplementary point cloud matrix to determine a second supplementary point cloud coordinate matrix;

[0262] Perform error calculation based on the target point cloud matrix and the second supplementary point cloud coordinate matrix to determine the target registration error, and determine whether the target registration error is less than or equal to a preset error threshold;

[0263] If yes, the target point cloud matrix and the second supplementary point cloud coordinate matrix are used as the liver point cloud registration result.

[0264] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0265] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A liver point cloud registration method based on Transformer architecture, characterized in that: include: Acquire a first point cloud and a second point cloud, and preprocess the first point cloud based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud; Inputting the first point cloud, the second point cloud and the third point cloud into a liver point cloud registration model based on a Transformer architecture, wherein the liver point cloud registration model based on a Transformer architecture comprises a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module; Using the feature extraction module to perform feature extraction based on the first point cloud and the second point cloud to generate a first point cloud matrix and a second point cloud matrix; Performing feature fusion matching according to the first point cloud matrix and the second point cloud matrix through the Transformer feature fusion matching module, and outputting a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask; Taking the first point cloud, the third point cloud, the predicted mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of the transformation matrix estimation module, and outputting a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds; Using the deformation field prediction module to perform registration according to the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result; The step of using the deformation field prediction module to register the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result includes: Input the first transformed point cloud, the third transformed point cloud and the second point cloud into a multi-layer perceptron respectively, and output a first increased dimension transformed point cloud matrix, a third increased dimension transformed point cloud matrix and a second increased dimension point cloud matrix; Based on the point correspondence relationship sequence between the first and second point clouds, segmenting the matrix elements in the third dimension-enlarged transformed point cloud matrix to determine a plurality of unmatched relationship matrix elements; Adding a plurality of the unmatched relationship matrix elements to the second increased dimension point cloud matrix to determine a second supplementary point cloud matrix; Fusing the second supplementary point cloud matrix and the third dimension-enlarged transformed point cloud matrix, and outputting second and third fused point cloud matrices; Using a multi-layer perceptron pair to generate a second and a third dimension enlarged feature matrix and a first dimension enlarged feature matrix according to the second and third fused point cloud matrices and the first dimension enlarged transformed point cloud matrix respectively; Performing maximum pooling on the second and third dimension-enlarged feature matrices and the first dimension-enlarged feature matrix respectively to generate second and third target global features and the first target global feature; Respectively splicing a plurality of rows of feature vectors in the first dimension-increased transformed point cloud matrix with the second and third target global features and the first target global feature to generate a plurality of matrix global splicing vectors; fusing a plurality of matrix global splicing vectors to output a predicted deformation field; generating a target point cloud matrix according to the predicted deformation field and the first transformed point cloud; Transforming the second supplementary point cloud matrix to determine a second supplementary point cloud coordinate matrix; Performing error calculation according to the target point cloud matrix and the second supplementary point cloud coordinate matrix to determine a target registration error, and judging whether the target registration error is less than or equal to a preset error threshold; If yes, the target point cloud matrix and the second supplementary point cloud coordinate matrix are used as the liver point cloud registration result.

2. The liver point cloud registration method based on Transformer architecture according to claim 1, characterized in that: The preprocessing of the first point cloud based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud includes: Based on the preset deformation condition, the preset finite element analysis solver is used to perform deformation processing on the first point cloud to generate a third point cloud.

3. The liver point cloud registration method based on Transformer architecture according to claim 1, characterized in that: The first point cloud matrix includes a first target matrix, a first restoration matrix and a first splicing matrix; the second point cloud matrix includes a second target matrix, a second restoration matrix and a second splicing matrix; the feature extraction module includes an encoder, a decoder and a geometric embedder; The step of using the feature extraction module to perform feature extraction based on the first point cloud and the second point cloud to generate a first point cloud matrix and a second point cloud matrix includes: Using an encoder to downsample the first point cloud and the second point cloud and extract point-by-point visual features to generate a first target matrix and a second target matrix; Performing feature restoration on the first target matrix and the second target matrix respectively through a decoder, and outputting a first restoration matrix and a second restoration matrix; Using the first target coordinate matrix in the first target matrix and the second target coordinate matrix in the second target matrix as inputs of a geometric embedder, respectively, and outputting a first geometric embedding matrix and a second geometric embedding matrix; splicing the first geometric embedding matrix and the first target high-level visual feature matrix in the first target matrix to generate a first splicing matrix; The second geometric embedding matrix and the second target high-level visual feature matrix in the second target matrix are concatenated to generate a second concatenated matrix.

4. The liver point cloud registration method based on Transformer architecture according to claim 3, characterized in that: The method of using the first target coordinate matrix in the first target matrix and the second target coordinate matrix in the second target matrix as inputs of a geometric embedder, and outputting a first geometric embedding matrix and a second geometric embedding matrix, comprises: Using a K nearest neighbor algorithm to collect neighboring points of a plurality of first matrix elements in the first target coordinate matrix and a plurality of second matrix elements in the second target coordinate matrix, respectively, to generate a first nearest neighbor point set corresponding to each of the first matrix elements and a second nearest neighbor point set corresponding to each of the second matrix elements; Constructing distance geometric features for the multiple first neighboring points in each of the first nearest neighboring point sets and the multiple second neighboring points in each of the second nearest neighboring point sets, respectively, and outputting multiple positional relationships and multiple Euclidean distances corresponding to the first neighboring points in each of the first nearest neighboring point sets, and multiple positional relationships and multiple Euclidean distances corresponding to the second neighboring points in each of the second nearest neighboring point sets; Performing matrix covariance calculation on each of the first nearest neighbor point sets and each of the second nearest neighbor point sets respectively, and outputting feature data corresponding to each of the first nearest neighbor point sets and feature data corresponding to each of the second nearest neighbor point sets; Perform feature connection based on multiple positional relationships and multiple Euclidean distances corresponding to each first neighbor point in each of the first nearest neighbor point sets and feature data corresponding to each of the first nearest neighbor point sets, and determine connection features corresponding to each of the first neighbor points in each of the first nearest neighbor point sets; Performing feature splicing on connection features corresponding to each first neighbor point in each of the first nearest neighbor point sets respectively, to generate first neighbor point splicing features corresponding to each of the first nearest neighbor point sets; Inputting the first neighbor point concatenation features corresponding to each of the first nearest neighbor point sets into a multi-layer perceptron to generate first geometric embedding features corresponding to each of the first nearest neighbor point sets; Using first geometric embedding features corresponding to each of the first nearest neighbor point sets to construct a first geometric embedding matrix; Perform feature connection based on multiple positional relationships and multiple Euclidean distances corresponding to each second neighbor point in each second nearest neighbor point set and feature data corresponding to each second nearest neighbor point set, and determine connection features corresponding to each second neighbor point in each second nearest neighbor point set; Performing feature splicing on connection features corresponding to each second neighbor point in each of the second nearest neighbor point sets respectively, to generate second neighbor point splicing features corresponding to each of the second nearest neighbor point sets; Inputting the second neighbor point concatenation features corresponding to each of the second nearest neighbor point sets into a multi-layer perceptron to generate second geometric embedding features corresponding to each of the second nearest neighbor point sets; A second geometric embedding matrix is ​​constructed using the second geometric embedding features corresponding to each of the second nearest neighbor point sets.

5. The liver point cloud registration method based on Transformer architecture according to claim 3, characterized in that: The Transformer feature fusion matching module includes a mask predictor and a Transformer feature fusion device; the Transformer feature fusion matching module performs feature fusion matching according to the first point cloud matrix and the second point cloud matrix, and outputs a first target fusion feature matrix, a second target fusion feature matrix and a predicted mask, including: Using a Transformer feature fuser to generate a first target fused feature matrix and a second target fused feature matrix according to the first splicing matrix and the second splicing matrix; A mask predictor is used to perform mask prediction based on a first target high-level visual feature matrix in the first target matrix, a second target high-level visual feature matrix in the second target matrix, and a first restored low-level point feature matrix in the first restored matrix to generate a predicted mask.

6. The liver point cloud registration method based on Transformer architecture according to claim 5, characterized in that: The using of the Transformer feature fuser to generate a first target fusion feature matrix and a second target fusion feature matrix according to the first splicing matrix and the second splicing matrix includes: Using a self-attention mechanism to perform calculations based on the first splicing matrix and the second splicing matrix, respectively, to determine a first attention feature matrix and a second attention feature matrix; Performing linear projection on the high-level visual feature matrix in the first stitching matrix and the high-level visual feature matrix in the second stitching matrix respectively to generate a first query matrix corresponding to the first stitching matrix and a second query matrix corresponding to the second stitching matrix; Using a multi-layer perceptron to generate a first initial fused feature matrix according to the first query matrix, the first attention feature matrix, and the high-level visual feature matrix in the first concatenated matrix; Using a multi-layer perceptron to generate a second initial fused feature matrix according to the second query matrix, the second attention feature matrix, and the high-level visual feature matrix in the second concatenated matrix; Based on the cross attention mechanism, a first target fusion feature matrix and a second target fusion feature matrix are determined according to the first initial fusion feature matrix and the second initial fusion feature matrix.

7. The liver point cloud registration method based on Transformer architecture according to claim 5, characterized in that: The mask predictor includes a plurality of convolutional layers and a Softmax activation function layer; the mask predictor performs mask prediction according to a first target high-level visual feature matrix in the first target matrix, a second target high-level visual feature matrix in the second target matrix, and a first restored low-level point feature matrix in the first restored matrix to generate a predicted mask, including: performing maximum pooling on the first target high-level visual feature matrix in the first target matrix and the second target high-level visual feature matrix in the second target matrix respectively, and outputting a first global feature vector and a second global feature vector; Splicing a plurality of rows of feature vectors in the first restored low-level point feature matrix in the first restored matrix with the first global feature vector and the second global feature vector to generate a plurality of matrix splicing vectors; A multi-layer perceptron is used to fuse a plurality of matrix splicing vectors, and output a splicing vector fusion matrix; Using multiple convolution layers to perform convolution operation on the splicing vector fusion matrix to determine the splicing vector convolution matrix; The concatenated vector convolution matrix is ​​nonlinearly mapped through a Softmax activation function layer to generate a prediction mask.

8. The liver point cloud registration method based on Transformer architecture according to claim 3, characterized in that: The method of using the first point cloud, the third point cloud, the predicted mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of the transformation matrix estimation module, and outputting a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds, includes: Generate a confidence matrix and a Gaussian correlation matrix according to the first target fusion feature matrix, the second target fusion feature matrix, the first restored low-level point feature matrix in the first restored matrix, and the second restored low-level point feature matrix in the second restored matrix; Based on a preset topk strategy, the prediction mask is used to screen the matrix elements in the confidence matrix and the matrix elements in the Gaussian correlation matrix to determine the point relationship between the first and second point clouds; Based on the singular value decomposition method, a preset optimization objective function is optimized and solved by using the point relationship between the first and second point clouds to determine the optimal rigid transformation matrix; Using the optimal rigid transformation matrix to transform the first point cloud and the third point cloud respectively, to generate a first transformed point cloud and a third transformed point cloud; According to the point relationships between the first and second point clouds, a point correspondence relationship sequence between the first and second point clouds is constructed.

9. A liver point cloud registration device based on Transformer architecture, characterized in that: include: An acquisition module, used for acquiring a first point cloud and a second point cloud, and preprocessing the first point cloud based on a preset finite element analysis solver and a preset deformation condition to generate a third point cloud; An input module, used to input the first point cloud, the second point cloud and the third point cloud into a liver point cloud registration model based on a Transformer architecture, wherein the liver point cloud registration model based on a Transformer architecture includes a feature extraction module, a Transformer feature fusion matching module, a transformation matrix estimation module and a deformation field prediction module; A feature extraction module, configured to extract features based on the first point cloud and the second point cloud using the feature extraction module to generate a first point cloud matrix and a second point cloud matrix; A feature fusion matching module, used to perform feature fusion matching according to the first point cloud matrix and the second point cloud matrix through the Transformer feature fusion matching module, and output a first target fusion feature matrix, a second target fusion feature matrix and a prediction mask; a transformation module, configured to use the first point cloud, the third point cloud, the prediction mask, the first target fusion feature matrix, the second target fusion feature matrix, the first point cloud matrix and the second point cloud matrix as inputs of the transformation matrix estimation module, and output a sequence of point correspondences between the first transformed point cloud, the third transformed point cloud and the first and second point clouds; a registration module, configured to use the deformation field prediction module to perform registration according to the first transformed point cloud, the third transformed point cloud, the point correspondence sequence between the first and second point clouds, and the second point cloud to generate a liver point cloud registration result; The registration module is specifically used for: Input the first transformed point cloud, the third transformed point cloud and the second point cloud into a multi-layer perceptron respectively, and output a first increased dimension transformed point cloud matrix, a third increased dimension transformed point cloud matrix and a second increased dimension point cloud matrix; Based on the point correspondence relationship sequence between the first and second point clouds, segmenting the matrix elements in the third dimension-enlarged transformed point cloud matrix to determine a plurality of unmatched relationship matrix elements; Adding a plurality of the unmatched relationship matrix elements to the second increased dimension point cloud matrix to determine a second supplementary point cloud matrix; Fusing the second supplementary point cloud matrix and the third dimension-enlarged transformed point cloud matrix, and outputting second and third fused point cloud matrices; Using a multi-layer perceptron pair to generate a second and a third dimension enlarged feature matrix and a first dimension enlarged feature matrix according to the second and third fused point cloud matrices and the first dimension enlarged transformed point cloud matrix respectively; Performing maximum pooling on the second and third dimension-enlarged feature matrices and the first dimension-enlarged feature matrix respectively to generate second and third target global features and the first target global feature; Respectively splicing a plurality of rows of feature vectors in the first dimension-increased transformed point cloud matrix with the second and third target global features and the first target global feature to generate a plurality of matrix global splicing vectors; fusing a plurality of matrix global splicing vectors to output a predicted deformation field; generating a target point cloud matrix according to the predicted deformation field and the first transformed point cloud; Transforming the second supplementary point cloud matrix to determine a second supplementary point cloud coordinate matrix; Performing error calculation according to the target point cloud matrix and the second supplementary point cloud coordinate matrix to determine a target registration error, and judging whether the target registration error is less than or equal to a preset error threshold; If yes, the target point cloud matrix and the second supplementary point cloud coordinate matrix are used as the liver point cloud registration result.

Images