2D / 3D registration method based on causal inference
The 2D/3D registration method using causal inference and feature fusion improves accuracy by eliminating confounding factors and enhancing feature extraction, addressing the limitations of existing methods.
Patent Information
- Application Number
- CN202411191963.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-20
AI Technical Summary
Existing 2D/3D registration methods in medical imaging are affected by irrelevant factors, leading to inaccurate registration results, especially at specific angles, and computational costs increase exponentially with the number of training samples.
A 2D/3D registration method based on causal inference, utilizing a convolutional neural network with multi-label classification models and an attention-based point cloud feature extraction network to extract and fuse 2D and 3D features, followed by pose prediction to achieve alignment.
The method enhances feature extraction by eliminating confounding factors and improves registration accuracy through feature interaction and fusion, resulting in precise 2D/3D alignment.
Smart Images
Figure CN119169062A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of 2D / 3D registration, and in particular to a 2D / 3D registration method based on causal inference. Background Art
[0002] In medical imaging technology, 2D / 3D registration plays a vital role. With this technology, doctors can build a three-dimensional model based on the medical imaging data obtained before surgery, and then estimate the position and posture of surgical instruments in real time during surgery. The application of this technology enables doctors to understand the current position of surgical instruments more accurately, so as to perform more precise surgical operations.
[0003] However, intraoperative images often lack stereoscopic information, which limits the surgical field of view. In order to broaden this field of view, two-dimensional images are usually matched with three-dimensional models. Currently, 2D / 3D registration is mainly based on rigid registration, and its core purpose is to estimate the pose of two-dimensional images.
[0004] Among the existing registration methods, a common strategy is to calculate the digitally reconstructed radiographic image (DRR) in the 3DCT image and compare it with the real X-ray image for similarity. Registration is achieved by maximizing the similarity measure between the two. For example, some studies have adopted algorithms based on the training-inference decoupling architecture, enhanced feature representation through multi-scale convolution, and introduced adaptive activation functions to improve the nonlinear expression ability of the network, thereby meeting the clinical real-time requirements.
[0005] In addition, there are studies that calculate descriptors by training convolutional neural networks and use the k-NearestNeighbor method to determine the pose. Another method uses the distance between the projection points of the same 3D point under different camera perspectives as a similarity measure between 2D projection images. However, these methods may not be able to obtain accurate registration results in some registration scenarios, especially at certain specific angles. Therefore, it is particularly important to explore high-precision registration methods.
[0006] In addition to the similarity measurement-based methods, some studies have tried to let the network directly output the predicted pose parameters. This method avoids the process of image similarity measurement and instead adopts a data-driven approach to learn the estimation of pose parameters. For example, some studies use Markov models for decision making and combine them with deep convolutional neural networks for modeling. However, this method has a significant problem: as the number of training samples increases, the computational cost increases exponentially.
[0007] In the field of medical image processing, the research on 2D / 3D registration has always received widespread attention and has made a series of important progress. These advances include registration methods based on feature point detection, technologies that use graph neural networks to aggregate anatomical features at different points to improve registration accuracy and stability, and intensity-driven registration methods using normalized gradient field distance measurement. These research works not only bring new technical means to the field of 2D / 3D registration, but also provide valuable ideas and inspiration for subsequent research.
[0008] In summary, 2D / 3D registration technology plays a vital role in medical imaging. Although a series of progress has been made, there are still many challenges and problems to be solved. Therefore, continuous exploration and innovation are particularly important in this field. Summary of the invention
[0009] The technical problem to be solved by the present invention is to propose a 2D / 3D registration method based on causal inference to make up for the defect that the image features extracted by the existing method will be affected by irrelevant factors such as the background, so that it can better complete the 2D / 3D registration task in various scenarios.
[0010] The technical solution adopted by the present invention to solve the above technical problems is:
[0011] A 2D / 3D registration method based on causal inference includes the following steps:
[0012] Step S1: performing image causal feature extraction on the input image to obtain 2D features; the network for image causal feature extraction consists of a convolutional neural network, a first multi-label classification model, a hybrid dictionary, a second multi-label classification model and a fully connected network;
[0013] Step S2: extracting 3D features of the 3D point cloud of the scene through an attention-based point cloud feature extraction network to obtain 3D features. The attention-based point cloud feature extraction network includes a first linear layer, an attention module, a second linear layer and a global maximum pooling layer;
[0014] Step S3: perform feature interaction and fusion on the 2D features and the 3D features to obtain interaction features and fusion features; the feature interaction and fusion network structure includes three one-dimensional convolutions, two linear rectifier units, a global average pooling layer, and three fully connected networks;
[0015] Step S4: perform pose prediction based on the interactive features and fusion features to achieve 2D / 3D alignment.
[0016] Furthermore, step S1 specifically includes the following steps:
[0017] Step S11: Extract features of the image to be registered through a convolutional neural network:
[0018] Step S12: Forming a class activation map based on the first multi-label classification model: Inputting the feature map extracted by the convolutional neural network into the first multi-label classification model, and multiplying the weight of the first multi-label classification model classifier and the feature map to obtain a class activation map:
[0019] Step S13: After the first multi-label classification model outputs the corresponding label, all context information of each class is continuously stored according to the label result through the activation map of the class with the highest cumulative probability, and the context information is stored in the mixed dictionary; then, all the context of each class is projected as attention onto the feature map of the last convolutional layer to generate an enhanced feature map;
[0020] Step S14: Input the enhanced feature map into the second multi-label classification model to generate a more accurate prediction result. At the same time, expand the enhanced feature map and input it into the fully connected network to obtain the final 2D feature.
[0021] Furthermore, step S11 specifically includes the following steps:
[0022] Step S111: building a convolutional neural network, using a FCN-based network with multiple non-local blocks inserted in low and high layers;
[0023] Step S112: Input the image to be registered into the constructed convolutional neural network to obtain the corresponding feature map X∈R c ×h×w , where c represents the number of image classification categories, h represents the height dimension of the feature map, and w represents the width dimension of the feature map.
[0024] Furthermore, step S12 specifically includes the following steps:
[0025] Step S121: Expand the feature graph extracted by the convolutional neural network to obtain a feature vector F1, which is then input into the first multi-label classification model to predict the classification Q = {q1, q2, ..., q n}, where n is the number of image categories;
[0026] Step S122: Class activation map M corresponding to category i i Calculate using the following formula:
[0027]
[0028] Where i∈{1,2,...,n}, c is the dimension of the feature vector; W i,k represents the weight parameter, X k Represents the feature map vector of the k-th channel.
[0029] Furthermore, step S13 specifically includes the following steps:
[0030] Step S131: The hybrid dictionary is a dynamically updated structure. The class activation maps corresponding to the labels predicted to be true are selected, and the hybrid dictionary is updated according to the class activation maps corresponding to these labels. The hybrid dictionary is updated by the following formula:
[0031] D π =BN(D π +λ×BN(M π )
[0032] Among them, π represents the category whose classification label is true, λ represents the update rate, BN represents batch normalization, and M π Class activation map indicating that the classification label is true;
[0033] Step S132: The feature map extracted by the convolutional neural network is enhanced to obtain an enhanced feature map, and the enhanced feature map is calculated by the following formula:
[0034] X e =X+X☉Conv1×1(D π )
[0035] Among them, X represents the feature map extracted by the convolutional neural network, ⊙ represents the matrix dot product, and Conv1×1 represents one-dimensional convolution.
[0036] Furthermore, step S14 specifically includes the following steps:
[0037] Step S141: Input the enhanced feature map into the second multi-label classification model to obtain a more accurate classification prediction result
[0038] Step S142: Expand the enhanced feature map obtained above and input it into the fully connected network to obtain the final 2D feature F c ;
[0039] Step S143: Train two multi-label classification models together in the entire network. The two models share weights. The first multi-label classification model is responsible for generating initial classification results and class activation maps, and the second multi-label classification model is responsible for more accurate classification. The multi-label classification loss L is used. c The calculation is as follows:
[0040]
[0041] in represents the true category of the image, q i represents the output of the first multi-label classification model, Represents the output of the second multi-label classification model.
[0042] Furthermore, step S2 specifically includes the following steps:
[0043] Step S21: embedding the point cloud coordinates of the scene into the new feature space through the first linear layer;
[0044] Step S22: The embedded features are input into the stacked attention module to learn the local feature representation of each point; the attention adopts the self-attention mechanism to calculate the corresponding Q, V, K, and then calculate the attention weight A' according to Q and V through the matrix dot product; then the weight is normalized, and the output feature of the attention module is the weighted sum of the corresponding attention weight A' and V;
[0045] (Q,K,V)=F in *(W q ,W k ,W v )
[0046] Where W q ,W k ,W v Represents the learned linear transformation weight matrix, F in Represents the characteristics of the input;
[0047] The attention weight A′ is calculated as follows:
[0048] A=(α) i，j =Q*K T
[0049]
[0050] (α") i，j =softmax((α′) i，j )
[0051] A′=(α") i，j
[0052] Among them, (α) i,j represents the unnormalized attention weight parameter, i represents the row of the weight matrix, j represents the column of the weight matrix, and K T represents the transpose of K, d a represents the dimension of Q and K vectors, (α′) i,j represents the unnormalized attention weight parameter divided by the feature dimension, (α″) i,j represents the normalized attention weight, and softmax represents a normalization function;
[0053] Output feature F out Calculated by the following formula:
[0054] F out=A′*V
[0055] Step S23: Output feature F out The output feature is input to the second linear layer, and the output feature is subjected to the global maximum pooling layer to obtain the global feature representation of the point cloud, that is, the extracted 3D feature F v .
[0056] Furthermore, step S3 specifically includes the following steps:
[0057] Step S31: 2D feature F c and 3D feature F v Add them together to form a comprehensive feature, calculate the local context and global context for this comprehensive feature respectively, calculate the fusion weight according to the local context and the global context, use the fusion weight to fuse the 2D feature and the 3D feature, and perform soft selection or weighted average between the 2D feature and the 3D feature;
[0058] Step S32: Perform feature interaction between the 2D features and the 3D features, exchange rich information between the 2D features and the 3D features, and form interactive features.
[0059] Furthermore, step S4 specifically includes the following steps:
[0060] S41, concatenating the interactive features and the fusion features, and then inputting them into a fully connected neural network to predict posture parameters, which include seven parameters, the first three of which represent translation parameters, and the last four represent rotation parameters;
[0061] S42. Calculate translation loss L t :
[0062] L t =SmoothL1(t gt -t pred )
[0063] where t gt is the real translation parameter, t pred is the predicted translation parameter, SmoothL1 represents the smooth L1 loss, specifically
[0064] Calculate the rotation loss L r :
[0065] L r =D a (q gt ,q pred )
[0066] Where D a is the angular distance between the two quaternions, q gtis the actual rotation parameter, q pred is the predicted rotation parameter;
[0067] Combined translation loss L t , rotation loss L r and the multi-label classification loss L c , obtain the final loss L of the model, and train the model by minimizing the final loss L through the stochastic gradient descent method;
[0068] L=a*L r +b*L t +L c
[0069] Where a represents the weight of the rotation loss, and b represents the weight of the translation loss;
[0070] According to the predicted pose parameters, the position of the camera corresponding to the 2D image is obtained to achieve 2D / 3D alignment.
[0071] Compared with the prior art, the present invention has the following beneficial technical effects:
[0072] 1) When extracting 2D features, the influence of confounding factors on the features is eliminated through causal intervention methods to obtain causal features with stronger expressive power. When extracting 3D features, the attention mechanism is used to extract effective global feature representations.
[0073] 2) In terms of 2D feature and 3D feature matching, a feature interaction and fusion module is designed to realize the interaction and fusion of 2D features and 3D features, so that 2D features and 3D features can perceive each other, thereby strengthening their respective features and improving the accuracy of model prediction of pose parameters and 2D / 3D registration. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0075] Figure 1 is an overall flow chart of the 2D / 3D registration method of the present invention;
[0076] Figure 2 A network structure diagram for image causal feature extraction according to the present invention;
[0077] Figure 3 It is a flow chart of 3D feature extraction of the present invention;
[0078] Figure 4A network structure diagram showing the interaction and integration of the features of the present invention; DETAILED DESCRIPTION
[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0080] The present invention aims to propose a 2D / 3D registration method based on causal inference, which can make up for the defect that the image features extracted by the existing method are affected by irrelevant factors such as background, so that it can better complete the 2D / 3D registration task in various scenarios. The overall implementation process is as follows Figure 1 As shown in the figure, first, the image to be registered is subjected to feature extraction through the image causal feature extraction network, and a class activation map is formed through the multi-label classification network. According to the prediction result, all context information maps (mixed dictionaries) of each class are continuously stored through the activation map of the class with the highest cumulative probability, and all contexts of each class are projected as attention onto the feature map of the last convolutional layer to generate an enhanced feature map; secondly, the 3D point cloud is subjected to 3D feature extraction through the attention-based point cloud feature extraction network to extract the geometric features of the 3D scene; then, the extracted 2D features and 3D features are calculated through the feature interaction and fusion network to calculate the fusion features and interaction features; finally, the interaction features and fusion features are spliced as the final feature vector, which is then input into the fully connected neural network to predict the pose parameters and realize 2D / 3D registration.
[0081] Example:
[0082] The 2D / 3D registration method based on causal inference in this embodiment takes the 3D point cloud of the input image and the scene as an example to explain the scheme of the present invention in detail. The specific implementation process is as follows:
[0083] Step S1, perform image causal feature extraction on the input image to obtain 2D features; the network for image causal feature extraction consists of a convolutional neural network, a first multi-label classification model, a hybrid dictionary, a second multi-label classification model and a fully connected network, such as Figure 2 As shown;
[0084] S11. The image to be registered is passed through a convolutional neural network for feature extraction: The convolutional neural network uses a network based on FCN (Fully Convolutional Network) with multiple non-local blocks inserted at low and high levels. It acts as a feature extractor, taking the RGB image as input and generating a high-level position-aware feature map:
[0085] S111. Build a convolutional neural network using FCN-based networks with multiple non-local blocks inserted at low and high levels;
[0086] S112, input the image to be registered into the constructed convolutional neural network to obtain the corresponding feature map X∈R c×h×w , where c represents the number of image classification categories, h represents the height dimension of the feature map, and w represents the width dimension of the feature map;
[0087] S12, forming a class activation map based on the first multi-label classification model: inputting the feature map extracted by the convolutional neural network into the first multi-label classification model, and multiplying the weight of the classifier of the first multi-label classification model and the feature map to obtain a class activation map:
[0088] S121, expand the feature map extracted by the convolutional neural network to obtain a feature vector F1, and then input it into the first multi-label classification model to predict the classification Q = {q1, q2, ..., q n}, the first multi-label classification model has a classifier with a fully connected layer, the classifier adopts a Sigmoid function, and the first multi-label classification model is responsible for generating an initial classification result and a class activation map.
[0089] S122, the weight matrix of the classifier is represented by W∈R n×c , where n is the number of image categories and c is the dimension of the feature vector.
[0090] Class Activation Maps (CAMs) M corresponding to category i i Calculate using the following formula:
[0091]
[0092] where i∈{1,2,...,n}, n is the number of image categories; W i,k represents the weight parameter, X k Represents the feature map vector of the k-th channel.
[0093] S13. After the first multi-label classification model outputs the corresponding label, all context information maps of each class are continuously stored according to the label result by accumulating the activation map of the class with the highest probability. This context information is stored in a structure called a mixed dictionary. Then, all the context of each class is projected as attention onto the feature map of the last convolutional layer to generate an enhanced feature map:
[0094] S131, a mixed dictionary D∈R is maintained in the network n×h×w , D iRepresenting the context of the i-th category image, the first multi-label classification model can output multiple classification labels corresponding to each image, and calculate the class activation map corresponding to category i. Based on these classification labels, the class activation map corresponding to the predicted true label is selected, and the mixed dictionary is updated according to the class activation map corresponding to these labels. The mixed dictionary is a dynamically updated structure that continuously accumulates and updates the context information of each category. Finally, the mixed dictionary is projected onto the feature map as attention. This projection process is actually a process of feature fusion and enhancement. It uses the context information in the mixed dictionary to optimize the feature map, making it more focused on those areas that have a decisive influence on the classification results. After such processing, the network can eliminate the influence of confounding factors and enhance the expressive power of the feature map.
[0095] The mixed dictionary is updated as follows:
[0096] D π =BN(D π +λ×BN(M π )
[0097] Among them, π represents the category whose classification label is true, λ represents the update rate, BN represents batch normalization, and M π CAM (Class Activation Mapping) indicates that the classification label is true.
[0098] S132, enhancing the feature map extracted by the convolutional neural network to obtain an enhanced feature map, and the enhanced feature map is calculated by the following formula:
[0099] X e =X+X☉Conv1×1(D π )
[0100] Among them, X represents the feature map extracted by the convolutional neural network, ⊙ represents the matrix dot product, and Conv1×1 represents one-dimensional convolution.
[0101] S14, input the enhanced feature map into the second multi-label classification model to produce a more accurate prediction result, and expand the enhanced feature map and input it into the fully connected network to obtain the final 2D feature. During the training process, two multi-label classification models are used to calculate the loss. The loss function is multi-classification cross entropy, which is part of the loss function of the entire model and participates in the training of the entire network:
[0102] S141. Input the enhanced feature map into the second multi-label classification model to obtain a more accurate classification prediction result.
[0103] S142, the enhanced feature map obtained above is expanded and input into the fully connected network to obtain the final 2D feature Fc , the dimension is 1024;
[0104] S143. Train two multi-label classification models together in the entire network. The two models share weights. The first multi-label classification model is responsible for generating initial classification results and class activation maps, and the second multi-label classification model is responsible for more accurate classification. The multi-label classification loss is as follows:
[0105]
[0106] in represents the true category of the image, q i represents the output of the first multi-label classification model, Represents the output of the second multi-label classification model.
[0107] S2, extract 3D features from the 3D point cloud of the scene through an attention-based point cloud feature extraction network to obtain 3D features. The attention-based point cloud feature extraction network includes a first linear layer, an attention module, a second linear layer, and a global maximum pooling layer, such as Figure 3 As shown:
[0108] S21, embedding the point cloud coordinates of the scene into the new feature space through the first linear layer;
[0109] S22. The embedded features are input into the stacked attention module to learn the local feature representation of each point. The attention adopts the self-attention mechanism to calculate the corresponding Q (Query), V (Value), K (Key), and then calculate the attention weight A' according to Q and V through the matrix dot product. The weights are then normalized, and the output feature of the attention module is the weighted sum of the corresponding attention weights A' and V.
[0110] (Q,K,V)=F in *(W q ,W k , W v )
[0111] Where W q ,W k ,W v represents the linear transformation weight matrix that can be learned, F in Represents the characteristics of the input.
[0112] The attention weight A′ is calculated as follows:
[0113] A=(α) i，j =Q*K T
[0114]
[0115] (α") i，j =softmax((α′) i，j )
[0116] A′=(α") i，j
[0117] Among them, (α) i,j represents the unnormalized attention weight parameter, i represents the row of the weight matrix, j represents the column of the weight matrix, and K T represents the transpose of K, d a represents the dimension of Q and K vectors, (α′) i,j represents the unnormalized attention weight parameter divided by the feature dimension, (α″) i,j represents the normalized attention weight, and softmax represents a normalization function.
[0118] Output feature F out Calculated by the following formula:
[0119] F out =A′*V
[0120] S23, output feature F out The output feature is input to the second linear layer, and the output feature is subjected to the global maximum pooling layer to obtain the global feature representation of the point cloud, that is, the extracted 3D feature F v , the feature dimension is 1024, which is used to represent the overall geometric features of this point cloud.
[0121] S3, perform feature interaction and fusion of 2D features and 3D features to obtain interaction features and fusion features; Figure 4 As shown in the figure, the feature interaction and fusion network structure includes three one-dimensional convolutions, two linear rectification units, a global average pooling layer, and three fully connected networks.
[0122] The interaction and fusion of 2D features and 3D features can fully explore the potential correlation between the two features, organically combine them, and thus enhance their respective expression capabilities. After being processed by the feature interaction and fusion module, 2D features and 3D features can not only retain their original information, but also complement each other to form a richer and more comprehensive feature representation, thereby improving the accuracy of pose parameter prediction.
[0123] S31, the 2D feature F c and 3D feature F vAdd them together to form a comprehensive feature, calculate the local context and global context for this comprehensive feature respectively, calculate the fusion weight according to the local context and the global context, use the fusion weight to fuse the 2D feature and the 3D feature, and perform soft selection or weighted average between the 2D feature and the 3D feature:
[0124] S311, for the extracted 2D feature F c and 3D feature F v First, these two features are added together to get V, and then local context calculation and global context calculation are performed on V respectively.
[0125] The local context is calculated through a one-dimensional convolution with channel dimension increase, a linear rectifier unit and a one-dimensional convolution with channel dimension reduction. The local context is calculated as follows:
[0126] L(V)=β(Conv1d2(δ(β(Conv1d1(V))))
[0127] Among them, δ represents the rectified linear unit (ReLU), β represents batch normalization, Conv1d1 represents the one-dimensional convolution of channel dimension increase, and Conv1d2 represents the one-dimensional convolution of channel dimension reduction.
[0128] The global context is calculated through a one-dimensional convolution, a linear rectifier unit and a global average pooling layer. The global context is calculated as:
[0129] G(V)=g(δ(β(Conv1d(V))))
[0130] Where g() represents global average pooling and Conv1d represents one-dimensional convolution.
[0131] S312, given the global context and the local context, the fusion weight Z can be calculated by the following formula:
[0132] Z=σ(L(V)+G(V))
[0133] Where σ represents the Sigmoid function.
[0134] S313, the fusion feature is calculated by the following formula:
[0135]
[0136] Where V′ represents the fusion feature, and the fusion weight Z ranges from 0 to 1. In this way, the network can perform soft selection or weighted averaging between 2D features and 3D features.
[0137] S32, perform feature interaction between 2D features and 3D features, exchange rich information between 2D features and 3D features, and form interactive features:
[0138] S321. For a given 3D feature F v Through a fully connected layer, linear and nonlinear operations are performed to capture the complex relationship between features, extract more advanced feature representations, and then the output features are combined with the 2D features F c Splice them together, so that the 2D feature integrates the 3D feature information, and the 2D feature is updated as follows:
[0139] F′ c =concate(MLP(F v ),F c )
[0140] Among them, concate represents concatenation operation, and MLP represents fully connected network.
[0141] S322. For a given 2D feature F c Through a fully connected layer, linear and nonlinear operations are performed to capture the complex relationship between features, extract more advanced feature representations, and then the output features are combined with the 3D features F v Splice them together, so that the 3D feature integrates the 2D feature information, and the 3D feature is updated as follows:
[0142] F′ v =concate(MLP(F c ),F v )
[0143] S323, finally update the 2D feature F′ c and the updated 3D feature F′ v Add together to get the interaction feature F′:
[0144] F′=F′ c +F′ v
[0145] S4. Perform pose prediction based on interactive features and fusion features to achieve 2D / 3D alignment.
[0146] S41. Concatenate the interactive features and the fusion features as the final feature vector with a dimension of 3072, and then input it into the fully connected neural network to predict the posture parameters. The posture parameters include seven parameters, the first three of which represent translation parameters, and the last four represent rotation parameters.
[0147] S42. Calculate translation loss L t :
[0148] L t =SmoothL1(t gt -t pred )
[0149] where t gt is the real translation parameter, t pred is the predicted translation parameter, SmoothL1 represents the smooth L1 loss, specifically
[0150] For the rotation loss, in order to avoid the double coverage problem of quaternions, the angular distance is used to evaluate the difference between the two quaternions. The rotation loss L r The calculation is as follows:
[0151] L r =D a (q gt ,q pred )
[0152] Where D a is the angular distance between the two quaternions, q gt is the actual rotation parameter, q pred is the predicted rotation parameter.
[0153] Combined translation loss L t , rotation loss L r and the multi-label classification loss L c , obtain the final loss L of the model, and train the model by minimizing the final loss L through the stochastic gradient descent method;
[0154] L=a*L r +b*L t +L c
[0155] Where a represents the weight of the rotation loss and b represents the weight of the translation loss.
[0156] According to the predicted pose parameters, the position of the camera corresponding to the 2D image can be obtained to achieve 2D / 3D registration.
[0157] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A 2D / 3D registration method based on causal inference, characterized in that: The steps include: Step S1: performing image causal feature extraction on the input image to obtain 2D features; the network for image causal feature extraction consists of a convolutional neural network, a first multi-label classification model, a hybrid dictionary, a second multi-label classification model and a fully connected network; Step S2: extracting 3D features of the 3D point cloud of the scene through an attention-based point cloud feature extraction network to obtain 3D features. The attention-based point cloud feature extraction network includes a first linear layer, an attention module, a second linear layer and a global maximum pooling layer; Step S3: Interact and fuse the 2D features with the 3D features to obtain interactive features and fused features; The feature interaction and fusion network structure includes three one-dimensional convolutions, two linear rectification units, a global average pooling layer, and three fully connected networks; Step S4: perform pose prediction based on the interactive features and fusion features to achieve 2D / 3D alignment.
2. The 2D / 3D registration method based on causal inference according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S11: Extract features of the image to be registered through a convolutional neural network: Step S12: Forming a class activation map based on the first multi-label classification model: Inputting the feature map extracted by the convolutional neural network into the first multi-label classification model, and multiplying the weight of the first multi-label classification model classifier and the feature map to obtain a class activation map: Step S13: After the first multi-label classification model outputs the corresponding label, all context information of each class is continuously stored according to the label result through the activation map of the class with the highest cumulative probability, and the context information is stored in the mixed dictionary; then, all the context of each class is projected as attention onto the feature map of the last convolutional layer to generate an enhanced feature map; Step S14: Input the enhanced feature map into the second multi-label classification model to generate a more accurate prediction result. At the same time, expand the enhanced feature map and input it into the fully connected network to obtain the final 2D feature.
3. The 2D / 3D registration method based on causal inference according to claim 2, characterized in that: Step S11 specifically includes the following steps: Step S111: building a convolutional neural network, using a FCN-based network with multiple non-local blocks inserted in low and high layers; Step S112: Input the image to be registered into the constructed convolutional neural network to obtain the corresponding feature map X∈R c×h×w , where c represents the number of image classification categories, h represents the height dimension of the feature map, and w represents the width dimension of the feature map.
4. The 2D / 3D registration method based on causal inference according to claim 3, characterized in that: Step S12 specifically includes the following steps: Step S121: Expand the feature graph extracted by the convolutional neural network to obtain a feature vector F1, which is then input into the first multi-label classification model to predict the classification Q = {q1, q2, ..., q n }, where n is the number of image categories; Step S122: Class activation map M corresponding to category i i Calculate using the following formula: Where i∈{1,2,...,n}, c is the dimension of the feature vector; W i,k represents the weight parameter, X k Represents the feature map vector of the k-th channel.
5. The 2D / 3D registration method based on causal inference according to claim 4, characterized in that: Step S13 specifically includes the following steps: Step S131: The hybrid dictionary is a dynamically updated structure. The class activation maps corresponding to the labels predicted to be true are selected, and the hybrid dictionary is updated according to the class activation maps corresponding to these labels. The hybrid dictionary is updated by the following formula: D π =BN(D π +λ×BN(M π ) Among them, π represents the category whose classification label is true, λ represents the update rate, BN represents batch normalization, and M π Class activation map indicating that the classification label is true; Step S132: The feature map extracted by the convolutional neural network is enhanced to obtain an enhanced feature map, and the enhanced feature map is calculated by the following formula: X e =X+X⊙Conv1×1(D π ) Among them, X represents the feature map extracted by the convolutional neural network, ⊙ represents the matrix dot product, and Conv1×1 represents one-dimensional convolution.
6. The 2D / 3D registration method based on causal inference according to claim 5, characterized in that: Step S14 specifically includes the following steps: Step S141: Input the enhanced feature map into the second multi-label classification model to obtain a more accurate classification prediction result Step S142: Expand the enhanced feature map obtained above and input it into the fully connected network to obtain the final 2D feature F c ; Step S143: Train two multi-label classification models together in the entire network. The two models share weights. The first multi-label classification model is responsible for generating initial classification results and class activation maps, and the second multi-label classification model is responsible for more accurate classification. The multi-label classification loss L is used. c The calculation is as follows: in represents the true category of the image, q i represents the output of the first multi-label classification model, Represents the output of the second multi-label classification model.
7. The 2D / 3D registration method based on causal inference according to claim 6, characterized in that: Step S2 specifically includes the following steps: Step S21: embedding the point cloud coordinates of the scene into the new feature space through the first linear layer; Step S22: The embedded features are input into the stacked attention module to learn the local feature representation of each point; The attention adopts the self-attention mechanism to calculate the corresponding Q, V, K, and then calculate the attention weight A' according to Q and V through the matrix dot product; then the weight is normalized, and the output feature of the attention module is the weighted sum of the corresponding attention weight A' and V; (Q,K,V)=F in *(W q ,W k ,W v ) Where W q ,W k ,W v Represents the learned linear transformation weight matrix, F in Represents the characteristics of the input; The attention weight A′ is calculated as follows: A=(a) i,j =Q*K T (a") i,j =softmax((α′) i,j ) A′＝(α″) i,j Among them, (α) i,j represents the unnormalized attention weight parameter, i represents the row of the weight matrix, j represents the column of the weight matrix, and K T represents the transpose of K, d a represents the dimension of Q and K vectors, (α′) i,j represents the unnormalized attention weight parameter divided by the feature dimension, (α″) i,j represents the normalized attention weight, and softmax represents a normalization function; Output feature F out Calculated by the following formula: F out =A′*V Step S23: Output feature F out The output feature is input to the second linear layer, and the output feature is subjected to the global maximum pooling layer to obtain the global feature representation of the point cloud, that is, the extracted 3D feature F v .
8. The 2D / 3D registration method based on causal inference according to claim 7, characterized in that: Step S3 specifically includes the following steps: Step S31: 2D feature F c and 3D feature F v Add them together to form a comprehensive feature, calculate the local context and global context for this comprehensive feature respectively, calculate the fusion weight according to the local context and the global context, use the fusion weight to fuse the 2D feature and the 3D feature, and perform soft selection or weighted average between the 2D feature and the 3D feature; Step S32: Perform feature interaction between the 2D features and the 3D features, exchange rich information between the 2D features and the 3D features, and form interactive features.
9. The 2D / 3D registration method based on causal inference according to claim 8, characterized in that: Step S4 specifically includes the following steps: Step S41: concatenate the interactive features and the fusion features, and then input them into a fully connected neural network to predict the pose parameters. The pose parameters include seven parameters, the first three of which represent translation parameters, and the last four represent rotation parameters. Step S42: Calculate the translation loss L t : L t =SmoothL1(t gt -t pred ) where t gt is the real translation parameter, t pred is the predicted translation parameter, SmoothL1 represents the smooth L1 loss, specifically Calculate the rotation loss L r : L r =D a (q gt ,q pred ) Where D a is the angular distance between the two quaternions, q gt is the actual rotation parameter, q pred is the predicted rotation parameter; Combined translation loss L t , rotation loss L r and the multi-label classification loss L c , obtain the final loss L of the model, and train the model by minimizing the final loss L through the stochastic gradient descent method; L=a*L r +b*L t +L c Where a represents the weight of the rotation loss, and b represents the weight of the translation loss; According to the predicted pose parameters, the position of the camera corresponding to the 2D image is obtained to achieve 2D / 3D alignment.