Cardiovascular cross-domain model registration method based on geometric conjugate learning

Abstract

The invention relates to the technical field of machine learning, in particular to a cardiovascular cross-domain model registration method based on geometric conjugate learning. The method comprises the following steps: synchronously acquiring a two-dimensional ultrasonic image and a three-dimensional medical image of a patient; performing data preprocessing on the sample data; constructing a training geometric conjugate learning model according to the sample data to obtain a corresponding relation soft mapping matrix between two-dimensional ultrasonic image pixel points and three-dimensional medical image vertexes; acquiring a two-dimensional ultrasonic image and a three-dimensional medical image of a patient in real time, and inputting the two-dimensional ultrasonic image and the three-dimensional medical image into the geometric conjugate learning model to register the high-precision cardiovascular three- And dynamic deformation field prediction is adopted to compensate for heart periodic motion to ensure synchronization of the cardiovascular three-dimensional model and the physiological state. According to the scheme, through combination of deep learning and geometric modeling, the cross-domain corresponding relation between the two-dimensional ultrasonic image and the three-dimensional human body scanning data is established, clear, visual and real-time three-dimensional operation information is provided for doctors, and the diagnosis accuracy and the operation success rate are improved.

Description

Technical Field

[0001] This application relates to the technical field of machine learning, and particularly to a cardiovascular cross-domain model registration method based on geometric conjugate learning. Background Art

[0002] In the medical field, especially in the surgical treatment of cardiovascular diseases, traditional surgical methods mainly rely on two-dimensional ultrasound images for diagnosis and surgical guidance. Two-dimensional ultrasound images can provide real-time internal organ images to help doctors observe the structure and function of the heart and blood vessels. In cardiovascular surgery, doctors usually need to associate three-dimensional vascular models from two-dimensional ultrasound images to better understand the structure and location of the lesion site. However, this technology has obvious limitations, and the main disadvantages include:

[0003] 1. Narrow field of view and low resolution. The traditional two-dimensional ultrasound image has a limited field of view and low resolution, making it difficult to provide sufficient detailed information. Especially in complex anatomical structures, doctors may not be able to clearly observe the specific situation of the lesion site.

[0004] 2. Lack of three-dimensional spatial information. Two-dimensional images cannot provide three-dimensional spatial information, and doctors can only associate three-dimensional structures through experience and imagination, which increases the uncertainty of diagnosis and surgery.

[0005] 3. Dependence on clinical experience. During the operation, doctors usually need to rely on clinical experience to judge the relative position of the ultrasound probe. This method is easily affected by subjective factors, resulting in misjudgment and increased surgical risks.

[0006] 4. Insufficient real-time performance. Traditional two-dimensional ultrasound images cannot provide real-time three-dimensional spatial information, making it difficult for navigation and positioning during the operation, and affecting the accuracy and success rate of the surgery.

[0007] 5. High risk of diagnostic errors. Due to the limitations of two-dimensional images, doctors are prone to errors during the diagnosis process. Especially in complex cases, the possibility of misdiagnosis is relatively high, affecting the treatment effect and the prognosis of patients.

[0008] Therefore, how to establish a cross-domain correspondence between two-dimensional ultrasound images and three-dimensional human body scan data to provide clear, intuitive, and real-time three-dimensional cardiovascular surgery information during cardiovascular surgery is one of the problems that urgently need to be solved in the current information-assisted medical field. Summary of the Invention

[0009] One of the objectives of the present invention is to provide a cardiovascular cross-domain model registration method based on geometric conjugate learning. By combining deep learning with geometric modeling, a cross-domain correspondence relationship between two-dimensional ultrasound images and three-dimensional human body scan data is established, so as to provide doctors with clear, intuitive, and real-time three-dimensional surgical guidance information, improving the diagnostic accuracy and surgical success rate.

[0010] To solve the above technical problems, the present invention provides a cardiovascular cross-domain model registration method based on geometric conjugate learning, including the following steps:

[0011] S1. Collect sample data of the patient; the sample data includes synchronously collected two-dimensional ultrasound images and three-dimensional medical images;

[0012] S2. Perform data preprocessing on the sample data;

[0013] S3. Construct a training geometric conjugate learning model according to the sample data to obtain a soft mapping matrix of the correspondence relationship between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image;

[0014] S4. Real-time collect the two-dimensional ultrasound image and three-dimensional medical image of the patient, input them into the geometric conjugate learning model, and correspond each pixel point of the real-time two-dimensional ultrasound image to the vertices of the real-time three-dimensional medical image according to the soft mapping matrix to register a high-precision cardiovascular three-dimensional model;

[0015] S5. Use a dynamic deformation field to predict and compensate for the periodic motion of the heart to ensure that the cardiovascular three-dimensional model is synchronized with the physiological state.

[0016] Further, the data preprocessing of the sample data includes the following steps:

[0017] S21. Perform anisotropic filtering or wavelet transform denoising on the two-dimensional ultrasound image to enhance the image contrast;

[0018] S22. Perform non-local means denoising on the three-dimensional medical image;

[0019] S23. Use rigid registration based on feature point matching to align the two-dimensional ultrasound image and the three-dimensional medical image;

[0020] S24. Use a non-rigid registration method to compensate for the cardiac motion artifacts of the two-dimensional ultrasound image;

[0021] S25. Use a U-Net model to automatically segment the cardiovascular structure of the sample data and label key anatomical landmark points.

[0022] In the technical solution of this application, after collecting a multi-modal medical image dataset, preprocessing operations such as denoising, registration, compensation, and segmentation are performed on the sample data, which can ensure the consistency and usability of data quality and improve the training accuracy of the geometric conjugate learning model.

[0023] Further, the step of constructing a training geometric conjugate learning model based on the sample data to obtain the soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image includes the following steps:

[0024] S31. Extract multi-scale two-dimensional image features of the two-dimensional ultrasound image based on the Transformer model;

[0025] S32. Extract three-dimensional model features of the three-dimensional medical image based on the feature pyramid and the 3D convolution module;

[0026] S33. Construct a feature similarity calculation module based on the geometric conjugate learning mechanism to calculate the similarity score matrix between the two-dimensional image features and the three-dimensional model features;

[0027] S34. Construct a soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image based on the cross-domain feature mapping mechanism according to the similarity score matrix.

[0028] In terms of feature processing, the technical solution of this application adopts a method that combines a multi-scale feature pyramid, 3D convolution, and an attention mechanism to extract cross-domain feature descriptors that fuse two-dimensional image and three-dimensional model geometric information from data in different domains, improving the interaction ability between feature extractions in different domains, strengthening the connection between similar features in different domains, and providing a guarantee for subsequent clustering of similar features.

[0029] Further, the step of constructing a feature similarity calculation module based on the geometric conjugate learning mechanism to calculate the similarity score matrix between the two-dimensional image features and the three-dimensional model features includes the following steps:

[0030] S331. Use the loss function L core to enhance the distribution consistency of similar features between the two-dimensional image features and the three-dimensional model features. The loss function L core has the following calculation formula: where r i,I represents the feature vector of the i-th pixel point in the two-dimensional image I, r j,S represents the feature vector of the j-th point in the three-dimensional model S, ||r i,I -r j,S ||2 represents the distance between the two vectors, and class I represents r i,IThe category label, class S represents r j,S The category label, indic() is an indicator function for calculating whether the category labels are consistent. If they are consistent, the output is 0; otherwise, it is 1;

[0031] S332. Construct the conjugate product graph G(v of the directed graph M corresponding to the contour feature in the two-dimensional image feature and the directed graph N corresponding to the contour feature of the three-dimensional model feature * , ε * ), and the conjugate loss function used to construct the conjugate product graph is: Among them, represents the distance correction weight of the q-th point in the directed graph M, represents the distance correction weight of the b-th point in the directed graph N, and the Softmax() function is the normalized exponential function;

[0032] S333. Use the Dijkstra method and the Euclidean distance to find the conjugate shortest distance d between the corresponding elements of M and N in the conjugate product graph G(v * , ε * );

[0033] S334. Calculate the similarity score matrix between the two domain feature descriptors according to the Euclidean distance and the conjugate shortest distance d. The calculation formula is: Among them, Sim is the similarity score matrix, d i,j represents the conjugate shortest distance between the feature vector of the i-th pixel point in the two-dimensional image I and the feature vector of the j-th point in the three-dimensional model S in the conjugate product graph, ||r i,I - r j,S ||2 represents the Euclidean distance between two vectors.

[0034] In the similarity calculation process between the two-dimensional image feature and the three-dimensional model feature, first, the technical solution of the present application enhances the distribution consistency of similar features in different domains through the loss function, considers the spatial proximity and category correlation of key points, reduces the differences between similar features in different domains, and at the same time does not overly penalize incorrect point-to-point correspondences, improving the accuracy of feature similarity calculation. Second, by constructing the conjugate product graph of the directed graph M corresponding to the contour feature in the two-dimensional image feature and the directed graph N corresponding to the three-dimensional model feature, and defining a distance correction weight w and a conjugate loss function L for the construction process of the conjugate product graph rect for correction, the calculation of the conjugate shortest distance d is supervised and optimized.

[0035] Further, the cross - domain feature mapping mechanism constructs a soft mapping matrix of the correspondence between the pixel points of the two - dimensional ultrasound image and the vertices of the three - dimensional medical image according to the similarity score matrix, including the following steps:

[0036] S341. Convert the similarity score matrix Sim into a soft mapping matrix Soft, and the calculation formula is:

[0037] where represents element - wise multiplication, dim = 0 represents the rows in the two - dimensional matrix, dim = 1 represents the columns in the two - dimensional matrix, and the Softmax() function is the normalized exponential function;

[0038] S342. Optimize and converge the soft mapping matrix through the corresponding reconstruction loss function, and the calculation formula of the corresponding reconstruction loss function is: where represents the Frobenius norm, represents the two - dimensional image matrix after being adjusted and reconstructed through geometric conjugate learning, represents the three - dimensional model matrix after being adjusted and reconstructed through geometric conjugate learning.

[0039] In the generation process of the soft mapping matrix of the correspondence between the pixel points of the two - dimensional ultrasound image and the vertices of the three - dimensional medical image, the solution of this application continuously repeats and iterates through the reconstruction loss function until the network converges to optimize the soft mapping matrix.

[0040] Further, the dynamic deformation field prediction is based on the LSTM neural network or the Transformer deep learning model.

[0041] Further, the cross - domain model registration method for cardiovascular based on geometric conjugate learning further includes the following steps:

[0042] S6. Identify the cardiovascular structures in the real - time two - dimensional ultrasound image and mark the corresponding three - dimensional cardiovascular structure positions in the registered three - dimensional cardiovascular model.

[0043] Different from the prior art, the beneficial effects of the technical solution of the present invention are:

[0044] 1. After collecting the multi - modal medical image data set, performing pre - processing operations such as denoising, registration, compensation, and segmentation on the sample data can ensure the consistency and usability of the data quality and improve the training accuracy of the geometric conjugate learning model.

[0045] 2. In feature processing, a method combining a multi-scale feature pyramid, 3D convolution, and an attention mechanism is adopted to extract cross-domain feature descriptors that fuse the geometric information of 2D images and 3D models from data in different domains, enhancing the interaction ability between feature extractions in different domains, strengthening the connection between similar features in different domains, and providing guarantee for subsequent clustering of similar features.

[0046] 3. In the similarity calculation process between the 2D image features and the 3D model features, first, a loss function is used to enhance the distribution consistency of similar features in different domains, taking into account the spatial proximity and category correlation of key points, reducing the differences between similar features in different domains, and not overly penalizing incorrect point-to-point correspondences, thereby improving the accuracy of feature similarity calculation. Second, by constructing the conjugate product graph of the directed graph M corresponding to the contour features in the 2D image features and the directed graph N corresponding to the 3D model features, and defining a distance correction weight w and a conjugate loss function L for the construction process of the conjugate product graph rect for correction, it realizes the supervision and optimization of calculating the conjugate shortest distance d.

[0047] 4. In the generation process of the soft mapping matrix for the correspondence between 2D ultrasound image pixel points and 3D medical image vertices, the soft mapping matrix is optimized by continuously repeating and iterating through the reconstruction loss function until the network converges. Description of the Drawings

[0048] Figure 1 is the flowchart of the steps of the cross-domain model registration method for cardiovascular based on geometric conjugate learning of the present invention.

[0049] Figure 2 is the flowchart of the steps for the present invention to perform data preprocessing on sample data.

[0050] Figure 3 is the flowchart of the steps for the present invention to construct a training geometric conjugate learning model to obtain a soft mapping matrix.

[0051] Figure 4 is the flowchart of the steps for the present invention to calculate the similarity score matrix between 2D image features and 3D model features.

[0052] Figure 5 is the flowchart of the steps for the present invention to construct a training geometric conjugate learning model to obtain a soft mapping matrix. Detailed Embodiment

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] As Figure 1 shown, it is a flowchart of the steps of the cardiovascular cross-domain model registration method based on geometric conjugate learning of the present invention, including the following steps:

[0055] S1. Collect sample data of the patient; the sample data includes synchronously collected two-dimensional ultrasound images and three-dimensional medical images.

[0056] Collect the two-dimensional ultrasound images and three-dimensional medical image data of the patient's cardiovascular system to construct a multi-modal medical image dataset. Among them, the two-dimensional ultrasound images can use a high-frequency ultrasound probe (such as a transesophageal echocardiogram probe) to collect dynamic cardiovascular images, covering the cross-sectional and longitudinal-sectional views of the heart and blood vessels. The three-dimensional medical images refer to the three-dimensional CT or MRI data of the patient. It should be noted that the two-dimensional ultrasound images and three-dimensional medical images must be a one-to-one corresponding data set collected at the same time respectively, ensuring the time synchronization and spatial consistency between the two, so as to perform point-to-point mapping correspondence later, which can be achieved through respiratory gating and electrocardiogram gating techniques.

[0057] S2. Perform data preprocessing on the sample data. As Figure 2 shown, it is a flowchart of the steps of the present invention for performing data preprocessing on the sample data, including the following steps:

[0058] S21. Perform anisotropic filtering or wavelet transform denoising on the two-dimensional ultrasound image to enhance the image contrast.

[0059] S22. Perform non-local means denoising on the three-dimensional medical image.

[0060] By performing denoising and enhancement processing on the collected two-dimensional ultrasound images and three-dimensional medical images, image denoising can improve the image quality and clarity, which is of great significance for subsequent image processing tasks such as image segmentation, feature extraction, and target recognition.

[0061] S23. Use rigid registration based on feature point matching to align the two-dimensional ultrasound image with the three-dimensional medical image.

[0062] Existing conventional rigid registration methods based on feature point matching (such as SIFT or SURF feature points) can be used to perform preliminary alignment matching on the two-dimensional ultrasound image and the three-dimensional medical image.

[0063] S24. Use a non-rigid registration method to compensate for cardiac motion artifacts in the two-dimensional ultrasound image.

[0064] For cardiac dynamic deformation, a non-rigid registration method (such as B-spline registration method or optical flow method) can be used to compensate for cardiac motion artifacts in the two-dimensional ultrasound image.

[0065] S25. Use a U-Net model to automatically segment the cardiovascular structure of the sample data and label key anatomical landmark points.

[0066] Use an existing conventional U-Net convolutional neural network segmentation model to automatically segment the cardiovascular structure (such as ventricles, vessel walls, plaque areas) of the two-dimensional ultrasound image and label key anatomical landmark points.

[0067] In the technical solution of this application, after collecting a multi-modal medical image dataset, preprocessing operations such as denoising, registration, compensation, and segmentation are performed on the sample data, which can ensure the consistency and usability of data quality and improve the training accuracy of the geometric conjugate learning model.

[0068] S3. Construct a training geometric conjugate learning model based on the sample data to obtain a corresponding relationship soft mapping matrix between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image. As Figure 3 shown, it is a step flow chart of constructing a training geometric conjugate learning model to obtain a soft mapping matrix in the present invention, including the following steps:

[0069] S31. Extract multi-scale two-dimensional image features of the two-dimensional ultrasound image based on the Transformer model.

[0070] In the two-dimensional image feature extraction stage, as the number of network layers increases, some key features in the two-dimensional image will disappear, and the size of some feature maps will also change as the number of network layers increases. Therefore, the technical solution of the present invention combines multiple convolutional layers with skip connections to extract feature information of different scales, and uses pooling operations to keep the size of the feature maps extracted by each layer consistent. The feature information extracted by each layer is transmitted to each subsequent layer through residual connections, so as to fully retain the dense multi-scale key features in the image. At the same time, the features obtained by each layer will interact with the geometric feature information of the three-dimensional model through the Transformer, and the interacted features will also be used as the input of the subsequent feature extraction layer, so that the extracted features consider the geometric feature information of different domains at the same time.

[0071] In the feature extraction of two-dimensional image data, the refined features obtained by each feature extraction layer are transferred and transmitted using a feature transfer mechanism, which can retain the dense multi-scale key features in the image, prevent the loss of feature information, and construct a more accurate two-dimensional image feature descriptor.

[0072] S32. Extract the 3D model features of the three-dimensional medical image based on the feature pyramid and the 3D convolution module.

[0073] Input the three-dimensional medical image into multiple 3D convolution modules to extract geometric key features. The features extracted in each layer are passed to the upsampling feature refinement layer through skip connections and simultaneously serve as the input for the next 3D convolution module. The extracted feature data is subjected to upsampling feature refinement. The features extracted by different 3D convolution modules are unified into the same feature dimension using an MLP. The 3D feature information extracted in each layer is also processed by a Transformer to interact geometric information with the two-dimensional image feature data, reducing the information difference between data in different domains and constructing a 3D model feature descriptor with better cross-domain performance.

[0074] In terms of feature processing, the technical solution of this application adopts a method combining a multi-scale feature pyramid, 3D convolution, and an attention mechanism to extract a cross-domain feature descriptor that fuses two-dimensional image and three-dimensional model geometric information from data in different domains, enhancing the interaction ability between different-domain feature extractions, strengthening the connection between similar features in different domains, and providing guarantee for subsequent clustering of similar features.

[0075] S33. Construct a feature similarity calculation module based on the geometric conjugate learning mechanism to calculate the similarity score matrix between the two-dimensional image features and the three-dimensional model features. As Figure 4 shown, it is the flowchart of the steps for calculating the similarity score matrix between the two-dimensional image features and the three-dimensional model features of the present invention, including the following steps:

[0076] S331. Use the loss function L core to enhance the distribution consistency of similar features between the two-dimensional image features and the three-dimensional model features. The calculation formula of the loss function L core is: where r i,I represents the feature vector of the i-th pixel point in the two-dimensional image I, r j,S represents the feature vector of the j-th point in the three-dimensional model S, ||r i,I -r j,S ||2 represents the distance between two vectors, class I represents the class label of r i,I , class S represents the class label of r j,S , and indic() is an indicator function for calculating whether the class labels are the same. If they are the same, the output is 0; otherwise, it is 1.

[0077] Since the features extracted from different domains usually follow different distributions and there are significant differences, it is extremely difficult to directly match the feature information of different domains. Therefore, the solution of this application uses a loss function to enhance the distribution consistency of similar features in different domains. This loss function fully considers the spatial proximity and category correlation of key points, reduces the differences between similar features in different domains, and at the same time does not overly penalize incorrect point-to-point correspondences, improving the accuracy of feature similarity calculation.

[0078] S332. Construct the conjugate product graph G(v * ,ε * ) of the directed graph M corresponding to the contour feature in the two-dimensional image feature and the directed graph N corresponding to the contour feature of the three-dimensional model feature. The conjugate loss function used to construct the conjugate product graph is as follows: Among them, represents the distance correction weight of the q-th point in the directed graph M, represents the distance correction weight of the b-th point in the directed graph N, and the Softmax() function is the normalized exponential function.

[0079] The method of this application designs a geometric conjugate learning mechanism to calculate the similarity score matrix between features by using the shallow information (i.e., contour information), deep semantic information of the two-dimensional image, and three-dimensional model feature information.

[0080] First, extract the features of the contour line part from the two-dimensional image features alone as auxiliary information for calculating the correspondence between the two-dimensional image and the three-dimensional model. Define the respective directed graphs M and N on the extracted two-dimensional image contour features and three-dimensional model features , and construct the product graph P(v, ε) between M and N, where v represents the vertex set of the product graph. Since the edges of the directed graph are used to represent the vertices in the product graph, this vertex set is composed of the edges of the directed graphs M and N, and ε represents the edge set of the product graph. The edges in the directed graphs M and N are not only the vertices in the product graph, but also use these vertices to form the edges of the product graph; at the same time, define the conjugate graphs P * (v M ,ε M ) and P * (v N ,ε N ) of M and N respectively, and construct a product graph between the conjugate graphs of M and N by using the conjugate operation, so as to obtain the conjugate product graph G(v * ,ε * ) of M and N. In the conjugate product graph, the edges of the product graph are used to represent the vertices in the conjugate product graph, and the edges e * in the conjugate product graph can be formed by connecting the adjacency relationships of the vertices in the product graph.

[0081] Secondly, define a distance correction weight w and a conjugate loss function L for the construction process of the conjugate product graph rect for correction, so as to play a supervisory role in calculating the conjugate shortest distance d. To construct the distance correction weight w, for each point in M and N, use a pointer to traverse the entire contour graph or 3D model from the current point in the opposite direction of the normal vector, and calculate the distance required for the pointer to traverse through the Euclidean distance. When the distances of traversing the contour graph and the 3D model are similar, assign a higher weight to w, indicating a higher similarity between the two points in space. Use the vertex v p and edge ε p in the product graph to construct the edge e * ∈ε * in the conjugate product graph, as shown in the following formula: In the formula, and represent the two vertices that make up the edge e * in the conjugate product graph, e k and e l represent two edges in the product graph. Since the edges of the product graph are used to represent the vertices in the conjugate product graph, e k corresponds to l and and represent the two vertices that make up the edge e k in the product graph, represents the two vertices that make up the edge e l in the product graph. Use the edge e* to calculate the distance correction weight w of the 2D contour and the 3D model, and thus construct the conjugate loss function l rect (e * ) through the following formula: In the formula, represents the distance correction weight of the q-th point in the directed graph M, represents the distance correction weight of the b-th point in the directed graph N. The conjugate loss function corrects the construction process of the conjugate product graph, making the calculated conjugate shortest distance d better reflect the similarity between the 2D image contour and the 3D model.

[0082] S333. Use the Dijkstra method and the Euclidean distance to find the conjugate shortest distance d between the corresponding elements of M and N in the conjugate product graph G(v * , ε * ).

[0083] Use the Dijkstra algorithm and the Euclidean distance to find the conjugate shortest distance d between the corresponding elements of M and N in the conjugate product graph G(v * , ε * ). This conjugate shortest distance d encodes the similarity degree between the contour information and the 3D vertices.

[0084] S334. Calculate the similarity score matrix between two domain feature descriptors according to the Euclidean distance and the conjugate shortest distance d, and the calculation formula is: where Sim is the similarity score matrix, and d i,j represents the conjugate shortest distance of the feature vector of the i-th pixel point in the two-dimensional image I and the feature vector of the j-th point in the three-dimensional model S in the conjugate product graph, ||r i,I -r j,S ||2 represents the Euclidean distance between two vectors.

[0085] In the similarity calculation process between the two-dimensional image features and the three-dimensional model features, first, the technical solution of the present application enhances the distribution consistency of similar features in different domains through a loss function, considers the spatial proximity and class correlation of key points, reduces the differences between similar features in different domains, and at the same time does not overly penalize incorrect point-to-point correspondences, improving the accuracy of feature similarity calculation. Secondly, by constructing the conjugate product graph of the directed graph M corresponding to the contour features in the two-dimensional image features and the directed graph N corresponding to the three-dimensional model features, and defining a distance correction weight w and a conjugate loss function L rect for correction, it realizes the supervision and optimization of the calculation of the conjugate shortest distance d.

[0086] S34. Based on the cross-domain feature mapping mechanism, construct a soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image according to the similarity score matrix. As Figure 5 shown, it is the flowchart of the steps for constructing the soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image in the present invention, including the following steps:

[0087] S341. Convert the similarity score matrix Sim into a soft mapping matrix Soft, and the calculation formula is:

[0088] where represents element-wise multiplication, dim = 0 represents the rows in the two-dimensional matrix, dim = 1 represents the columns in the two-dimensional matrix, and the Softmax() function is the normalized exponential function.

[0089] In order to encode the correspondence between the two-dimensional image and the three-dimensional model, the mapping values calculated in the cross direction can play a role in correcting the input similarity score matrix. Therefore, the technical solution of the present application uses the dual Softmax operator to convert the similarity score matrix Sim into a soft mapping matrix Soft, as shown in the following formula:

[0090] In the formula, Denote element-wise multiplication. dim = 0 represents rows in a two-dimensional matrix, and dim = 1 represents columns in a two-dimensional matrix. The soft mapping matrix implicitly encodes the non-rigid correspondence between a two-dimensional image and a three-dimensional model. Using the soft mapping matrix Soft, the correspondence between any two points between the two-dimensional image and the three-dimensional model can be established, that is, for each pixel point in the image, the corresponding point in the three-dimensional model can be found using the soft mapping matrix.

[0091] S342. Optimize and converge the soft mapping matrix through the corresponding reconstruction loss function. The calculation formula of the corresponding reconstruction loss function is: Where, Denotes the Frobenius norm, Denotes the two-dimensional image matrix after reconstruction adjusted by geometric conjugate learning, Denotes the three-dimensional model matrix after reconstruction adjusted by geometric conjugate learning.

[0092] Use the reconstructed two-dimensional image Three-dimensional model Compare with the input original image I and three-dimensional model S. Continuously optimize the corresponding reconstruction loss function L through backpropagation recon , and iterate this process until the network converges to obtain the optimized soft mapping matrix.

[0093] S4. Real-time collect the two-dimensional ultrasound image and three-dimensional medical image of the patient, input them into the geometric conjugate learning model, and correspond each pixel point of the real-time two-dimensional ultrasound image to the vertex of the real-time three-dimensional medical image according to the soft mapping matrix to register a high-precision cardiovascular three-dimensional model.

[0094] In an actual application scenario, during a cardiovascular surgery, real-time collect the two-dimensional ultrasound image of the patient, input it into the above-trained geometric conjugate learning model, and correspond each pixel point of the two-dimensional ultrasound image to the vertex in the space of the cardiovascular three-dimensional model according to the soft mapping matrix, convert it into a high-precision three-dimensional model, and register the preliminary three-dimensional cardiovascular point cloud or voxel model.

[0095] S5. Use dynamic deformation field prediction to compensate for the cardiac periodic motion to ensure that the cardiovascular three-dimensional model is synchronized with the physiological state.

[0096] Correspond the real-time two-dimensional ultrasound input image with the cardiovascular three-dimensional model point by point through the geometric conjugate learning model, and use dynamic deformation field prediction to compensate for the cardiac periodic motion of the cardiovascular three-dimensional model to ensure that the cardiovascular three-dimensional model is synchronized with the physiological state.

[0097] In a preferred embodiment, the dynamic deformation field prediction described in this application is based on an LSTM neural network or a Transformer deep learning model.

[0098] In another preferred embodiment, the cardiovascular cross-domain model registration method based on geometric conjugate learning further includes the following steps:

[0099] S6. Identify the cardiovascular structures in the real-time two-dimensional ultrasound image, and mark the corresponding three-dimensional cardiovascular structure positions in the registered three-dimensional cardiovascular model.

[0100] Segment and identify the cardiovascular structures in the real-time two-dimensional ultrasound image, and mark the corresponding vertices of the identified contour pixels in the registered three-dimensional cardiovascular model in a special way, which can more intuitively display the organ tissues in the three-dimensional model.

[0101] In addition, based on the technical solution of the present application, an intuitive intraoperative three-dimensional navigation can also be provided for doctors. For example, through AR glasses or a surgical navigation screen, the three-dimensional vascular model is superimposed on the actual anatomical position of the patient. Use the Simultaneous Localization and Mapping (SLAM) technology to real-time track the positions of the surgical instruments and the ultrasound probe to ensure the spatial alignment of the model and the surgical scene. Or the doctor can mark the lesion areas (such as plaques, stenosis segments) in the three-dimensional model through gestures or voice commands. In addition, the lesion characteristics can be automatically analyzed and assist in surgical decision-making. For example, perform lesion quantification analysis based on the three-dimensional model: automatically calculate key parameters such as vascular stenosis rate, plaque volume, fractional flow reserve (FFR), etc., and use the Graph Convolutional Network (GCN) to analyze the topological anomalies of the vascular network (such as the lesion risk at bifurcations). Or risk warning and plan recommendation: combine the clinical guideline library (such as the ACC / AHA guidelines), generate warning prompts for high-risk areas (such as "left anterior descending artery stenosis > 70%"), and recommend surgical plans according to the lesion characteristics (such as stent size, balloon dilation pressure range).

[0102] The above specific embodiments only explain the technical solution of the present invention in detail. The present invention is not limited only to the above embodiments. Any improvement or replacement based on the principle of the present invention shall be within the protection scope of the present invention.

Claims

1. A cardiovascular cross-domain model registration method based on geometric conjugate learning, characterized in that It includes the following steps: S1. Collect sample data of the patient; the sample data includes two-dimensional ultrasound images and three-dimensional medical images collected synchronously; S2. Perform data preprocessing on the sample data; S3. Construct a training geometric conjugate learning model based on the sample data to obtain a soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image; S4. Collect the two-dimensional ultrasound image and three-dimensional medical image of the patient in real time, input them into the geometric conjugate learning model, and correspond each pixel point of the real-time two-dimensional ultrasound image to the vertices of the real-time three-dimensional medical image according to the soft mapping matrix to register a high-precision cardiovascular three-dimensional model; S5. Use a dynamic deformation field to predict and compensate for the periodic movement of the heart to ensure that the cardiovascular three-dimensional model is synchronized with the physiological state.

2. The cardiovascular cross-domain model registration method based on geometric conjugate learning according to claim 1, wherein The data preprocessing of the sample data includes the following steps: S21. Perform anisotropic filtering or wavelet transform denoising on the two-dimensional ultrasound image to enhance the image contrast; S22. Perform non-local means denoising on the three-dimensional medical image; S23. Use rigid registration based on feature point matching to align the two-dimensional ultrasound image and the three-dimensional medical image; S24. Use a non-rigid registration method to compensate for cardiac motion artifacts in the two-dimensional ultrasound image; S25. Automatically segment the cardiovascular structure of the sample data using a U-Net model and label key anatomical landmark points.

3. The cardiovascular cross-domain model registration method based on geometric conjugate learning according to claim 1, wherein, The construction of a training geometric conjugate learning model based on the sample data to obtain a soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image includes the following steps: S31. Extract multi-scale two-dimensional image features of the two-dimensional ultrasound image based on the Transformer model; S32. Extract three-dimensional model features of the three-dimensional medical image based on a feature pyramid and a 3D convolutional module; S33. Construct a feature similarity calculation module based on a geometric conjugate learning mechanism to calculate a similarity score matrix between the two-dimensional image features and the three-dimensional model features; S34. Construct a soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image based on the similarity score matrix according to a cross-domain feature mapping mechanism.

4. The method for cardiovascular cross-domain model registration based on geometric conjugate learning according to claim 3, wherein The construction of a feature similarity calculation module based on a geometric conjugate learning mechanism to calculate a similarity score matrix between the two-dimensional image features and the three-dimensional model features includes the following steps: S331. Utilize the loss function L core Enhance the distribution consistency of the similar features between the two-dimensional image features and the three-dimensional model features. The loss function L core has the following calculation formula: where r i,I represents the feature vector of the i-th pixel point in the two-dimensional image I, and r j,S represents the feature vector of the j-th point in the three-dimensional model S. ||r i,I -r j,S ||2 represents the distance between the two vectors. class I represents the class label of r i,I , and class S represents the class label of r j,S . indic() is an indicator function for calculating whether the class labels are consistent. If they are consistent, the output is 0; otherwise, it is 1. S332. Construct the conjugate product graph \(G(v\) * , ε * ) of the directed graph \(M\) corresponding to the contour feature of the two-dimensional image feature and the directed graph \(N\) corresponding to the contour feature of the three-dimensional model feature. The conjugate loss function used to construct the conjugate product graph is: Among them, represents the distance correction weight of the \(q\)-th point in the directed graph \(M\), represents the distance correction weight of the \(b\)-th point in the directed graph \(N\), and the Softmax() function is the normalized exponential function; S333. Use the Dijkstra method and the Euclidean distance to find the conjugate shortest distance d between the corresponding elements of M and N in the conjugate product graph G(v * , ε * ); S334. Calculate the similarity score matrix between two domain feature descriptors based on the Euclidean distance and the conjugate shortest distance d, and the calculation formula is: where Sim is the similarity score matrix, d i,j represents the conjugate shortest distance in the conjugate product graph between the feature vector of the i-th pixel point in the two-dimensional image I and the feature vector of the j-th point in the three-dimensional model S, ||r i,I -r j,S ||2 represents the Euclidean distance between two vectors.

5. The method for cardiovascular cross-domain model registration based on geometric conjugate learning according to claim 4, characterized in that, The construction of a soft mapping matrix of the correspondence between the pixel points of the two-dimensional ultrasound image and the vertices of the three-dimensional medical image based on the similarity score matrix according to a cross-domain feature mapping mechanism includes the following steps: S341. Convert the similarity score matrix Sim into a soft mapping matrix Soft, and the calculation formula is: Among them, denotes element-wise multiplication, dim = 0 represents the rows in a two-dimensional matrix, dim = 1 represents the columns in a two-dimensional matrix, and the Softmax() function is the normalized exponential function; S342. Optimize and converge the soft mapping matrix through the corresponding reconstruction loss function, and the calculation formula of the corresponding reconstruction loss function is as follows: Wherein, represents the Frobenius norm, represents the two-dimensional image matrix after reconstruction adjusted by geometric conjugate learning, represents the three-dimensional model matrix after reconstruction adjusted by geometric conjugate learning.

6. The method for cardiovascular cross-domain model registration based on geometric conjugate learning according to claim 1, wherein , The dynamic deformation field prediction is based on an LSTM neural network or a Transformer deep learning model.

7. The cardiovascular cross-domain model registration method based on geometric conjugate learning according to claim 1, wherein , It also includes the following steps: S6. Identify the cardiovascular structure in the real-time two-dimensional ultrasound image and mark the corresponding three-dimensional cardiovascular structure position in the registered cardiovascular three-dimensional model.

Images