3D-TEE image left auricle opening plane positioning method based on multi-scale information guide diffusion model

Abstract

A 3D-TEE image left atrial appendage opening plane positioning method based on a multi-scale information guide diffusion model comprises the steps that firstly, a to-be-predicted 3D-TEE image x is input into a ResNet image encoder to obtain an image feature rho (x), meanwhile, the to-be-predicted 3D-TEE image x is input into a trained multi-scale information guide module, a multi-scale prior feature prior is generated, and the image feature rho (x) is obtained; according to the 3D-TEE image left atrial appendage opening plane positioning method based on the multi-scale information guide diffusion model, HRNet is utilized to extract high-resolution multi-scale features, a multi-scale attention block is designed to fuse the multi-scale features, the multi-scale features are fully utilized, and the positioning accuracy of the 3D-TEE image left atrial appendage opening plane positioning method based on the multi-scale information guide diffusion model is improved. And perception of the model on an image space structure is improved. In addition, according to the target of the task, the plane positioning parameters are subjected to attitude decoupling and decomposed into rotation and translation components, different loss functions are adopted for different components, and the prediction precision of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of medical artificial intelligence algorithm technology, and in particular to a 3D-TEE image left atrial appendage opening plane positioning method based on a multi-scale information guided diffusion model. Background Art

[0002] Left atrial appendage occlusion is a minimally invasive cardiac interventional procedure for preventing stroke in patients with non-valvular atrial fibrillation. The left atrial appendage occluder is delivered to the left atrial appendage opening through a catheter delivery system for release, thereby isolating the blood flow between the left atrial appendage and the left atrium, thereby avoiding the formation of left atrial appendage thrombus. Before left atrial appendage occlusion, imaging tests are required to exclude left atrial appendage thrombus and measure left atrial appendage parameters to determine the type and size of the occluder. The left atrial appendage opening plane is an important reference plane for clinicians to determine the size of the occluder and plan the implantation position of the occluder during left atrial appendage occlusion. However, the individual variability of the anatomical structure of the left atrial appendage is large. When clinicians plan the size and release position of the occluder based on the left atrial appendage opening plane determined by preoperative ultrasound images, they are greatly affected by personal subjectivity, so that preoperative judgments are often inconsistent with intraoperative judgments.

[0003] At present, in the relevant research on the positioning of the left atrial appendage opening plane, the method of deep reinforcement learning model is mainly used. It constructs the center line of the left atrial appendage, uses the deep reinforcement model to search the left atrial appendage opening plane in the center line of the left atrial appendage, and thus locates the left atrial appendage opening plane. This method is applicable to CT images, and lacks the posture parameters of the left atrial appendage opening plane. It only searches the plane determined by the center line, and cannot locate the correct direction of the left atrial appendage opening plane. Secondly, the anatomical variability of the left atrial appendage is large. For the "cauliflower-shaped" and "cactus-shaped" left atrial appendages with multi-leaf structures, it is difficult to represent them with the center line in the TEE image. Moreover, the deep reinforcement learning model is mainly applicable to structures with small anatomical structures and obvious feature changes. For structures with large anatomical variability such as the left atrial appendage, the current model is difficult to accurately complete the motion prediction of locating the left atrial appendage opening plane.

[0004] Therefore, in view of the shortcomings of the existing technology, it is necessary to provide a 3D-TEE image left atrial appendage opening plane positioning method based on a multi-scale information guided diffusion model to solve the shortcomings of the existing technology. Summary of the invention

[0005] The purpose of the present invention is to avoid the shortcomings of the prior art and provide a 3D-TEE image left atrial appendage opening plane positioning method based on a multi-scale information guided diffusion model. The 3D-TEE image left atrial appendage opening plane positioning method based on a multi-scale information guided diffusion model can improve the model's perception of the image spatial structure and enhance the model's prediction accuracy.

[0006] The above-mentioned purpose of the present invention is achieved by the following technical measures:

[0007] A method for locating the left atrial appendage opening plane in a 3D-TEE image based on a multi-scale information guided diffusion model is provided, comprising the following steps:

[0008] S1. Input the 3D-TEE image x to be predicted into the ResNet image encoder to obtain the image feature ρ(x). At the same time, input the 3D-TEE image x to be predicted into the trained multi-scale information guidance module to generate the multi-scale prior feature y. prior ;

[0009] S2, at time T, the multi-scale prior feature y obtained in S1 prior Add a random Gaussian noise to get the noise variable y T , T is a positive integer;

[0010] S3, the image feature ρ(x) and multi-scale prior feature y obtained in S1 prior , and the noise variable y T Splicing, the spliced ​​image features ρ(x) and multi-scale prior features y prior and the noise variable y T Input the trained denoising model ∈ θ , predict the noise at time T-1 and get the noise variable y T-1 ;

[0011] S4, determine whether T-1 reaches 0, if yes, go to S6, if no, go to S5;

[0012] S5, let T-1 = T, noise variable y T-1 = noise variable y T , return to S3;

[0013] S6. The noise variable y T-1 Defined as predicted positioning result The predicted positioning result It includes a rotation component and a translation component. The rotation component is the rotation transformation from the target plane to the xoy plane. The rotation transformation is represented by a quaternion of R: [w, x, y, z]. The translation component is the Euclidean distance from the target plane to the center of the 3D-TEE image.

[0014] Preferably, the above denoising model ∈ θ The training method is carried out in the following steps:

[0015] A1. Obtain the 3D-TEE training image x, then obtain the true label y0 based on the 3D-TEE training image x, input the 3D-TEE training image x into the post-training multi-scale information guidance module, and obtain the multi-scale prior feature yprior ;

[0016] A2, 3D-TEE training image x is input into the ResNet image encoder to obtain image features ρ(x), and then the image features ρ(x) and the multi-scale prior features y obtained in A1 are combined. prior The true label y0 obtained by A1 is diffused according to a randomly generated time step t to generate a noise variable y t , there exists t<T, t is a positive integer;

[0017] A3, the image feature ρ(x) obtained in A1 and the multi-scale prior feature y prior , and A2 obtains the noise variable y t Splicing, the spliced ​​image features ρ(x) and multi-scale prior features y prior and the noise variable y t Input denoising model ∈ θ , and obtain the noise distribution prediction result;

[0018] A4. Calculate the loss function based on the mean square error. The noise distribution prediction results of A3 and the noise variable y of A1 t The loss result is then used to perform gradient update on the loss result to determine the denoising model ∈ θ , enter A5;

[0019] A5, determine the current number of training times. When the current number of training times is less than the maximum number of training times, return to A2; when the current number of training times is equal to the maximum number of training times, use the currently updated denoising model ∈ θ As the trained denoising model ∈ θ .

[0020] Preferably, the above-mentioned multi-scale information guidance model is composed of a backbone, a channel mapping layer and a multi-scale attention block, wherein the backbone is HRNet, and the channel mapping layer is added to the output head structure of the HRNet to form a fully connected layer, and the output of the channel mapping layer serves as the input of the multi-scale attention block.

[0021] Preferably, the training process of the multi-scale information guidance module is carried out by the following steps:

[0022] B1. Input the 3D-TEE training image x into the multi-scale information guidance model, extract the multi-scale features, and then calculate the attention of the multi-scale features to obtain the multi-scale prior features y prior ;

[0023] B2, the multi-scale prior features y obtained from B1 priorApply the maximum pooling operation and the fully connected layer to output the quaternion and translation value results, and obtain the quaternion and translation value results and the parameters of the multi-scale information guidance model of the attitude loss function optimization B1;

[0024] B3, determine the current training times. When the current training times are less than the target training times, go to B4; when the current training times are equal to the target training times, go to B10;

[0025] B4, determine whether the early stopping mechanism is satisfied. If not, proceed to B1; if satisfied, proceed to B5. The early stopping mechanism is satisfied when the loss value of the current training is equal to or greater than the loss value of the previous training.

[0026] B5. Determine the difference between the current training times and the target training times. When the difference between the training times is greater than or equal to the early stop training times, go to B6. When the difference between the training times is less than the early stop training times, return to B1.

[0027] B6, set the current training times to 1 and go to B7;

[0028] B7. Input the 3D-TEE training image x into the optimized multi-scale information guidance model, extract the multi-scale features, and then calculate the attention of the multi-scale features to obtain the multi-scale prior features y prior ;

[0029] B8, multi-scale prior features y obtained in B7 prior Apply the maximum pooling operation and the fully connected layer to output the quaternion and translation value results, and use the quaternion and translation value results and the attitude loss function to optimize the parameters of the multi-scale information guidance model of B7;

[0030] B9, determine whether the current training times reach the early stop training times. When the current training times are less than the early stop training times, go to B7; when the current training times are equal to the early stop training times, go to B10;

[0031] B10. Use the current optimized multi-scale information guidance model as the trained multi-scale information guidance model.

[0032] Preferably, the above posture loss function is expressed by equations (1) to (3):

[0033]

[0034] in, is the posture loss value, is the quaternion loss; is the translation loss; q gt is the quaternion true label of the target plane; q pred is the quaternion predicted positioning result of the target plane; Tgt is the true label of the translation distance from the target plane to the center of the image; T pred Predict the positioning result for the translation distance from the target plane to the center point of the image.

[0035] Preferably, the target training times are 300; and the early stopping training times are 10.

[0036] Preferably, the multi-scale attention block calculates the multi-scale attention in a serial superposition manner, and adjusts the order of layer normalization operations in the scale attention, performs self-attention calculation on the features of each scale, and superimposes the self-attention calculation result of the previous scale, and then sends it to the multi-layer perceptron MLP, and applies layer normalization Layer Norm after each operation.

[0037] The multi-scale attention series superposition calculation is expressed by equations (4) and (5):

[0038]

[0039] y′ i =y i +y′ i-1 ......Formula (5);

[0040] where y i and y′ i are the input and output features of the i-th multi-scale self-attention respectively; y′ i-1 is the output feature of the i-1th multi-scale attention; Attn(·) is the self-attention, Q, K, V are the query, key and value in the self-attention respectively; W i Q , W i K and W i V are the learnable parameters of the linear mapping layer.

[0041] In A2, the noise variable y t It is calculated by the diffusion formula, specifically expressed by formula (6):

[0042]

[0043] Where ∈~N(0,I), N(0,I) is Gaussian distribution, I is unit matrix, α t =1-β t , {β t} t＝1：T ∈(0, 1) T , α t Preserve ratio for signal, is α t The cumulative product of β t is the noise intensity.

[0044] In A6, the loss result is obtained by formula (7):

[0045]

[0046] in As a result of loss.

[0047] Preferably, the above predicted positioning result It is expressed by formula (8):

[0048]

[0049] in, is a normal distribution, I is the unit matrix, y 0：T-1 is the noise variable y t , and t∈[0,...,T-1].

[0050] Preferably, the above T is 1000.

[0051] Preferably, the above denoising model ∈ θ It consists of multiple fully connected layers embedded with time codes, and in each fully connected layer, the output vector of the fully connected layer is subjected to the Hadamard product operation with the time code, and the Softplus activation function is applied to the output to embed the time step. The fully connected layer embedded with time code is expressed by formula (8):

[0052] l i =σ(g(l i-1 )⊙g(t))......Formula (8);

[0053] Among them l i is the denoising model ∈ θ The i-th layer output, l i-1 is the denoising model ∈ θ is the output of the i-1th layer, σ is the Softplus activation function, g(·) is the fully connected layer, and ⊙ is the Hadamard product.

[0054] The present invention provides a 3D-TEE image left atrial appendage opening plane positioning method based on a multi-scale information guided diffusion model, comprising the following steps: S1, inputting the 3D-TEE image x to be predicted into a ResNet image encoder to obtain an image feature ρ(x), and at the same time inputting the 3D-TEE image x to be predicted into a trained multi-scale information guided module to generate a multi-scale prior feature y prior ; S2, at time T, the multi-scale prior feature y obtained in S1prior Add a random Gaussian noise to get the noise variable y T , T is a positive integer; S3, the image feature ρ(x) obtained in S1, the multi-scale prior feature y prior , and the noise variable y T Splicing, the spliced ​​image features ρ(x) and multi-scale prior features y prior and the noise variable y T Input the trained denoising model ∈ θ , predict the noise at time T-1 and get the noise variable y T-1 ; S4, determine whether T-1 reaches 0, if yes, go to S6, if no, go to S5; S5, set T-1 = T, noise variable y T-1 = noise variable y T , return to S3; S6, set the noise variable y T-1 Defined as predicted positioning result The predicted positioning result It includes a rotation component and a translation component, wherein the rotation component is a rotation transformation from the target plane to the xoy plane, the rotation transformation is represented by a quaternion of R: [w, x, y, z], and the translation component is the Euclidean distance from the target plane to the center of the 3D-TEE image. The 3D-TEE image left atrial appendage opening plane positioning method based on the multi-scale information guided diffusion model utilizes HRNet to extract high-resolution multi-scale features, and designs a multi-scale attention block to fuse multi-scale features, making full use of multi-scale features to improve the model's perception of the image space structure. The present invention also decouples the plane positioning parameters according to the objectives of the task, decomposes them into rotation and translation components, and uses different loss functions for different components to enhance the prediction accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0056] Figure 1 This is a flow chart of a method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information-guided diffusion model.

[0057] Figure 2 Schematic diagram of the structure of the multi-scale information guidance model.

[0058] Figure 3 Schematic diagram of the denoising model.

[0059] Figure 4 The figure is a comparison chart of the predicted positioning results of the present invention and the predicted positioning results of the prior art. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described in conjunction with the following embodiments.

[0061] Example 1

[0062] A method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information-guided diffusion model. Figure 1 As shown, the following steps are included:

[0063] S1. Input the 3D-TEE image x to be predicted into the ResNet image encoder to obtain the image feature ρ(x). At the same time, input the 3D-TEE image x to be predicted into the trained multi-scale information guidance module to generate the multi-scale prior feature y. prior ;

[0064] S2, at time T, the multi-scale prior feature y obtained in S1 prior Add a random Gaussian noise to get the noise variable y T , T is a positive integer;

[0065] S3, the image feature ρ(x) and multi-scale prior feature y obtained in S1 prior , and the noise variable y T Splicing, the spliced ​​image features ρ(x) and multi-scale prior features y prior and the noise variable y T Input the trained denoising model ε θ , predict the noise at time T-1 and get the noise variable y T-1 ;

[0066] S4, determine whether T-1 reaches 0, if yes, go to S6, if no, go to S5;

[0067] S5, let T-1 = T, noise variable y T-1 = noise variable y T , return to S3;

[0068] S6. The noise variable y T-1 Defined as predicted positioning result Predicted positioning results It includes rotation component and translation component. The rotation component is the rotation transformation from the target plane to the xoy plane. The rotation transformation is represented by the quaternion R: [w, x, y, z]. The translation component is the Euclidean distance from the target plane to the center of the 3D-TEE image.

[0069] like Figure 2As shown in the figure, the multi-scale information guided model consists of a backbone, a channel mapping layer and a multi-scale attention block, where the backbone is HRNet. A channel mapping layer is added to the output head structure of HRNet to form a fully connected layer, and the output of the channel mapping layer is used as the input of the multi-scale attention block.

[0070] The training process of the multi-scale information guidance module is carried out in the following steps:

[0071] B1. Input the 3D-TEE training image x into the multi-scale information guidance model, extract the multi-scale features, and then calculate the attention of the multi-scale features to obtain the multi-scale prior features y prior ;

[0072] B2, the multi-scale prior features y obtained from B1 prior Apply the maximum pooling operation and the fully connected layer to output the quaternion and translation value results, and obtain the quaternion and translation value results and the parameters of the multi-scale information guidance model of the attitude loss function optimization B1;

[0073] B3, determine the current training times. When the current training times are less than the target training times, go to B4; when the current training times are equal to the target training times, go to B10;

[0074] B4, determine whether the early stopping mechanism is satisfied. If not, go to B1; if satisfied, go to B5. The early stopping mechanism is satisfied when the loss value of the current training is equal to or greater than the loss value of the previous training.

[0075] B5. Determine the difference between the current training times and the target training times. When the difference between the training times is greater than or equal to the early stop training times, go to B6. When the difference between the training times is less than the early stop training times, return to B1.

[0076] B6, set the current training times to 1 and go to B7;

[0077] B7. Input the 3D-TEE training image x into the optimized multi-scale information guidance model, extract the multi-scale features, and then calculate the attention of the multi-scale features to obtain the multi-scale prior features y prior ;

[0078] B8, multi-scale prior features y obtained in B7 prior Apply the maximum pooling operation and the fully connected layer to output the quaternion and translation value results, and use the quaternion and translation value results and the attitude loss function to optimize the parameters of the multi-scale information guidance model of B7;

[0079] B9, determine whether the current training times reach the early stop training times. When the current training times are less than the early stop training times, go to B7; when the current training times are equal to the early stop training times, go to B10;

[0080] B10. Use the current optimized multi-scale information guidance model as the trained multi-scale information guidance model.

[0081] It should be noted that the training process of the multi-scale information guidance module of the present invention adopts the Early Stop training strategy and uses posture loss as the loss function for training. When the early stop mechanism is triggered, the parameters of the multi-scale information guidance conditional model are frozen, and no gradient update is performed in the subsequent regression diffusion model training. Among them, the target training times, early stop training times, maximum training times and T specific values ​​of the present invention are determined by actual conditions. In this embodiment, the target training times is 300; the early stop training times is 10, the maximum training times is 350, and T is 1000 for example.

[0082] The posture loss function is expressed by equations (1) to (3):

[0083]

[0084] in, is the posture loss value, is the quaternion loss; is the translation loss; q gt is the quaternion true label of the target plane; q pred is the quaternion predicted positioning result of the target plane; T gt is the true label of the translation distance from the target plane to the center of the image; T pred Predict the positioning result for the translation distance from the target plane to the center point of the image.

[0085] It should be noted that in the task definition, the present invention decomposes the plane positioning parameters into quaternion components and translation components. Therefore, the present invention designs an attitude loss function to train the multi-scale information guidance model. The attitude loss value consists of quaternion loss and translation loss, where quaternion loss Used to measure the similarity between the target quaternion and the predicted quaternion; translation loss Used to measure the distance between the target plane and the predicted plane.

[0086] The multi-scale attention block calculates multi-scale attention in a serial superposition manner, and adjusts the order of layer normalization operations in the scale attention, calculates self-attention for the features of each scale, and superimposes the self-attention calculation results of the previous scale, and then sends them to the multi-layer perceptron MLP, and applies layer normalization LayerNorm after each operation;

[0087] The serial superposition calculation of multi-scale attention is expressed by equations (4) and (5):

[0088]

[0089] y′ i =y i +y′ i-1 ......Formula (5);

[0090] where y i and y′ i are the input and output features of the i-th multi-scale self-attention respectively; y′ i-1 is the output feature of the i-1th multi-scale attention; Attn(·) is the self-attention, Q, K, V are the query, key and value in the self-attention respectively; W i Q , W i K and W i V are the learnable parameters of the linear mapping layer.

[0091] The denoising model ∈ θ The training method is carried out in the following steps:

[0092] A1. Obtain the 3D-TEE training image x, then obtain the true label y0 based on the 3D-TEE training image x, input the 3D-TEE training image x into the post-training multi-scale information guidance module, and obtain the multi-scale prior feature y prior ;

[0093] A2, 3D-TEE training image x is input into the ResNet image encoder to obtain image features ρ(x), and then the image features ρ(x) and the multi-scale prior features y obtained in A1 are combined. prior The true label y0 obtained by A1 is diffused according to a randomly generated time step t to generate a noise variable y t , there exists t<T, t is a positive integer;

[0094] A3, the image feature ρ(x) obtained in A1 and the multi-scale prior feature y prior , and A2 obtains the noise variable y t Splicing, the spliced ​​image features ρ(x) and multi-scale prior features y prior and the noise variable y t Input denoising model ∈ θ , and obtain the noise distribution prediction result;

[0095] A4. Calculate the loss function based on the mean square error. The noise distribution prediction results of A3 and the noise variable y of A1 t The loss result is then used to perform gradient update on the loss result to determine the denoising model ∈ θ , enter A5;

[0096] A5, determine the current number of training times. When the current number of training times is less than the maximum number of training times, return to A2; when the current number of training times is equal to the maximum number of training times, use the currently updated denoising model ∈ θ As the trained denoising model ∈ θ .

[0097] It should be noted that S1-S6 of the present invention can also be called the inference stage of the model, or the reverse process, and A1-A5 can also be called the training stage of the model, or the forward process. The forward process is to continuously add noise to the real label according to the time step t (that is, the so-called diffusion), that is, y0→…→y t The reverse process is to continuously denoise a noise according to the time step T until it is restored to the true label (the so-called denoising process), that is,

[0098] The denoising model ∈ θ The pseudo code for training is as follows:

[0099]

[0100] The inference calculation plane prediction result of the present invention is as follows: First, the test set image is input into the multi-scale information guidance model and the image encoder to obtain the multi-scale prior feature y prior and image features ρ(x), sample random Gaussian noise z. Then the diffusion process is reversed according to And the following formula I and formula II, from t = 1000 to 1, infer and reconstruct the predicted positioning result

[0101]

[0102] The pseudo code of the reasoning process is as follows:

[0103]

[0104]

[0105] In the denoising model ∈ θ In the training, the noise variable is calculated by the diffusion formula, which is expressed by formula (6):

[0106]

[0107] Where ∈~N(0,I), N(0,I) is Gaussian distribution, I is unit matrix, α t =1-β t , {β t} t＝1：T∈(0, 1) T , α t Preserve ratio for signal, is α t The cumulative product of β t is the noise intensity. In the denoising model ∈ θ In the training of t , multi-scale prior features y prior And the image feature ρ(x) is input to the denoising model ∈ θ (ρ(x), y t ,y prior , t), the noise distribution is learned.

[0108] In A6, the loss result is obtained by formula (7):

[0109]

[0110] in As a result of loss.

[0111] The denoising model ∈ θ The role of is to use the deep neural network as a function approximator of the parameterized reverse denoising process to learn the noise distribution in the forward diffusion process, ∈ θ The prediction result (i.e. output) is the noise variable y generated after sampling t time steps from the true label y0 t The added real noise distribution ∈, its training can be expressed as the noise estimation loss given by formula (7) minimize.

[0112] Predicted positioning results It is expressed by formula (8):

[0113]

[0114] in, is a normal distribution, I is the unit matrix, y 0：T-1 is the noise variable y t , and t∈[0,...,T-1].

[0115] It should be noted that in the reverse denoising process, the present invention uses the denoising model ∈ trained in the training phase θ , for random Gaussian noise variable distribution p θ (y T ) to carry out the real noise variable distribution p θ The prediction of (y0) is gradually denoised to finally generate the predicted positioning result The predicted positioning results are the quaternion and translation value, which together define the spatial position of the left atrial appendage opening plane.

[0116] like Figure 3 As shown, the denoising model ∈ θ It consists of multiple fully connected layers embedded with time codes, and in each fully connected layer, the output vector of the fully connected layer is subjected to the Hadamard product operation with the time code, and the Softplus activation function is applied to the output to embed the time step. The fully connected layer embedded with time code is expressed by formula (8):

[0117] l i =σ(g(l i-1 )⊙g(t))......Formula (8);

[0118] Among them l i is the denoising model ∈ θ The i-th layer output, l i-1 is the denoising model ∈ θ is the output of the i-1th layer, σ is the Softplus activation function, g(·) is the fully connected layer, and ⊙ is the Hadamard product.

[0119] The 3D-TEE image left atrial appendage opening plane positioning method based on the multi-scale information guided diffusion model utilizes HRNet to extract high-resolution multi-scale features, and designs a multi-scale attention block to fuse multi-scale features, making full use of multi-scale features to improve the model's perception of the image space structure. According to the objectives of the task, the present invention also decouples the plane positioning parameters into rotation and translation components, uses different loss functions for different components, and enhances the prediction accuracy of the model.

[0120] Example 2

[0121] A method for locating the left atrial appendage opening plane of a 3D-TEE image based on a multi-scale information-guided diffusion model, with other features being the same as those of Example 1. The data set used in this embodiment was collected from the Department of Cardiology of Zhujiang Hospital, containing a total of 407 3D-TEE image data. The data was pre-processed with resampling, padding, and cropping to ensure that the size of the input image was 128*128*128. The model was then trained and inferred according to the steps of Example 1. The angle Angle (°) and the Euclidean distance (mm) between the target plane and the prediction plane were used as evaluation indicators.

[0122] The quantitative results of the present invention and the prior art are shown in Table 1:

[0123] Table 1. Quantitative results of the present invention and prior art

[0124] Diffusion MSGM Pose-Loss Angle(°) Distance(mm) ResNet — — — 23.65 4.44 M.1 √ — — 16.59 6.3 M.2 √ √ — 13.57 8.11 The present invention √ √ √ 13.86 7.95

[0125] Among them, √ represents the model to which this technology is applied, M1 represents the classic regression diffusion model, and M2 represents the regression diffusion model with the addition of a multi-scale information guidance model.

[0126] As can be seen from Table 1, compared with the simple convolutional neural network ResNet, the technology of the present invention has been greatly improved in the prediction of the rotation component, but the prediction error of the translation component has become larger. According to M1 compared with M2, when multi-scale information guidance is added, the rotation component is further improved, but the prediction of the translation component is not improved, which shows that multi-scale information has an improvement in the spatial structure information of the left atrial appendage for model perception. Comparing M2 with the technology of the present invention, it can be seen that the prediction of the translation component has been improved, while the prediction of the rotation component has been slightly reduced, which shows that the posture loss has an improvement effect on the perception of the model.

[0127] Figure 4 This is a qualitative result of the present invention and the prior art. Figure 4 The following are visualization results of two examples of left atrial appendage opening plane positioning. The purple plane is the ground truth, and the blue plane is the model prediction result. Compared with other models, the present invention has a better left atrial appendage opening plane positioning effect.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information-guided diffusion model, characterized in that: The steps include: S1. Input the 3D-TEE image x to be predicted into the ResNet image encoder to obtain the image feature ρ(x). At the same time, input the 3D-TEE image x to be predicted into the trained multi-scale information guidance module to generate the multi-scale prior feature y. prior ; S2, at time T, the multi-scale prior feature y obtained in S1 prior Add a random Gaussian noise to get the noise variable y T , T is a positive integer; S3, the image feature ρ(x) and multi-scale prior feature y obtained in S1 prior , and the noise variable y T Splicing, the spliced ​​image features ρ(x) and multi-scale prior features y prior and the noise variable y T Input the trained denoising model ε θ , predict the noise at time T-1 and get the noise variable y T-1 ; S4, determine whether T-1 reaches 0, if yes, go to S6, if no, go to S5; S5, let T-1 = T, noise variable y T-1 = noise variable y T , return to S3; S6. The noise variable y T-1 Defined as predicted positioning result The predicted positioning result It includes a rotation component and a translation component. The rotation component is the rotation transformation from the target plane to the xoy plane. The rotation transformation is represented by a quaternion of R: [w, x, y, z]. The translation component is the Euclidean distance from the target plane to the center of the 3D-TEE image.

2. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information guided diffusion model according to claim 1, characterized in that: The denoising model ∈ θ The training method is carried out in the following steps: A1. Obtain the 3D-TEE training image x, then obtain the true label y0 based on the 3D-TEE training image x, input the 3D-TEE training image x into the post-training multi-scale information guidance module, and obtain the multi-scale prior feature y prior ; A2, 3D-TEE training image x is input into the ResNet image encoder to obtain image features ρ(x), and then the image features ρ(x) and the multi-scale prior features y obtained in A1 are combined. prior The true label y0 obtained by A1 is diffused according to a randomly generated time step t to generate a noise variable y t , there exists t<T, t is a positive integer; A3, the image feature ρ(x) obtained in A1 and the multi-scale prior feature y prior , and A2 obtains the noise variable y t Splicing, the spliced ​​image features ρ(x) and multi-scale prior features y prior and the noise variable y t Input denoising model ∈ θ , and obtain the noise distribution prediction result; A4. Calculate the loss function based on the mean square error. The noise distribution prediction results of A3 and the noise variable y of A1 t The loss result is then used to perform gradient update on the loss result to determine the denoising model ∈ θ , enter A5; A5. Determine the current number of training times. When the current number of training times is less than the maximum number of training times, return to A2. When the current number of training times is equal to the maximum number of training times, the currently updated denoising model ∈ θ As the trained denoising model ∈ θ .

3. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information-guided diffusion model according to any one of claims 1 to 2, characterized in that: The multi-scale information guided model consists of a backbone, a channel mapping layer and a multi-scale attention block, wherein the backbone is HRNet, the channel mapping layer is added to the output head structure of the HRNet to form a fully connected layer, and the output of the channel mapping layer is used as the input of the multi-scale attention block.

4. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information guided diffusion model according to claim 3, characterized in that: The training process of the multi-scale information guidance module is carried out by the following steps: B1. Input the 3D-TEE training image x into the multi-scale information guidance model, extract the multi-scale features, and then calculate the attention of the multi-scale features to obtain the multi-scale prior features y prior ; B2, the multi-scale prior features y obtained from B1 prior Apply the maximum pooling operation and the fully connected layer to output the quaternion and translation value results, and obtain the quaternion and translation value results and the parameters of the multi-scale information guidance model of the attitude loss function optimization B1; B3, determine the current training times. When the current training times are less than the target training times, go to B4; when the current training times are equal to the target training times, go to B10; B4, determine whether the early stopping mechanism is satisfied. If not, proceed to B1; if satisfied, proceed to B5. The early stopping mechanism is satisfied when the loss value of the current training is equal to or greater than the loss value of the previous training. B5, determine the difference between the current training times and the target training times. When the difference between the training times is greater than or equal to the early stop training times, proceed to B6; When the difference in the number of training times is less than the number of early stopping training times, return to B1; B6, set the current training times to 1, and go to B7; B7. Input the 3D-TEE training image x into the optimized multi-scale information guidance model, extract the multi-scale features, and then calculate the attention of the multi-scale features to obtain the multi-scale prior features y prior ; B8, multi-scale prior features y obtained in B7 prior Apply the maximum pooling operation and the fully connected layer to output the quaternion and translation value results, and use the quaternion and translation value results and the attitude loss function to optimize the parameters of the multi-scale information guidance model of B7; B9, determine whether the current training times reach the early stop training times. If the current training times are less than the early stop training times, go to B7; When the current number of training times is equal to the number of early stop training times, enter B10; B10. Use the current optimized multi-scale information guidance model as the trained multi-scale information guidance model.

5. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information guided diffusion model according to claim 4, characterized in that: The posture loss function is expressed by equations (1) to (3): in, is the posture loss value, is the quaternion loss; is the translation loss; q gt is the quaternion true label of the target plane; q pred is the quaternion predicted positioning result of the target plane; T gt is the true label of the translation distance from the target plane to the center of the image; T pred Predict the positioning result for the translation distance from the target plane to the center point of the image.

6. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information guided diffusion model according to claim 3, characterized in that: The multi-scale attention block calculates the multi-scale attention in a serial superposition manner, and adjusts the order of the layer normalization operations in the multi-scale attention, calculates the self-attention of the features of each scale, and superimposes the self-attention calculation result of the previous scale, and then sends it to the multi-layer perceptron MLP, and applies layer normalization Layer Norm after each operation; The multi-scale attention series superposition calculation is expressed by equations (4) and (5): y′ i = Attn(y′ i W i Q , y′ i W i K , y′ i W i V )...... Equation (4); y' i = y i + y' i-1 ...... Equation (5); where y i and y′ i are the input features and output features of the i-th multi-scale self-attention respectively; y′ i-1 is the output feature of the i-1th multi-scale attention; Attn(·) is the self-attention, Q, K, V are the query, key and value in the self-attention respectively; W i Q , W i k and W i V are the learnable parameters of the linear mapping layer.

7. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information-guided diffusion model according to claim 6, characterized in that: In A2, the noise variable yt is calculated by the diffusion formula, which is specifically expressed by formula (6): Where ∈~N(0,I), N(0,I) is Gaussian distribution, I is unit matrix, α t =1-β t , {β t } t＝1：T ∈(0, 1) T , α t Preserve the ratio for the signal, is α t The cumulative product of β t is the noise intensity.

8. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information guided diffusion model according to claim 7, characterized in that: In A6, the loss result is obtained by formula (7): in As a result of loss.

9. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information-guided diffusion model according to any one of claims 1 to 2, characterized in that: The predicted positioning result It is expressed by formula (8): in, is a normal distribution, I is the unit matrix, y 0：T-1 is the noise variable yt, and t∈[0,...,T-1]; The T is 1000.

10. The method for locating the left atrial appendage opening plane in 3D-TEE images based on a multi-scale information guided diffusion model according to any one of claims 1 to 2, characterized in that: The denoising model ∈ θ It consists of multiple fully connected layers embedded with time codes, and in each fully connected layer, the output vector of the fully connected layer is subjected to the Hadamard product operation with the time code, and the Softplus activation function is applied to the output to embed the time step. The fully connected layer embedded with time code is expressed by formula (8): l i =σ(g(l i-1 )☉g(t)) ......Equation (8); Among them l i is the denoising model ∈ θ The i-th layer output, l i-1 is the denoising model ∈ θ is the i-1th layer output, σ is the Softplus activation function, g(·) is the fully connected layer, and ⊙ is the Hadamard product.

Images