An ultrasound myocardial motion tracking method based on a data enhancement end-to-end multi-frame optical flow estimation network

Abstract

An ultrasonic myocardial motion tracking method based on a data enhancement end-to-end multi-frame optical flow estimation network, including 7 steps to obtain the forward optical flow of the to-be-estimated multi-ultrasound cardiogram. The ultrasonic myocardial motion tracking method based on the data enhancement end-to-end multi-frame optical flow estimation network discards the segmentation dependence, introduces a data enhancement strategy for the characteristics of the ultrasonic image, so that the network automatically focuses on the myocardial region during the training process, effectively suppresses the interference of non-myocardial tissue, and realizes the end-to-end tracking without segmentation. Meanwhile, based on the network structure of multi-frame optical flow estimation, the motion feature transmission mechanism is introduced, which can fuse multi-frame motion information and utilize the periodic characteristics of the heart motion for cross-period iterative correction, so that the application breaks through the limitation of short-time inter-frame tracking, and stable and accurate myocardial motion estimation can be obtained in a complex acoustic environment, such as low signal-to-noise ratio and boundary blur.

Description

Technical Field

[0001] This invention relates to the field of ultrasound myocardial motion technology, and in particular to an ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network with data augmentation. Background Technology

[0002] Ultrasound imaging technology offers advantages such as non-invasiveness, real-time imaging, bedside operation, and multiple imaging modes. However, its inherent limitations also lead to problems such as low image contrast, speckle noise, and artifacts. In the field of echocardiography-based myocardial motion tracking, image quality is often limited by factors such as low signal-to-noise ratio and blurred boundaries. This causes information from non-myocardial regions to severely interfere with the tracking process, affecting the stability and accuracy of tracking, and thus impacting myocardial strain assessment.

[0003] Existing echocardiography motion tracking methods typically employ a segmentation + tracking process, which is convenient and effective in ensuring the accuracy of the tracking method. Recently, Andreas Østvik et al. designed a fully automated myocardial motion analysis framework that includes view judgment, event detection, myocardial segmentation, motion tracking, and strain estimation (Østvik A, Salte IM, Smistad E, et al. Myocardial function imaging in echocardiography using deep learning[J]. ieee transactions on medical imaging,2021,40(5):1340-1351.). The motion tracking network is designed based on PWC-Net and EchoPWC-Net is proposed. By using simulated synthesized ultrasound images and related enhancements, the model learns the feature distribution of echocardiography and has a certain degree of adaptability to image artifacts. This method has two technical drawbacks: (1) In order to improve the accuracy of tracking, a segmentation + tracking process is adopted. However, this process increases the dependence of the tracking results on the segmentation performance. (2) This method only tracks the short-term myocardial motion estimation model between adjacent frames, without considering the continuous and periodic myocardial motion information, which makes it difficult to maintain the consistency of overall myocardial tissue motion tracking.

[0004] Therefore, in order to address the shortcomings of existing technologies, it is essential to provide an ultrasound myocardial motion tracking method based on a data-enhanced end-to-end multi-frame optical flow estimation network. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for ultrasound myocardial motion tracking based on a data-augmented end-to-end multi-frame optical flow estimation network. This method can achieve stable and accurate myocardial motion estimation even in complex acoustic environments.

[0006] The above-mentioned objectives of the present invention are achieved through the following technical measures: A method for ultrasound myocardial motion tracking based on an end-to-end multi-frame optical flow estimation network with data augmentation is provided, comprising the following steps: S1. Acquire multiple frames of echocardiograms arranged in time sequence, and generate a dense optical flow map of myocardial motion based on the echocardiograms; the echocardiograms contain multiple complete cardiac cycles; S2. Perform data augmentation on the echocardiogram to obtain an enhanced image, and combine the enhanced image with the echocardiogram to form a training dataset; S3. Take training images of the same cardiac cycle as processing images, and input the corresponding weighted feature encoder to obtain the corresponding feature map; calculate the pixel-wise inner product of the feature map to obtain the 4D bidirectional correlation volume; fuse and encode all 4D bidirectional correlation volumes through the correlation encoder; at the same time, extract the context information features of the intermediate frames in the processing image. S4. Input the context information features and fusion encoding from S3 into multiple iterative update modules. The multiple iterative update modules are connected in series in time sequence, and motion features are configured in each iterative update module. S5. Divide the T-frame processed images into multiple frame triples in chronological order. Each set of three adjacent processed images constitutes a frame triple, and each frame triple corresponds to an iterative update module. Perform training and iterative update on each frame triple. S6. Obtain motion propagation features for each frame triplet based on the current forward optical flow, reverse optical flow and adjacent motion state features, and update the network weights based on the loss function until the convergence condition is met, thus obtaining the trained backend-to-end multi-frame optical flow estimation network. S7. Input the multiple echocardiograms to be estimated into the training backend-to-end multi-frame optical flow estimation network to obtain the forward optical flow.

[0007] Preferably, S1 specifically involves: acquiring multiple frames of echocardiograms arranged in chronological order, preprocessing the motion field data of the echocardiograms to obtain the corresponding sparse optical flow map of myocardial motion, and then performing multiple interpolation processes on the sparse optical flow map to obtain a dense optical flow map of myocardial motion.

[0008] Preferably, S2 specifically involves: dividing the echocardiogram into two parts, performing data augmentation on one part to obtain an enhanced echocardiogram, and using the enhanced echocardiogram and the other part of the original echocardiogram together as training images to form a training dataset.

[0009] Preferably, the above data augmentation method includes the following steps: S2.1. Each echocardiogram is processed through a segmentation mask to obtain multiple myocardial regions and multiple background regions; S2.2. Randomly overlay the myocardial region of one echocardiogram with the background region of another echocardiogram to obtain multiple enhanced echocardiograms. S2.3, The timing of the myocardial region in enhanced echocardiography is used as the timing of enhanced echocardiography; S2.4. Enhanced echocardiograms and another portion of original echocardiograms are used together as training images to form a training dataset; the proportion of enhanced echocardiograms in the training dataset is... b %, and 3≤ b≤10 .

[0010] Preferably, the above S3 is obtained by the following steps: S3.1. Take all training images within the same complete cardiac cycle as processing images. Let the processing images be T frames, where T > 3, and T is an integer. S3.2. Input the T frames of processed images into T shared-weight feature encoders, and each feature encoder outputs a feature map. S3.3 Pair the T feature maps in time sequence and calculate the pixel-wise inner product to obtain T-1 4D bidirectional correlation volumes. Input all 4D bidirectional correlation volumes into the cost correlation encoder for fusion encoding to obtain T-2 fused codes. At the same time, extract T-2 context information features from the intermediate frames in the T-frame processed image through the context feature extraction module.

[0011] Preferably, S4 specifically involves: inputting T-2 context information features and T-2 fusion codes into T-2 iterative update modules respectively, wherein the T-2 iterative update modules are connected in sequence according to time order, and a motion feature is set in each iterative update module, where k is the current iteration number.

[0012] Preferably, S5 specifically involves: dividing the T-frame processed images into multiple frame triplets in chronological order, with each set of three adjacent processed images forming a frame triplet, and each frame triplet corresponding to a separate iterative update module. Training and iterative updates are performed on each frame triplet, and the processed image of the frame triplet is defined as { I a-1 , Ia , I a+1}

[0013] In S3, the frame triple { I a-1 , I a , I a+1 After being fused and encoded by the correlation encoder, the corresponding correlation features are obtained. F corr〈a-1〉 , F corr〈a〉 , F corr〈a+1〉}

[0014] In step S3, the context feature extraction module extracts the {{} of the frame triples. I a-1 , I a , I a+1 The contextual information features of} are used to obtain { g 〈a-1〉 , g 〈a〉 , g 〈a+1〉}

[0015] Preferably, S6 is performed by the following steps: S6.1 Initialize the motion features of all iterative update modules M k a Initialize the hidden state of the iterative update module. h k Enter S6.2; S6.2, Order k=1,k This represents the current iteration number; S6.3, Use the warp algorithm to convert the forward optical flow of each frame triplet in the current frame. f k a→a+1 Reverse optical flow f k a→a-1 Features of adjacent motion states M k a-1 and motion state characteristics M k a+1 By fusing the data, we can obtain motion information from adjacent time intervals. m k f and time motion information m kb ; S6.4, Transfer time motion information m k f Time motion information m k b With motion characteristics M a k Connect the points to obtain the motion propagation characteristics. F k m ; S6.5, Relevant Quantity Characteristics F corr〈a〉 and characteristics of motion propagation F k m The input motion encoder is used for assisted encoding by an optical flow encoder to generate the final fused coded motion features. F k M ; S6.6, Using fused encoding motion features F k M Contextual features g a Hidden state h k The hidden state of the update module is updated in response to input. h k+1 ; S6.7, By hiding the state h k+1 Predicted forward optical flow f k+1 a→a+1 ; S6.8, Judgment k Number of iterations with target N When comparing, k < N At that time, the forward optical flow is calculated using the loss function. f k +1 a→a+1 The optical flow prediction loss value obtained from S1 is used to backpropagate and update the weights of the motion encoder and iteratively update the weights of the module, then proceeds to S6.9; when k = N hour ，Using the current iteration update module as the post-training iteration update module and the current motion encoder as the post-training motion encoder, the post-training motion encoder, the post-training iteration update module, the feature encoder, and the cost quantity correlation encoder constitute the training backend-to-end multi-frame optical flow estimation network, and proceed to S7; S6.9, Order k=k+1 And return to S6.3.

[0016] Preferably, the above loss function is expressed by the following formula: ; in, L The loss value. f gt,a→a+1 The dense optical flow map obtained from S1, γ This is the attenuation coefficient.

[0017] This invention discloses an end-to-end multi-frame optical flow estimation network-based ultrasound myocardial motion tracking method, comprising the following steps: S1, acquiring multiple frames of echocardiograms arranged in temporal order, and generating a dense optical flow map of myocardial motion based on the echocardiograms; the echocardiograms contain multiple complete cardiac cycles; S2, performing data augmentation on the echocardiograms to obtain augmented images, and using the augmented images and the echocardiograms together to form a training dataset; S3, taking training images of the same cardiac cycle as processing images, inputting the corresponding processed images into a feature encoder with shared weights to obtain corresponding feature maps; calculating the pixel-wise inner product of the feature maps to obtain a 4D bidirectional correlation volume; fusing and encoding all 4D bidirectional correlation volumes through a correlation encoder; and simultaneously extracting contextual information features from intermediate frames in the processed images. S4. Input the context information features and fusion encoding from S3 into multiple iterative update modules, which are sequentially connected in time. Configure motion features in each iterative update module. S5. Divide the T-frame processed images into multiple frame triplets in time order. Each frame triplet consists of three adjacent processed images, and each frame triplet corresponds to a separate iterative update module. Perform training and iterative updates on each frame triplet. S6. Obtain motion propagation features for each frame triplet based on the current forward optical flow, reverse optical flow, and adjacent motion state features. Update the network weights based on the loss function until the convergence condition is met, thus obtaining the trained end-to-end multi-frame optical flow estimation network. S7. Input the multiple echocardiograms to be estimated into the trained end-to-end multi-frame optical flow estimation network to obtain the forward optical flow. This ultrasound myocardial motion tracking method based on data augmentation-based end-to-end multi-frame optical flow estimation network abandons segmentation dependence. By introducing a data augmentation strategy targeting the characteristics of ultrasound images, the network automatically focuses on the myocardial region during training, effectively suppressing interference from non-myocardial tissues, and achieving end-to-end tracking without segmentation. Meanwhile, this invention introduces a motion feature transfer mechanism based on a network structure for multi-frame optical flow estimation, which can fuse multi-frame motion information and utilize the periodic characteristics of cardiac motion for cross-cycle iterative correction. Therefore, this invention breaks through the limitations of short-time inter-frame tracking, enabling stable and accurate myocardial motion estimation to be obtained even in complex acoustic environments, such as low signal-to-noise ratio and blurred boundaries. Attached Figure Description

[0018] The invention will be further described with reference to the accompanying drawings, but the contents of the drawings do not constitute any limitation on the invention.

[0019] Figure 1 This is a flowchart of the ultrasound myocardial motion tracking method based on a data-enhanced end-to-end multi-frame optical flow estimation network according to the present invention.

[0020] Figure 2 This is a flowchart of S2-S8 of Example 2.

[0021] Figure 3 This is a pseudo-color visualization comparison of the optical flow field predictions made by this invention and existing technologies.

[0022] Figure 4 This is a regression scatter plot of the overall longitudinal strain of the present invention compared with different devices.

[0023] Figure 5 This is a flowchart of S2-S8 of Example 3. Detailed Implementation

[0024] The technical solution of the present invention will be further described in conjunction with the following embodiments.

[0025] Example 1 A method for ultrasound myocardial motion tracking based on an end-to-end multi-frame optical flow estimation network with data augmentation, such as... Figure 1 As shown, it includes the following steps: S1. Acquire multiple frames of echocardiograms arranged in time sequence, and generate a dense optical flow map of myocardial motion based on the echocardiograms; the echocardiograms contain multiple complete cardiac cycles; S2. Perform data augmentation on the echocardiogram to obtain an enhanced image, and combine the enhanced image with the echocardiogram to form a training dataset. S3. Take training images of the same cardiac cycle as processing images, and input the corresponding weighted feature encoder to obtain the corresponding feature map; calculate the pixel-wise inner product of the feature map to obtain the 4D bidirectional correlation volume; fuse and encode all 4D bidirectional correlation volumes through the correlation encoder; at the same time, extract the context information features of the intermediate frames in the processing image. S4. Input the context information features and fusion encoding from S3 into multiple iterative update modules. The multiple iterative update modules are connected in sequence according to time order, and motion features are configured in each iterative update module. S5. Divide the T-frame processed images into multiple frame triples in chronological order. Each set of three adjacent processed images constitutes a frame triple, and each frame triple corresponds to an iterative update module. Perform training and iterative update on each frame triple. S6. Obtain motion propagation features for each frame triplet based on the current forward optical flow, reverse optical flow and adjacent motion state features, and update the network weights based on the loss function until the convergence condition is met, thus obtaining the trained backend-to-end multi-frame optical flow estimation network. S7. Input the multiple echocardiograms to be estimated into the training end-to-end multi-frame optical flow estimation network to obtain the forward optical flow, and the forward optical flow constitutes... Figure 1 Mid-optical flow diagram.

[0026] S1 specifically involves: acquiring multiple frames of echocardiograms arranged in chronological order, preprocessing the motion field data of the echocardiograms to obtain the corresponding sparse optical flow map of myocardial motion, and then performing multiple interpolation processes on the sparse optical flow map to obtain a dense optical flow map of myocardial motion. In this embodiment, three interpolation processes were specifically performed.

[0027] S2 specifically involves dividing the echocardiogram into two parts, performing data augmentation on one part to obtain an enhanced echocardiogram, and using the enhanced echocardiogram and the other part of the original echocardiogram together as training images to form a training dataset.

[0028] Data augmentation methods include the following steps: S2.1. Each echocardiogram is processed through a segmentation mask to obtain multiple myocardial regions and multiple background regions; S2.2. Randomly overlay the myocardial region of one echocardiogram with the background region of another echocardiogram to obtain multiple enhanced echocardiograms. S2.3, The timing of the myocardial region in enhanced echocardiography is used as the timing of enhanced echocardiography; S2.4. Enhanced echocardiograms and another portion of original echocardiograms are used together as training images to form a training dataset; the proportion of enhanced echocardiograms in the training dataset is... b %, and 3≤ b≤10 In this embodiment, the percentage is 5%.

[0029] It should be noted that the purpose of enhancing the pre-processed echocardiogram in this invention is to weaken the influence of non-myocardial regions and enhance the feature learning of the main myocardial structure. This effectively improves the robustness and cross-sample generalization performance of the model, allowing the network to focus on myocardial region motion, thus achieving a myocardial tracking method without segmentation. This invention inputs the enhanced echocardiogram sequence into a pre-designed end-to-end multi-frame optical flow estimation network, introduces a motion feature transfer mechanism, and fuses multi-frame motion features. This enables the network to iteratively correct instantaneous motion estimation using cross-period information, overcoming the limitation of traditional optical flow methods to short-term inter-frame tracking and achieving more accurate myocardial tracking.

[0030] This is because pixel-by-pixel segmentation and annotation of targets within a sector is not only extremely time-consuming, but also limited by the fact that sector boundaries are often blurry in low-quality images, making it difficult to obtain a large amount of accurate annotation data. Therefore, this invention employs a semi-supervised learning strategy, which only performs pixel-by-pixel segmentation and annotation of targets within a sector. b % of ultrasound image segmentation masks are supervised using available sector segmentation masks within the training set. b The % data does not rely on fine annotation. Background augmentation occurs only on random data from the training dataset. bThe augmentation is performed on the % sample, while the remaining images remain unchanged. This balances the diversity of the augmentation with the authenticity of the original data, and avoids the model from becoming overly reliant on the data distribution generated by the augmentation.

[0031] Traditional methods often employ a "segmentation-then-tracking" process, making their results highly dependent on the accuracy of myocardial segmentation. Any segmentation deviation will propagate and accumulate in the tracking stage. Furthermore, existing optical flow methods typically only utilize two adjacent frames for short-term motion estimation, lacking constraints from global temporal information and struggling to address the transient blurring caused by speckle noise or periodic deformation of the myocardium in ultrasound images. This invention addresses this by designing a data augmentation strategy tailored to the characteristics of ultrasound images, enabling the network to automatically focus on the myocardial region and effectively reduce the influence of non-myocardial tissues. This eliminates the need for the traditional "segmentation-then-tracking" process, avoiding the accumulation of segmentation errors and constructing a novel end-to-end myocardial motion analysis method. A multi-frame end-to-end multi-frame optical flow estimation network is constructed, introducing a motion feature transfer mechanism and fusing multi-frame spatiotemporal information. Utilizing the periodicity of cardiac motion, cross-cycle motion correction is achieved, overcoming the limitations of traditional optical flow methods that rely solely on short-term estimation of adjacent frames, significantly improving the accuracy and robustness of myocardial motion tracking.

[0032] S3 is obtained through the following steps: S3.1. Take all training images within the same complete cardiac cycle as processing images. Let the number of processing images be T frames, where T > 3, and T is an integer. S3.2. Input the T frames of processed images into T shared-weight feature encoders, and each feature encoder outputs a feature map. S3.3 Pair the T feature maps in time sequence and calculate the pixel-wise inner product to obtain T-1 4D bidirectional correlation volumes. Input all 4D bidirectional correlation volumes into the cost correlation encoder for fusion encoding to obtain T-2 fused codes. At the same time, extract T-2 context information features from the intermediate frames in the T-frame processed image through the context feature extraction module.

[0033] S4 specifically involves inputting T-2 contextual information features and T-2 fusion codes into T-2 iterative update modules. These T-2 iterative update modules are then concatenated sequentially, with a motion feature set within each module. S5 specifically involves dividing the T frames of processed images into multiple frame triplets in chronological order. Each frame triplet consists of three adjacent processed images, and each frame triplet corresponds to a separate iterative update module. Training and iterative updates are performed on each frame triplet. The processed image of each frame triplet is defined as { I a-1 , I a , I a+1}

[0034] In S3, the frame triple { I a-1 , I a , I a+1 After being fused and encoded by the correlation encoder, the corresponding correlation features are obtained. F corr〈a-1〉 , F corr〈a〉 , F corr〈a+1〉 In S3, the context feature extraction module extracts the {{} of the frame triples. I a-1 , I a , I a+1 The contextual information features of} are used to obtain { g 〈a-1〉 , g 〈a〉 , g 〈a+1〉}

[0035] It should be noted that the purpose of concatenating multiple iterative update modules to process optical flow estimation for more than three frames is to adapt to the myocardial motion tracking task of this invention. Because temporal information is limited to each individual unit, simple concatenation cannot pass on motion context information. Moreover, in order to propagate motion information between adjacent modules, we add a motion feature to each iterative update module to store motion information, and train the motion encoder by cyclically fusing the motion features of adjacent modules through the motion propagation module.

[0036] S6 is performed by the following steps: S6.1 Initialize the motion features of all iterative update modules M k a Initialize the hidden state of the iterative update module. h k Enter S6.2; S6.2, Order k=1,k This represents the current iteration number; S6.3, Use the warp algorithm to convert the forward optical flow of each frame triplet in the current frame. f k a→a+1 Reverse optical flow f k a→a-1 Features of adjacent motion states M k a-1 and motion state characteristics Mk a+1 By fusing the data, we can obtain motion information from adjacent time intervals. m k f and time motion information m k b ; S6.4, Transfer time motion information m k f Time motion information m k b With motion characteristics M a k Connect the points to obtain the motion propagation characteristics. F k m ; S6.5, Relevant Quantity Characteristics F corr〈a〉 and characteristics of motion propagation F k m The input motion encoder is used for assisted encoding by an optical flow encoder to generate the final fused coded motion features. F k M ; S6.6, Using fused encoding motion features F k M Contextual features g a Hidden state h k The hidden state of the update module is updated in response to input. h k+1 ; S6.7, By hiding the state h k+1 Predicted forward optical flow f k+1 a→a+1 ; S6.8, Judgment k Number of iterations with target N When comparing, k < N At that time, the forward optical flow is calculated using the loss function. f k +1 a→a+1 The optical flow prediction loss value obtained from S1 is used to backpropagate and update the weights of the motion encoder and iteratively update the weights of the module, then proceeds to S6.9; when k =N hour ， Using the current iteration update module as the post-training iteration update module and the current motion encoder as the post-training motion encoder, the post-training motion encoder, the post-training iteration update module, the feature encoder, and the cost quantity correlation encoder constitute the training backend-to-end multi-frame optical flow estimation network, and proceed to S7; S6.9, Order k=k+1 And return to S6.3.

[0037] The time motion information in S6 m k f and time motion information m k b We obtain it from the following formula: .

[0038] Among them, the motion propagation characteristics in S6 F k m We obtain it from the following formula: .

[0039] Among them, the motion propagation characteristics in S6 F k M We obtain it from the following formula: .

[0040] The hidden state of the iterative update module in S6 h k+1 We obtain it from the following formula: .

[0041] It should be noted that in each iteration of this invention, bidirectional motion state features are included. M k a-1 and motion state characteristics M k a+1 Fusion encoding is performed, specifically through a correlation feature encoder (CorrEncoder) and an optical flow feature encoder (FlowEncoder) respectively, and then the motion features are fused and encoded through a motion encoder (MotionEncoder). F k M To predict displacement changes in bidirectional optical flow. This invention utilizes motion propagation characteristics. F k mThe original motion encoder and iterative update module are input for training and learning to expand the model's temporal receptive field. The iterative update module uses a gated recurrent unit (GRU) update mechanism, introducing a hidden state. h k Buffering and transmitting intermediate features of optical flow, with the center frame I a Contextual features g a and fusion encoding motion features F k M As input, the hidden state is updated incrementally within the loop structure.

[0042] The loss function is expressed by the following formula: ; in, L The loss value. f ga,t→a+1 The dense optical flow map obtained from S1, γ This is the attenuation coefficient.

[0043] It should be noted that, due to the differences between myocardial motion data and natural data, the true label of the optical flow field is predicted unidirectionally. The bidirectional motion features of this invention are only used as an optimization tool during the training phase and are not used to evaluate the reverse prediction of the optical flow field. Since the proposed network needs to update the weights of the predicted optical flow stepwise from 0 in subsequent iterations, the attenuation coefficient γ is set to 0.85 in this invention.

[0044] This ultrasound myocardial motion tracking method, based on an end-to-end multi-frame optical flow estimation network using data augmentation, abandons segmentation dependence. By introducing a data augmentation strategy tailored to the characteristics of ultrasound images, the network automatically focuses on the myocardial region during training, effectively suppressing interference from non-myocardial tissues and achieving segmentation-free end-to-end tracking. Furthermore, this invention, based on a multi-frame optical flow estimation network structure, introduces a motion feature transfer mechanism, enabling the fusion of multi-frame motion information and cross-cycle iterative correction utilizing the periodic characteristics of cardiac motion. Therefore, this invention overcomes the limitations of short-time inter-frame tracking, achieving stable and accurate myocardial motion estimation even in complex acoustic environments, such as low signal-to-noise ratios and blurred boundaries.

[0045] Example 2 An ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network, as described in Example 1, is as follows: Figure 2 The other features are the same as in Example 1, except that this example specifically uses a complete cardiac cycle of 5 frames for image processing. The shape of the processed image is... The 5 frames of images are processed sequentially as follows: I t-2 ,I t-1 , I t , I t+1 , I t+2 The image processing takes five frames as input and five feature encoders with shared weights. Each frame outputs a shape of... Feature map, where set Then, the inner product of each pair of feature maps is calculated pixel-wise, and visual similarity is measured to construct their respective 4D bidirectional correlation volumes. Corr 〈t-2，t-1〉 , Corr 〈t-1，t〉 , Corr 〈t，t+1〉 , Corr 〈t+1，t+2〉} .Will{ Corr 〈t-2，t-1〉 , Corr 〈t-1，t〉 , Corr 〈t，t+1〉 , Corr 〈t+1，t+2〉} The input cost correlation encoder further fuses the encoding to obtain { F k corr〈t-1〉 , F k corr〈t〉 , F k corr〈t+1〉}

[0046] Five frames of processed images were extracted using the Context Encoder module. I t-2 , I t-1 , I t , I t+1 , I t+2 The intermediate frame in} I t-1 , I t , I t+1} context information features { g t-1 , g t , gt+1} serves as the input for iterative updates.

[0047] In S5, the 5 frames of processed images are divided into 3 overlapping frame triplets in chronological order. I t-2 , I t-1 , I t},{ I t-1 , I t , I t+1}and{ I t , I t+1 , I t+2 In this embodiment, in the frame triple { I t-2 , I t-1 , I t t-1 in} is the same as t-1 in Example 1. I a-1 , I a , I a+1 In the frame triple {}, 'a' is in the frame triple { I t-1 , I t , I t+1 The t in} is the same as the t in Example 1. I a-1 , I a , I a+1 In the frame triple {}, 'a' is in the frame triple { I t , I t+1 , I t+2 t+1 in} is the same as t+1 in Example 1. I a-1 , I a , I a+1} in 'a'. Figure 4 middle,{ I t-2 , I t-1 , I t} Input the first iterative update module on the left, { I t-1 , I t , I t+1} Input the second iterative update module from the left, { I t , I t+1 , I t+2 Enter the third iterative update module from the left.

[0048] For ease of understanding, this embodiment uses { I t-1 , I t , I t+1 Taking a single triplet as an example, motion features are first randomly initialized and gradually updated through training, then used to obtain communication motion features from neighboring sub-modules. Then, in each iteration, the current forward optical flow is fused using a warp algorithm. f k t→t+1 Reverse optical flow f k t→t-1 Features of adjacent motion states M k t-1 and motion state characteristics M k t+1 To obtain motion information in adjacent time periods m k f and time motion information m k b Then combine them with the motion features of the current frame. M t k Connecting them together yields the motion propagation characteristics. F k m In this part, t is the same as a in Example 1.

[0049] In each iteration, we fuse and encode the bidirectional motion features using a correlation encoder (CorrEncoder) and an optical flow encoder (FlowEncoder) respectively, and then fuse the encoded motion features using a motion encoder (MotionEncoder). To predict the displacement changes of bidirectional optical flow.

[0050] Finally, we input this feature into the original motion encoder and iterative update module for training, thereby expanding the model's temporal receptive field. The iterative update module borrows the update mechanism of the Gated Recurrent Unit (GRU) and introduces a hidden state. Buffering and transmitting intermediate features of optical flow, with the center frame Contextual features and motion characteristics As input, the hidden state is updated incrementally within the loop structure. In this embodiment, the target number of iterations N is set to 12.

[0051] The comparison results between the present invention and existing technical methods are shown in Table 1. Figure 3 As shown, the evaluation parameter is the average endpoint error. The results demonstrate that this invention outperforms existing methods in terms of tracking accuracy.

[0052] Table 1. Comparison of mean endpoint errors of different optical flow estimation methods on the public datasets SimulatedST and CAMUS (mean (standard deviation), unit: ...). ) The tracking network results were applied to myocardial function assessment, calculating the degree of myocardial strain in each segment, and compared with clinical system results to evaluate correlation indicators. The results are as follows: Figure 4 As shown. The evaluation parameter is the global longitudinal strain (GLS).

[0053] Through Table 1, Figure 3 and Figure 4 As can be seen, the present invention has a high correlation with existing clinical system methods in clinical trials, achieving precise tracking and strain assessment of myocardial motion, and systematically verifying the accuracy and robustness of the present invention.

[0054] Example 3 An ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network, as described in Example 1, is as follows: Figure 5 The other features are the same as in Example 1, except that this example specifically uses a complete cardiac cycle of 6 frames for image processing. The shape of the processed image is... 6 frames of processed images are { I t-2 , I t-1 , I t , I t+1 , I t+2 , It+3 The image processing involves 6 frames. The input image is processed by 6 FeatureEncoders with shared weights. Each frame outputs a shape... Feature map, where set .

[0055] Then, the inner product of each pair of feature maps is calculated pixel-wise, and visual similarity is measured to construct their respective 4D bidirectional correlation volumes. Corr 〈t-2，t-1〉 , Corr 〈t-1，t〉 , Corr 〈t，t+1〉 , Corr 〈t+1，t+2〉 , Corr 〈t+2，t+3〉} There are 5 in total. (The rest of the text appears to be a list of numbers and symbols, possibly related to a document or instruction Corr 〈t-2，t-1〉 , Corr 〈t-1，t〉 , Corr 〈t，t+1〉 , Corr 〈t+1，t+2〉 , Corr 〈t+2，t+3〉} The input cost correlation encoder further fuses the encoding to obtain { F k corr〈t-1〉 , F k corr〈t〉 , F k corr〈t+1〉 , F k corr〈t+2〉}

[0056] Five frames of processed images were extracted using the Context Encoder module. I t-2 , I t-1 , I t , I t+1 , I t+2 , I t+3 The intermediate frame in} I t-1 , I t , I t+1 ,I t+2} context information features { g t-1 , g t , g t+1 , g t+2} serves as the input for iterative updates.

[0057] In S5, the 6-frame processed image is divided into 4 overlapping frame triplets in chronological order. I t-2 , I t-1 , I t},{ I t-1 , I t , I t+1}、{ I t , I t+1 , I t+2}and{ I t+1 , I t+2 , I t+3},in{ I t-2 , I t-1 , I t} in which t-1 is a in Example 1, { I t-1 , I t , I t+1 In}, t is the same as a in Example 1, { I t , I t+1 , I t+2 In}, t+1 is a in Example 1, { I t+1 , I t+2 , I t+3 In}, t+2 is equivalent to a in Example 1. Figure 5 middle,{ I t-2 , I t-1 ,I t} Input the first iterative update module on the left, { I t-1 , I t , I t+1} Input the second iterative update module from the left, { I t , I t+1 , I t+2} Input the third iterative update module from the left, { I t+1 , I t+2 , I t+3 Enter the fourth iterative update module from the left. Other processing methods in this embodiment are the same as in Example 2.

[0058] This ultrasound myocardial motion tracking method, based on an end-to-end multi-frame optical flow estimation network using data augmentation, abandons segmentation dependence. By introducing a data augmentation strategy tailored to the characteristics of ultrasound images, the network automatically focuses on the myocardial region during training, effectively suppressing interference from non-myocardial tissues and achieving segmentation-free end-to-end tracking. Furthermore, this invention, based on a multi-frame optical flow estimation network structure, introduces a motion feature transfer mechanism, enabling the fusion of multi-frame motion information and cross-cycle iterative correction utilizing the periodic characteristics of cardiac motion. Therefore, this invention overcomes the limitations of short-time inter-frame tracking, achieving stable and accurate myocardial motion estimation even in complex acoustic environments, such as low signal-to-noise ratios and blurred boundaries.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for ultrasound myocardial motion tracking based on an end-to-end multi-frame optical flow estimation network with data augmentation, characterized in that, Includes the following steps: S1. Acquire multiple frames of echocardiograms arranged in time sequence, and generate a dense optical flow map of myocardial motion based on the echocardiograms; the echocardiograms contain multiple complete cardiac cycles; S2. Perform data augmentation on the echocardiogram to obtain an enhanced image, and combine the enhanced image with the echocardiogram to form a training dataset; S3. Take training images of the same cardiac cycle as processing images, and input the corresponding weighted feature encoder to obtain the corresponding feature map; calculate the pixel-wise inner product of the feature map to obtain the 4D bidirectional correlation volume; fuse and encode all 4D bidirectional correlation volumes through the correlation encoder; at the same time, extract the context information features of the intermediate frames in the processing image. S4. Input the context information features and fusion encoding from S3 into multiple iterative update modules. The multiple iterative update modules are connected in series in time sequence, and motion features are configured in each iterative update module. S5. Divide the T-frame processed images into multiple frame triples in chronological order. Each set of three adjacent processed images constitutes a frame triple, and each frame triple corresponds to an iterative update module. Perform training and iterative update on each frame triple. S6. Obtain motion propagation features for each frame triplet based on the current forward optical flow, reverse optical flow and adjacent motion state features, and update the network weights based on the loss function until the convergence condition is met, thus obtaining the trained backend-to-end multi-frame optical flow estimation network. S7. Input the multiple echocardiograms to be estimated into the training backend-to-end multi-frame optical flow estimation network to obtain the forward optical flow; S1 specifically involves: acquiring multiple frames of echocardiograms arranged in time sequence, preprocessing the motion field data of the echocardiograms to obtain the corresponding sparse optical flow map of myocardial motion, and then performing multiple interpolation processes on the sparse optical flow map to obtain a dense optical flow map of myocardial motion.

2. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to claim 1, characterized in that, S2 specifically involves: dividing the echocardiogram into two parts, performing data augmentation on one part to obtain an enhanced echocardiogram, and using the enhanced echocardiogram and the other part of the original echocardiogram together as training images to form a training dataset.

3. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to claim 2, characterized in that, The data augmentation method includes the following steps: S2.

1. Each echocardiogram is processed through a segmentation mask to obtain multiple myocardial regions and multiple background regions; S2.

2. Randomly overlay the myocardial region of one echocardiogram with the background region of another echocardiogram to obtain multiple enhanced echocardiograms. S2.3, The timing of the myocardial region in enhanced echocardiography is used as the timing of enhanced echocardiography; S2.

4. Enhanced echocardiograms and another portion of original echocardiograms are used together as training images to form a training dataset; the proportion of enhanced echocardiograms in the training dataset is... b %, and 3≤ b≤10 .

4. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to claim 3, characterized in that, S3 is obtained through the following steps: S3.

1. Take all training images within the same complete cardiac cycle as processing images. Let the processing images be T frames, where T > 3, and T is an integer. S3.

2. Input the T frames of processed images into T shared-weight feature encoders, and each feature encoder outputs a feature map. S3.3 Pair the T feature maps in time sequence and calculate the pixel-wise inner product to obtain T-1 4D bidirectional correlation volumes. Input all 4D bidirectional correlation volumes into the cost correlation encoder for fusion encoding to obtain T-2 fused codes. At the same time, extract T-2 context information features from the intermediate frames in the T-frame processed image through the context feature extraction module.

5. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to claim 4, characterized in that, Specifically, S4 involves inputting T-2 context information features and T-2 fusion codes into T-2 iterative update modules, with the T-2 iterative update modules connected in sequence according to time order, and setting a motion feature within each iterative update module.

6. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to claim 5, characterized in that, S5 specifically involves: dividing the T-frame processed images into multiple frame triplets in chronological order, with each set of three adjacent processed images forming a frame triplet, and each frame triplet corresponding to a separate iterative update module. Training and iterative updates are performed on each frame triplet, and the processed image of the frame triplet is defined as { I a-1 , I a , I a+1 } 7. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to claim 6, characterized in that: In S3, the frame triple { I a-1 , I a , I a+1 After being fused and encoded by the correlation encoder, the corresponding correlation features are obtained. F corr〈a-1〉 , F corr〈a〉 , F corr〈a+1〉 }; In step S3, the context feature extraction module extracts the {{} of the frame triples. I a-1 , I a , I a+1 The contextual information features of} are used to obtain { g 〈a-1〉 , g 〈a〉 , g 〈a+1〉 } 8. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to claim 7, characterized in that, S6 is performed by the following steps: S6.1 Initialize the motion features of all iterative update modules M k a Initialize the hidden state of the iterative update module. h k Enter S6.2; S6.2, Order k=1,k This represents the current iteration number; S6.3, Use the warp algorithm to convert the forward optical flow of each frame triplet in the current frame. f k a→a+1 Reverse optical flow f k a→a-1 Features of adjacent motion states M k a-1 and motion state characteristics M k a+1 By fusing the data, we can obtain motion information from adjacent time intervals. m k f and time motion information m k b ; S6.4, Transfer time motion information m k f Time motion information m k b With motion characteristics M a k Connect the points to obtain the motion propagation characteristics. F k m ; S6.5, Relevant Quantity Characteristics F corr〈a〉 and characteristics of motion propagation F k m The input motion encoder is used for assisted encoding by an optical flow encoder to generate the final fused coded motion features. F k M ; S6.6, Using fused encoding motion features F k M Contextual features g a Hidden state h k The hidden state of the update module is updated in response to input. h k+1 ; S6.7, By hiding the state h k+1 Predicted forward optical flow f k+1 a→a+1 ; S6.8, Judgment k Number of iterations with target N When comparing, k < N At that time, the forward optical flow is calculated using the loss function. f k+1 a→a+1 The optical flow prediction loss value obtained from S1 is used to backpropagate and update the weights of the motion encoder and iteratively update the weights of the module, then proceeds to S6.9; when k = N hour ， Using the current iteration update module as the post-training iteration update module and the current motion encoder as the post-training motion encoder, the post-training motion encoder, the post-training iteration update module, the feature encoder, and the cost quantity correlation encoder constitute the training backend-to-end multi-frame optical flow estimation network, and proceed to S7; S6.9, Order k=k+1 And return to S6.

3.

9. The ultrasound myocardial motion tracking method based on an end-to-end multi-frame optical flow estimation network according to any one of claims 1 to 8, characterized in that: The loss function is expressed by the following formula: ； in, L The loss value. f gt,a→a+1 The dense optical flow map obtained from S1, γ The attenuation coefficient is... f k a→a+1 Forward optical flow, T is the number of image frames processed. N The target number of iterations.

Images