Respiratory motion estimation method, apparatus, device, and medium

Abstract

The application discloses a respiratory motion estimation method, device and equipment and a medium, relates to the technical field of motion estimation analysis, adopts an unsupervised sparse-dense motion estimation framework with sequence motion consistency constraints, and can quickly and accurately realize motion estimation of a liver ultrasound image sequence under the influence of respiratory motion. A sparse-dense coarse-to-fine registration strategy based on sparse point guidance is used to accurately predict the motion field between adjacent respiratory state images, a sparse key point automatic detection guided rigid registration network is designed to automatically detect sparse key points from the images in an unsupervised manner, and a multi-source structured feature guided deformation densification network is constructed to predict the dense elastic deformation component of the motion between the images. In combination with the idea of motion decomposition and coincidence, sequence motion consistency constraints are constructed based on a small motion sequence between a plurality of adjacent respiratory states, the continuity of motion in the time flow is strengthened, and the motion estimation accuracy is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion estimation and analysis, and in particular to a method, device, equipment and medium for estimating respiratory motion based on sequence motion consistency constraints. Background Art

[0002] Affected by respiratory movement, the liver will produce spatial displacement and flexible deformation with a maximum amplitude of several centimeters. This displacement and deformation poses a huge challenge to the precise positioning of liver tumors during liver puncture biopsy surgery and radiofrequency ablation radiotherapy. In clinical practice, some work has adopted alternative strategies such as abdominal compression, breath holding, and respiratory gating to limit the respiratory movement of the liver, but they cannot completely eliminate the flexible deformation of the liver, and there are shortcomings such as increased treatment time and causing physical discomfort to patients. Another part of the work adopts an image-guided non-invasive strategy to estimate the flexible movement of the liver caused by respiration. It provides comprehensive spatiotemporal information of the target structure through multiple three-dimensional data sampled over a period of time, estimates the correspondence between volume data at different times / respiratory states, compensates for the flexible movement caused by respiration, and thus assists clinical treatment and diagnosis.

[0003] Motion estimation is to establish the corresponding deformation field between images at different time points and estimate the movement of the target structure, thereby allowing subsequent evaluation and analysis of its structural characteristics, and ultimately assisting in improving clinical treatment. Affected by the amplitude and depth of respiratory movement, the amplitude of movement between two moments that are far apart in the respiratory movement cycle (such as the end of exhalation and the end of inhalation) is larger. According to statistics, the average amplitude of liver movement caused by respiratory movement in the head and foot direction is 13-25mm during shallow breathing, and can reach 55mm during deep breathing, resulting in large-scale flexible deformation of the liver and blood vessels in the liver during treatment, interfering with the doctor's treatment process.

[0004] Therefore, it is crucial to accurately estimate and compensate for large-scale motion. Furthermore, respiratory motion exhibits cyclical biological characteristics, and existing respiratory motion estimation algorithms mostly focus on establishing correspondences between single pairs of static images, ignoring the temporal continuity characteristics of the image sequence. This lacks motion continuity constraints in the temporal dimension, making it difficult to obtain smooth, continuous temporal motion.

[0005] The corresponding deformation fields between images at different time points can be realized through image registration algorithms. Registration algorithms find the optimal spatial deformation field between two images at different times by solving an objective function composed of an image similarity measure and a deformation regularization constraint.

[0006] Traditional registration algorithms typically employ computationally intensive iterative optimization strategies, which are time-consuming and time-consuming. Deep learning-based registration algorithms use convolutional neural networks and their variants to extract latent representations of the image pairs to be registered and calculate the corresponding motion fields. This approach offers high computational efficiency and significantly reduces registration time. However, their performance is limited by the receptive field size of the convolution kernel, making them ineffective for estimating large motions. Furthermore, their excessive pursuit of image intensity similarity often leads to unrealistic physiological distortions and miscalculations.

[0007] Deep learning-based registration algorithms aim to find the optimal spatial correspondence between images. One approach generates a deformation field by detecting key points with specific spatial structure and anatomical characteristics. These key points can be manually annotated landmarks or derived using handcrafted key point extraction strategies based on geometric features, image intensity, and other information. However, these methods are limited by their task sensitivity and high time cost.

[0008] Such sparse point-based registration methods are limited by the scale of keypoints, as the sparse displacement information at discrete points is insufficient to capture the more complex and extensive flexible motions caused by breathing. Furthermore, without incurring the additional cost of introducing supervisory information such as segmentation and a gold standard for labeled points, automatic keypoint detection using only registration similarity as a constraint makes it difficult to establish a direct association between keypoints and salient regions such as blood vessels, making alignment in key structural regions difficult to guarantee. Another class of methods estimates the optimal dense deformation field by optimizing a registration objective function composed of an image similarity measure and a regularization constraint. These methods are limited by the receptive field size of the convolution kernel, resulting in limited prediction accuracy for large-scale deformations. Furthermore, most methods directly generate full-image deformation fields by optimizing global registration similarity losses such as MSE and NCC, which are intensity-based, lacking a focus on salient anatomical structures and regions.

[0009] Some algorithms use gold standards, such as segmentation labels, to provide supervisory information for salient structural regions. However, this increases the requirements for data collection and places additional pressure on clinical experts to label the images. Furthermore, because blood within blood vessels weakly reflects ultrasound waves, vascular regions appear as low-grayscale, hypoechoic areas in ultrasound images. This ultimately results in a low contribution of vascular regions to the global dense deformation field prediction based solely on image intensity similarity loss, and the entire optimization process places greater emphasis on the alignment of high-intensity regions such as the liver edge. A small number of works combine point-based and image-based registration methods to leverage the strengths of both approaches. However, the sparse points currently used in such algorithms are mostly obtained through manual annotation or traditional feature extraction methods, which have limitations such as high time costs. Furthermore, the integration of key points and images is limited to embedding and combining different constraints at the loss function level, failing to capture the underlying deep connections between the two.

[0010] Registration algorithms can be used to establish dense motion fields between images at different times to achieve motion estimation. Some work, building on this registration framework, focuses on the sequential characteristics of image sequences. This work aims to exploit the potential temporal features in image sequences by establishing temporal feature extraction modules to analyze the temporal patterns of target anatomical structures within the images. While these methods consider sequential characteristics, they still primarily use single-step prediction methods based on image registration to predict the motion field between a single image pair, which can easily lead to inaccurate estimates of large motions caused by breathing.

[0011] It can be seen that most current deep learning-based motion estimation algorithms introduce registration algorithms to model the dense motion field between images under different respiratory states. Their effectiveness is mainly limited by the registration algorithm's deformation field modeling capabilities. However, existing deep learning registration algorithms based on points, images, and a combination of the two have room for improvement in considering ultrasound images with poor image quality and large respiratory motion. Therefore, existing deep learning-based motion estimation algorithms still have the following shortcomings:

[0012] 1. Although the method of modeling dense deformation fields based solely on image intensity similarity as an optimization benchmark can obtain global deformation information at a relatively fast computational speed, its effectiveness is largely limited by image quality. The accuracy of the predicted deformation field is limited in ultrasound images with poor image quality and severe speckle noise.

[0013] 2. Considering that the liver has a large range of motion due to respiratory motion, this is reflected at the image level as a large information difference between the two images to be registered, especially for images at the two end points. This results in the existing methods being ineffective in estimating large-scale motion.

[0014] 3. A small number of works build temporal feature extraction modules based on the motion field between a single set of static images. Although they can dig deeper into the potential temporal features in image sequences, they have high requirements for the integrity and standardization of image sequences, and their usage scenarios are greatly limited. Summary of the Invention

[0015] In view of the above problems, the present invention provides a respiratory motion estimation method, apparatus, device, and medium for overcoming or at least partially resolving these problems. The method is particularly suitable for estimating and analyzing liver flexible motion based on three-dimensional liver ultrasound sequences. This method overcomes the shortcomings of existing deep learning-based motion estimation algorithms, which have limited effectiveness in estimating large-amplitude respiratory motion and insufficiently exploit potential temporal features within the sequence.

[0016] The present invention provides the following solutions:

[0017] A respiratory motion estimation method, comprising:

[0018] Obtaining raw ultrasound slice data;

[0019] Reconstructing the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0020] Combining the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and inputting the plurality of image pairs into a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0021] The motion estimation model is used to perform the following operations:

[0022] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0023] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0024] Using the fixed image key point set as the control point, a multi-source guidance vector including spatial position, semantic features and residual displacement is generated, and sent to the feature generation module for feature modeling and extraction to obtain the corresponding multi-source structured guidance features;

[0025] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a velocity field prediction module is used to predict a dense deformation field between two adjacent frames of images under sparse point guidance as an elastic deformation result;

[0026] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0027] The motion fields between the images at adjacent moments corresponding to all the image pairs are combined to obtain the motion field sequence; during the generation of the motion field sequence, sequence motion consistency constraints are optimized to enhance the continuity of the motion field in the time dimension.

[0028] Preferably, the constructed 2D ultrasound sequence acquisition system is used to acquire the original ultrasound slice data.

[0029] Preferably, after performing image preprocessing on the 3D ultrasound image sequence, the 3D ultrasound image sequence is combined into a plurality of image pairs according to two frames of images at adjacent moments; the image preprocessing includes at least cropping, resampling, normalization, and histogram matching.

[0030] Preferably, the original ultrasound slice data includes liver ultrasound image data with large motion differences caused by respiratory motion.

[0031] Preferably, the feature generation module includes a deep learning 3D network PointNet.

[0032] A respiratory motion estimation device, characterized in that it is used to perform the above-mentioned respiratory motion estimation method, the device comprising:

[0033] A data acquisition unit, used for acquiring original ultrasound slice data;

[0034] an image sequence reconstruction unit, configured to reconstruct the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0035] a motion estimation unit, configured to combine the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and to input the plurality of image pairs into a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0036] The motion estimation model is used to perform the following operations:

[0037] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0038] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0039] Using the fixed image key point set as the control point, a multi-source guidance vector including spatial position, semantic features and residual displacement is generated, and sent to the feature generation module for feature modeling and extraction to obtain the corresponding multi-source structured guidance features;

[0040] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a velocity field prediction module is used to predict a dense deformation field between two adjacent frames of images under sparse point guidance as an elastic deformation result;

[0041] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0042] The motion fields between the images at adjacent moments corresponding to all the image pairs are combined to obtain the motion field sequence; during the generation of the motion field sequence, sequence motion consistency constraints are optimized to enhance the continuity of the motion field in the time dimension.

[0043] A respiratory motion estimation device, comprising a processor and a memory:

[0044] The memory is used to store program code and transmit the program code to the processor;

[0045] The processor is configured to execute the above-mentioned respiratory motion estimation method according to instructions in the program code.

[0046] A computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned respiratory motion estimation method.

[0047] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] The embodiments of the present application provide a respiratory motion estimation method, apparatus, device and medium, which adopt an unsupervised sparse-dense motion estimation framework with sequence motion consistency constraints, and can quickly and accurately realize the motion estimation of liver ultrasound image sequences under the influence of respiratory motion. The motion field between adjacent respiratory state images is accurately predicted by a sparse-dense coarse-to-fine registration strategy guided by sparse points, a rigid registration network guided by automatic detection of sparse key points is designed to automatically detect sparse key points from the image in an unsupervised manner, and a deformation densification network guided by multi-source structured features is constructed to predict the dense elastic deformation component of the motion between images. Combining the ideas of motion decomposition and conformity, it is proposed to construct a sequence motion consistency constraint based on a sequence of small motions between several adjacent respiratory states, strengthen the continuity of motion in the time flow, and further improve the accuracy of motion estimation.

[0049] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0051] Figure 1 is a flow chart of a respiratory motion estimation method provided by an embodiment of the present invention;

[0052] Figure 2 This is a training and workflow diagram of a respiratory motion estimation method provided by an embodiment of the present invention;

[0053] Figure 3 2 is a schematic diagram of a motion estimation model framework provided by an embodiment of the present invention;

[0054] Figure 4 This is a comparison chart of the registration visualization effects of different registration algorithms provided by the embodiments of the present invention on the CLUST dataset;

[0055] Figure 5 is a blood vessel distance error map showing the motion estimation effects of different motion estimation algorithms provided by an embodiment of the present invention on a CLUST dataset;

[0056] Figure 6 This is a radar chart that quantitatively evaluates the registration effects of all methods provided in the embodiments of the present invention on the CLUST public dataset;

[0057] Figure 7 is a schematic diagram of a respiratory motion estimation device provided by an embodiment of the present invention;

[0058] Figure 8 2 is a schematic diagram of a respiratory motion estimation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0060] See also Figure 1 , is a respiratory motion estimation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0061] S101: Acquire raw ultrasound slice data. Specifically, embodiments of the present application can utilize a constructed 2D ultrasound sequence acquisition system to acquire raw ultrasound slice data. Furthermore, the raw ultrasound slice data includes liver ultrasound image data with significant motion differences caused by respiratory motion.

[0062] S102: Use an ultrasound reconstruction algorithm to reconstruct the original ultrasound slice data to obtain a corresponding 3D ultrasound image sequence; in specific implementation, after performing an image preprocessing operation on the 3D ultrasound image sequence, the 3D ultrasound image sequence is combined into a plurality of image pairs according to two frames of images at adjacent moments; the image preprocessing includes at least cropping, resampling, normalization, and histogram matching.

[0063] S103: Combining the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and sending the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0064] The motion estimation model is used to perform the following operations:

[0065] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0066] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0067] The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features; in specific implementation, the embodiment of the present application can provide that the feature generation module includes a deep learning 3D network PointNet.

[0068] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a velocity field prediction module is used to predict a dense deformation field between two adjacent frames of images under sparse point guidance as an elastic deformation result;

[0069] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0070] The motion fields between the images at adjacent moments corresponding to all the image pairs are combined to obtain the motion field sequence; during the generation of the motion field sequence, sequence motion consistency constraints are optimized to enhance the continuity of the motion field in the time dimension.

[0071] The respiratory motion estimation method provided in the embodiment of the present application uses the idea of ​​motion composite decomposition to construct a sequence motion consistency constraint, decomposes the single-step estimation of large-amplitude motion into a gradual composite of small-amplitude motions between several adjacent moments, and improves the smoothness and credibility of the dense motion estimation of the image sequence in the time dimension. The small-amplitude motion between two frames of images at adjacent moments is jointly realized by a rigid registration network guided by automatic detection of sparse key points and a deformation densification network guided by multi-source structured features. The former's rigid registration network automatically detects and extracts a set of sparse key points that are effective for the registration task from the image sequence and calculates the rigid transformation components. The latter's deformation densification network receives the image after rigid registration, models the spatial structure at the sparse points, the residual displacement and the deep semantic multi-source structured features as guiding information to predict the dense elastic deformation components. Finally, the motion field between two frames of images at adjacent moments is composed of a composite of rigid components and elastic components.

[0072] The motion field sequence is composed of the motion fields between two frames at numerous adjacent moments. The generation of the motion field sequence is optimized using sequence motion consistency constraints, enhancing the temporal continuity of the motion field and improving the accuracy of motion estimation. The entire system uses a 3D ultrasound image sequence as input, and the optimization process does not require any supervisory information.

[0073] The method provided in the embodiment of the present application fully considers the poor image quality of ultrasound images and the large-scale motion differences caused by respiratory movement, thereby improving the modeling accuracy of the motion field for large-scale motion; on the premise of reducing the requirements for data standardization as much as possible, it fully models the temporal characteristics of the image sequence, explores the sequential dependence of the motion field, and further improves the accuracy and continuity of the motion field.

[0074] The following is a detailed introduction to the respiratory motion estimation method provided in the embodiments of the present application.

[0075] like Figure 2 As shown, the training and implementation process of the method may include the following steps:

[0076] Step 1: Build a 2D ultrasound sequence acquisition system to acquire raw ultrasound slice data; for example, liver ultrasound image data included in public data can be obtained.

[0077] The second step: the collected original 2D ultrasound slice data is reconstructed using an ultrasound reconstruction algorithm to obtain the corresponding 3D ultrasound image sequence, and after performing image preprocessing operations such as cropping, resampling, normalization, and histogram matching, the training set, validation set, and test set are divided. In the specific implementation, an ultrasound image sequence image dataset is constructed, and the training set and test set are divided. In order to ensure the full verification of the generalization of the invented method, experimental verification is carried out on public datasets and clinical self-collected datasets, and preprocessing operations such as size cropping, intensity normalization, and histogram equalization are performed on each original dataset. The preprocessed dataset is divided into training data and test data according to a certain ratio.

[0078] Step 3: The 3D ultrasound image sequence is fed into the motion estimation framework in the form of two adjacent frames, each consisting of a floating image and a fixed image. The keypoint sets corresponding to the two images to be registered are first extracted: the floating image keypoint set and the fixed image keypoint set. The image that requires mapping is called the floating image, while the image that remains unchanged during the registration process is called the fixed image or reference image.

[0079] The sparse-dense motion estimation framework with sequence motion consistency constraints is constructed, as shown in the following diagram Figure 3 The entire framework consists of a rigid registration network guided by sparse keypoint automatic detection and a deformation densification network guided by multi-source structured features. The rigid registration network guided by sparse keypoint automatic detection consists of a keypoint extraction network and a rigid transformation matrix solver, while the deformation densification network guided by multi-source structured features consists of a guided feature generator, a guided feature fusion module, and a velocity field generation module.

[0080] The framework accepts a complete 3D ultrasound image sequence as input and outputs a sequence of motion fields consisting of the motion fields between two images in adjacent respiratory states. Specifically, each pair of images to be estimated is fed into the motion estimation framework to generate the corresponding motion field. The motion field between images that are far apart within the respiratory cycle is then composited from the motion fields between several adjacent images.

[0081] Step 4: Calculate the optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transform the floating image and the corresponding key point set according to the rigid transformation matrix to obtain the rigid transformation result;

[0082] Step 5: The sparse fixed image key point set corresponding to the fixed image is used as the control point to generate the spatial position, semantic features and residual displacement multi-source guidance vectors, and then send them to the feature generation module including the deep learning 3D network PointNet for feature modeling to extract the corresponding multi-source structured guidance features;

[0083] Step 6: Combine the multi-source structured guidance features with the image pairs to be registered after the initial rigid transformation, and predict the dense deformation field between images under the guidance of sparse points as the elastic deformation result through the velocity field prediction module. The elastic deformation result is combined with the rigid transformation result obtained after the initial rigid transformation matrix transformation to obtain the motion field between images at adjacent moments corresponding to each image pair, that is, multiple small motion fields are obtained.

[0084] Step 7: Model the motion field between two images with large differences in respiratory states and motion as a composite of several small motion fields between adjacent images, and construct a sequence motion consistency constraint based on this and add it to the overall loss function.

[0085] Step 8: The final network output is the sequential motion field corresponding to the input image sequence. Various loss functions are calculated and the complete motion estimation network is iteratively optimized. The network is trained on the training set data until convergence, and the network model parameters are saved. Different training strategies and hyperparameter combinations are tried, and an optimal set of training strategies and hyperparameter combinations is ultimately determined through ablation experiments.

[0086] Step 9: In the testing phase, the trained network model parameters are imported to achieve fast and accurate motion field estimation based on the input image pairs to be registered.

[0087] To validate the effectiveness of the method presented in this application, extensive experimental validation was conducted on the publicly available 3D liver ultrasound dataset, CLUST, and a clinical dataset. Furthermore, to quantitatively compare the proposed method with existing methods, comparative experiments were conducted using six different registration algorithms: LSD, KeyMorph, VoxelMorph, VXM_Prob, DeepSim, and GraphRegNet, as well as three different motion estimation algorithms: DSD, DeepTag, and SequenceMorph.

[0088] The experimental results can be found in Figure 4 、 Figure 5 、 Figure 6 .

[0089] Figure 4 Comparison of the registration visualization results of different registration algorithms on the CLUST dataset. White boxes represent magnified areas. (a) Floating image; (b) VoxelMorph; (c) VoxelMorph_Prob; (d) LSD; (e) GraphRegNet; (f) DeepSim; (g) KeyMorph; (h) SCSDME; (i) Fixed image.

[0090] Figure 5Figure 2 shows the vessel distance error maps for different motion estimation algorithms on the CLUST dataset. (a) The images in each row represent a sequence of original 3D ultrasound images within the same respiratory cycle, with images 1 to 6 representing unidirectional respiratory motion from end-expiration to end-inspiration. The end-inspiratory EI image is used as the fixed image, and images at other states within the respiratory cycle are aligned to the end-inspiratory EI image. A distance map is generated based on the vessel labels in the first two images before registration.

[0091] In the figure, dark colors represent smaller distances between blood vessels, while light colors represent larger distances between blood vessels. The remaining rows represent the alignment effects of different motion estimation methods on data under different respiratory states: (b) DSD; (c) DeepTag; (d) SequenceMorph; and (e) SCSDME.

[0092] Figure 6 Figure 2 shows a quantitative evaluation of the registration performance of all methods on the CLUST public dataset. (a1-2) evaluate the proposed SCSDME against six registration methods, while (b1-2) compare three motion estimation methods. The coordinate axes in the radar chart have the same numerical range, from 0 to 1. The specific value of SCSDME (red) and the best performance of the compared methods are marked with the corresponding colors.

[0093] As shown in the various result graphs, compared to existing registration algorithms, the SCSDME method provided by the embodiments of this application can achieve the best alignment results and the most accurate deformation. Compared with LSD and KeyMorph, which are based solely on key point registration, SCSDME's deformation is more comprehensive and detailed. This is mainly due to the dense deformation field prediction function provided by the deformation densification network guided by multi-source structured features. It can diffuse displacement information at sparse locations to the entire image, driving better alignment of other areas within the image.

[0094] Compared with the VoxelMorph, VoxelMorph_Prob and DeepSim methods based solely on image registration, SCSDME has better matching effects in key structural areas such as blood vessels, with fewer abnormal and erroneous deformations. This is mainly due to the fact that the rigid registration network guided by automatic detection of sparse key points can extract key points that are beneficial to image alignment and closer to significant areas such as blood vessels. Propagating the multi-source structural features of these key points as guiding information into dense deformation field modeling improves the visibility of key structures during the deformation process to a certain extent.

[0095] Compared to existing motion estimation algorithms, the SCSDME provided by the present embodiment achieves superior motion estimation results under the influence of respiratory motion of varying amplitudes, particularly achieving the best alignment of vascular regions. In contrast, other comparison methods experience a certain degree of degradation in their motion estimation results, with errors increasing as respiratory amplitude increases.

[0096] In summary, the respiratory motion estimation method provided by the present application adopts an unsupervised sparse-dense motion estimation framework with sequence motion consistency constraints, which can quickly and accurately realize the motion estimation of liver ultrasound image sequences under the influence of respiratory motion. The motion field between adjacent respiratory state images is accurately predicted by a sparse-dense coarse-to-fine registration strategy guided by sparse points, a rigid registration network guided by automatic detection of sparse key points is designed to automatically detect sparse key points from the image in an unsupervised manner, and a deformation densification network guided by multi-source structured features is constructed to predict the dense elastic deformation component of the motion between images. Combining the ideas of motion decomposition and conformity, it is proposed to construct a sequence motion consistency constraint based on a sequence of small motions between several adjacent respiratory states, strengthen the continuity of motion in the time flow, and further improve the accuracy of motion estimation.

[0097] See also Figure 7 , the embodiment of the present application can also provide a respiratory motion estimation device, such as Figure 7 As shown, the apparatus for performing the above-mentioned respiratory motion estimation method may include:

[0098] A data acquisition unit 701 is used to acquire original ultrasound slice data;

[0099] An image sequence reconstruction unit 702 is configured to reconstruct the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0100] A motion estimation unit 703 is configured to combine the 3D ultrasound image sequence into a plurality of image pairs by pairwise combining two frames of images at adjacent moments, and to input the plurality of image pairs into a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0101] The motion estimation model is used to perform the following operations:

[0102] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0103] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0104] Using the fixed image key point set as the control point, a multi-source guidance vector including spatial position, semantic features and residual displacement is generated, and sent to the feature generation module for feature modeling and extraction to obtain the corresponding multi-source structured guidance features;

[0105] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a velocity field prediction module is used to predict a dense deformation field between two adjacent frames of images under sparse point guidance as an elastic deformation result;

[0106] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0107] The motion fields between the images at adjacent moments corresponding to all the image pairs are combined to obtain the motion field sequence; during the generation of the motion field sequence, sequence motion consistency constraints are optimized to enhance the continuity of the motion field in the time dimension.

[0108] The embodiment of the present application may further provide a respiratory motion estimation device, the device comprising a processor and a memory:

[0109] The memory is used to store program code and transmit the program code to the processor;

[0110] The processor is configured to execute the steps of the above-mentioned respiratory motion estimation method according to the instructions in the program code.

[0111] like Figure 8 As shown, a respiratory motion estimation device provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 communicate with each other via the communication bus 13.

[0112] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices.

[0113] The processor 10 may call a program stored in the memory 11 . Specifically, the processor 10 may execute the operations in the embodiment of the respiratory motion estimation method.

[0114] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operating instructions. In the embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:

[0115] Obtaining raw ultrasound slice data;

[0116] Reconstructing the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0117] Combining the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and inputting the plurality of image pairs into a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0118] The motion estimation model is used to perform the following operations:

[0119] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0120] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0121] Using the fixed image key point set as the control point, a multi-source guidance vector including spatial position, semantic features and residual displacement is generated, and sent to the feature generation module for feature modeling and extraction to obtain the corresponding multi-source structured guidance features;

[0122] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a velocity field prediction module is used to predict a dense deformation field between two adjacent frames of images under sparse point guidance as an elastic deformation result;

[0123] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0124] The motion fields between the images at adjacent moments corresponding to all the image pairs are combined to obtain the motion field sequence; during the generation of the motion field sequence, sequence motion consistency constraints are optimized to enhance the continuity of the motion field in the time dimension.

[0125] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0126] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0127] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0128] Of course, it needs to be explained that Figure 8 The structure shown does not constitute a limitation on the respiratory motion estimation device in the embodiment of the present application. In actual applications, the respiratory motion estimation device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0129] An embodiment of the present application may further provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned respiratory motion estimation method.

[0130] It should be noted that, in this application, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0131] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0132] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A respiratory motion estimation method, characterized in that: include: Obtaining raw ultrasound slice data; Reconstructing the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence; Combining the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and inputting the plurality of image pairs into a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence; The motion estimation model is used to perform the following operations: Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set; Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result; Using the fixed image key point set as the control point, a multi-source guidance vector including spatial position, semantic features and residual displacement is generated, and sent to the feature generation module for feature modeling and extraction to obtain the corresponding multi-source structured guidance features; The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a velocity field prediction module is used to predict a dense deformation field between two adjacent frames of images under sparse point guidance as an elastic deformation result; Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair; The motion fields between the images at adjacent moments corresponding to all the image pairs are combined to obtain the motion field sequence; during the generation of the motion field sequence, sequence motion consistency constraints are optimized to enhance the continuity of the motion field in the time dimension.

2. The respiratory motion estimation method according to claim 1, wherein The original ultrasound slice data were acquired using the constructed 2D ultrasound sequence acquisition system.

3. The respiratory motion estimation method according to claim 1, wherein After performing image preprocessing on the 3D ultrasound image sequence, the 3D ultrasound image sequence is combined into a plurality of image pairs according to two frames of images at adjacent moments; the image preprocessing includes at least cropping, resampling, normalization, and histogram matching.

4. The respiratory motion estimation method according to claim 1, wherein The original ultrasound slice data includes liver ultrasound image data with large motion differences caused by respiratory motion.

5. The respiratory motion estimation method according to claim 1, wherein The feature generation module includes a deep learning 3D network PointNet.

6. A respiratory motion estimation device, characterized in that: For executing the respiratory motion estimation method according to any one of claims 1 to 5, the device comprises: A data acquisition unit, used for acquiring original ultrasound slice data; an image sequence reconstruction unit, configured to reconstruct the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence; a motion estimation unit, configured to combine the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and to input the plurality of image pairs into a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence; The motion estimation model is used to perform the following operations: Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set; Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution based on the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result; Using the fixed image key point set as the control point, a multi-source guidance vector including spatial position, semantic features and residual displacement is generated, and sent to the feature generation module for feature modeling and extraction to obtain the corresponding multi-source structured guidance features; The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a velocity field prediction module is used to predict a dense deformation field between two adjacent frames of images under sparse point guidance as an elastic deformation result; Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair; The motion fields between the images at adjacent moments corresponding to all the image pairs are combined to obtain the motion field sequence; during the generation of the motion field sequence, sequence motion consistency constraints are optimized to enhance the continuity of the motion field in the time dimension.

7. A respiratory motion estimation device, characterized in that The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the respiratory motion estimation method according to any one of claims 1 to 5 according to instructions in the program code.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the respiratory motion estimation method according to any one of claims 1 to 5.

Images