Respiratory motion estimation method and device, equipment and medium

Abstract

The invention discloses a respiratory motion estimation method, device and equipment and a medium, relates to the technical field of motion estimation analysis, adopts an unsupervised sparse-dense motion estimation framework with sequence motion consistency constraint, and can quickly and accurately realize motion estimation of a liver ultrasonic image sequence under the influence of respiratory motion. A motion field between adjacent breathing state images is accurately predicted through a sparse-dense coarse-to-fine registration strategy based on sparse point guidance, and a rigid registration network guided by sparse key point automatic detection is designed to automatically detect sparse key points from the images in an unsupervised mode. A deformation densification network guided by multi-source structural features is constructed to predict a dense elastic deformation component of motion between images. In combination with the idea of motion decomposition and coincidence, a sequence motion consistency constraint is constructed based on a small motion sequence between a plurality of adjacent breathing states, the continuity of motion in a time stream is enhanced, and the motion estimation precision is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion estimation and analysis, and in particular to a method, device, equipment and medium for estimating respiratory motion based on sequence motion consistency constraints. Background Art

[0002] Affected by respiratory movement, the liver will produce spatial displacement and flexible deformation with a maximum amplitude of several centimeters. This displacement and deformation poses a huge challenge to the precise positioning of liver tumors during liver puncture biopsy and radiofrequency ablation radiotherapy. In clinical practice, some work has adopted alternative strategies such as abdominal compression, breath holding, and respiratory gating to limit the respiratory movement of the liver, but it is impossible to completely eliminate the flexible deformation of the liver, and there are shortcomings such as increasing treatment time and causing physical discomfort to patients. Another part of the work adopts a non-invasive strategy based on image guidance to estimate the flexible movement of the liver caused by respiration. It provides comprehensive spatiotemporal information of the target structure through multiple three-dimensional data sampled over a period of time, estimates the correspondence between volume data at different times / respiratory states, and compensates for the flexible movement caused by respiration, thereby assisting clinical treatment and diagnosis.

[0003] Motion estimation is to establish the corresponding deformation field between images at different time points and estimate the movement of the target structure, so as to allow subsequent evaluation and analysis of its structural characteristics, and ultimately assist in improving clinical treatment. Affected by the amplitude and depth of respiratory movement, the amplitude of movement between two moments far apart in the respiratory movement cycle (such as the end of exhalation and the end of inhalation) is larger. According to statistics, the average amplitude of liver movement caused by respiratory movement in the head and foot direction is 13-25mm during shallow breathing, and can reach 55mm during deep breathing, resulting in large-scale flexible deformation of the liver and blood vessels in the liver during treatment, interfering with the doctor's treatment process.

[0004] Therefore, it is crucial to estimate and compensate for large movements. In addition, respiratory movement has cyclical biological characteristics, and most existing respiratory movement estimation algorithms focus on building the correspondence between a single set of static image pairs, ignoring the temporal continuity characteristics in the image sequence in which they are located, lacking motion continuity constraints in the time dimension, and it is difficult to obtain smooth and continuous motion in time.

[0005] The corresponding deformation field between images at different time points can be realized by image registration algorithm. The registration algorithm finds the best spatial deformation field between two images at different times by solving the optimal solution of the objective function composed of image similarity measure and deformation regularization constraint.

[0006] Traditional registration algorithms usually use computationally intensive iterative optimization strategies with high time costs. Registration algorithms based on deep learning use convolutional neural networks and their variants to extract the potential representation of the image pairs to be registered and calculate the corresponding motion field. They have high computational efficiency and significantly reduce the registration time. However, their performance is limited by the size of the receptive field of the convolution kernel, and the estimation effect of large-scale motion is insufficient. At the same time, due to the excessive pursuit of image intensity similarity, non-physiological false deformations and misestimations often occur.

[0007] Registration algorithms based on deep learning aim to find the best spatial correspondence between images. One type of method generates a deformation field by detecting key points that are specific in terms of spatial structure and anatomy. These key points can be manually annotated landmarks or can be obtained by manually crafted key point extraction strategies based on information such as geometric features and image intensity. However, they are task-sensitive and time-consuming.

[0008] This type of registration method based on sparse points is limited by the scale of key points. The sparse displacement information at discrete points is not enough to cover the more complex and wider flexible movements caused by breathing. In addition, without introducing supervision information such as segmentation and gold standard of marking points at the cost of additional annotation, the key points are automatically detected only with registration similarity as a constraint, which makes it difficult to establish a direct association between key points and significant areas such as blood vessels. Therefore, the alignment effect in key structural areas is difficult to guarantee. Another type of method estimates the optimal dense deformation field by optimizing the registration objective function composed of image similarity measure and regularization constraint. This type of method is limited by the size of the receptive field of the convolution kernel, and the prediction accuracy for large-scale deformation is limited. Moreover, most of them directly generate the full-image deformation field by optimizing the global registration similarity loss based on intensity such as MSE and NCC, lacking the focus on significant anatomical structures and regions.

[0009] Some algorithms use gold standards such as segmentation labels to provide supervisory information guidance for significant structural regions, but this will increase the requirements for data collection and add additional pressure on clinical experts to annotate. In addition, due to the weak reflection ability of blood in blood vessels to ultrasound, the vascular region in the ultrasound image appears as a low-grayscale low-echo region, which ultimately leads to a low contribution of the vascular region in the global dense deformation field prediction based only on the similarity loss based on image intensity. The entire optimization process focuses more on the alignment of high-intensity regions such as the edge of the liver. A small number of works combine point-based registration and image-based registration methods to give full play to the advantages of both methods. However, at this stage, the sparse points used by such algorithms are mostly obtained based on manual annotation or traditional feature extraction methods, which have limitations such as high time cost. In addition, the combination of key points and images is limited to the embedding combination of different constraints at the loss function level, and does not capture the potential deep connection between the two.

[0010] With the help of registration algorithms, dense motion fields between images at different times can be established to achieve motion estimation, and some work focuses on the sequence characteristics of image sequences based on the registration framework. This type of work aims to mine potential temporal features in image sequences by establishing a temporal feature extraction module and analyze the time-varying patterns of target anatomical structures in images. Although this type of method takes sequence characteristics into account, the prediction of the motion field between a single set of image pairs is still mainly achieved using a single-step prediction method based on image registration, which is prone to incorrect estimation of large-scale motion caused by breathing.

[0011] It can be seen that most of the current deep learning-based motion estimation algorithms introduce registration algorithms to model the dense motion field between images under different respiratory states, and their effectiveness is mainly limited by the deformation field modeling capability of the registration algorithm. However, the existing deep learning registration algorithms based on points, images, and a combination of the two can be optimized for ultrasound images with poor image quality and large respiratory motion. Therefore, the existing deep learning-based motion estimation algorithms still have the following defects:

[0012] 1. Although the method of modeling dense deformation field based solely on image intensity similarity as the optimization benchmark can obtain deformation information in the global range at a faster calculation speed, its effect is largely restricted by the image quality. In ultrasound image scenes with poor image quality and severe speckle noise, the accuracy of the deformation field predicted is limited;

[0013] 2. Considering that the liver has a large movement amplitude under the influence of respiratory motion, the information difference between the two images to be registered is large at the image level, especially for images at the two end points, which leads to the inadequate estimation effect of existing methods for large-amplitude motion.

[0014] 3. A small number of works build temporal feature extraction modules based on the motion field between a single set of static images. Although they can dig deeper into the potential temporal features in the image sequence, they have high requirements for the integrity and standardization of the image sequence, and their usage scenarios are greatly limited. Summary of the invention

[0015] In view of the above problems, the present invention provides a respiratory motion estimation method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems. It is particularly suitable for liver flexible motion estimation analysis based on liver three-dimensional ultrasound sequence. It overcomes the problems that the existing motion estimation algorithm based on deep learning has limited effect on large-amplitude respiratory motion estimation and insufficient mining of potential time series features in the sequence.

[0016] The present invention provides the following scheme:

[0017] A respiratory motion estimation method, comprising:

[0018] Obtaining raw ultrasound slice data;

[0019] Reconstructing the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0020] Combining the 3D ultrasound image sequence in pairs according to two frames of images at adjacent moments to form a plurality of image pairs, and sending the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0021] The motion estimation model is used to perform the following operations:

[0022] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0023] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0024] The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features;

[0025] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a dense deformation field between two frames of images at adjacent moments under sparse point guidance is predicted through a velocity field prediction module as an elastic deformation result;

[0026] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0027] The motion field sequence is obtained by compounding the motion fields between the images at adjacent moments corresponding to all the image pairs; during the generation of the motion field sequence, the sequence motion consistency constraint is optimized to enhance the continuity of the motion field in the time dimension.

[0028] Preferably, the original ultrasound slice data is acquired using the constructed 2D ultrasound sequence acquisition system.

[0029] Preferably, after the 3D ultrasound image sequence is subjected to image preprocessing, the 3D ultrasound image sequence is combined in pairs according to two frames of images at adjacent moments to form a plurality of image pairs; the image preprocessing at least includes cropping, resampling, normalization, and histogram matching.

[0030] Preferably, the original ultrasound slice data includes liver ultrasound image data with large motion differences caused by respiratory motion.

[0031] Preferably: the feature generation module includes a deep learning 3D network PointNet.

[0032] A respiratory motion estimation device, characterized in that it is used to perform the above-mentioned respiratory motion estimation method, and the device comprises:

[0033] A data acquisition unit, used for acquiring original ultrasonic slice data;

[0034] An image sequence reconstruction unit, configured to reconstruct the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0035] A motion estimation unit, configured to combine the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and send the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0036] The motion estimation model is used to perform the following operations:

[0037] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0038] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0039] The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features;

[0040] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a dense deformation field between two frames of images at adjacent moments under sparse point guidance is predicted through a velocity field prediction module as an elastic deformation result;

[0041] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0042] The motion field sequence is obtained by compounding the motion fields between the images at adjacent moments corresponding to all the image pairs; during the generation of the motion field sequence, the sequence motion consistency constraint is optimized to enhance the continuity of the motion field in the time dimension.

[0043] A respiratory motion estimation device, the device comprising a processor and a memory:

[0044] The memory is used to store program code and transmit the program code to the processor;

[0045] The processor is used to execute the above-mentioned respiratory motion estimation method according to the instructions in the program code.

[0046] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned respiratory motion estimation method.

[0047] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] The embodiments of the present application provide a respiratory motion estimation method, device, equipment and medium, which adopt an unsupervised sparse-dense motion estimation framework with sequence motion consistency constraints, and can quickly and accurately realize the motion estimation of liver ultrasound image sequences under the influence of respiratory motion. The motion field between adjacent respiratory state images is accurately predicted through a sparse-dense coarse-to-fine registration strategy guided by sparse points, and a rigid registration network guided by automatic detection of sparse key points is designed to automatically detect sparse key points from the image in an unsupervised manner. A deformation densification network guided by multi-source structured features is constructed to predict the dense elastic deformation component of the motion between images. Combining the ideas of motion decomposition and conformity, it is proposed to construct a sequence motion consistency constraint based on a sequence of small motions between several adjacent respiratory states, strengthen the continuity of motion in the time flow, and further improve the accuracy of motion estimation.

[0049] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 is a flow chart of a respiratory motion estimation method provided by an embodiment of the present invention;

[0052] Figure 2 A respiratory motion estimation method training and workflow diagram provided by an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of a motion estimation model framework provided by an embodiment of the present invention;

[0054] Figure 4 is a comparison diagram of the registration visualization effects of different registration algorithms provided by the embodiments of the present invention on the CLUST dataset;

[0055] Figure 5 is a blood vessel distance error mapping diagram of motion estimation effects of different motion estimation algorithms provided by an embodiment of the present invention on a CLUST data set;

[0056] Figure 6 It is a radar chart for quantitative evaluation of the registration effects of all the methods provided in the embodiments of the present invention on the CLUST public dataset;

[0057] Figure 7 is a schematic diagram of a respiratory motion estimation device provided by an embodiment of the present invention;

[0058] Figure 8 is a schematic diagram of a respiratory motion estimation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0060] See also Figure 1 , is a respiratory motion estimation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0061] S101: Acquire original ultrasound slice data; in specific implementation, the embodiment of the present application can provide a method of acquiring original ultrasound slice data using a constructed 2D ultrasound sequence acquisition system. Further, the original ultrasound slice data includes liver ultrasound image data with large motion differences caused by respiratory motion.

[0062] S102: Use an ultrasound reconstruction algorithm to reconstruct the original ultrasound slice data to obtain a corresponding 3D ultrasound image sequence; in specific implementation, after performing an image preprocessing operation on the 3D ultrasound image sequence, the 3D ultrasound image sequence is combined in pairs according to two frames of images at adjacent moments to form a plurality of image pairs; the image preprocessing includes at least cropping, resampling, normalization, and histogram matching.

[0063] S103: Combining the 3D ultrasound image sequence in pairs according to two frames of images at adjacent moments to form a plurality of image pairs, and sending the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0064] The motion estimation model is used to perform the following operations:

[0065] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0066] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0067] The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features; in specific implementation, the embodiment of the present application can provide that the feature generation module includes a deep learning 3D network PointNet.

[0068] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a dense deformation field between two frames of images at adjacent moments under sparse point guidance is predicted through a velocity field prediction module as an elastic deformation result;

[0069] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0070] The motion field sequence is obtained by compounding the motion fields between the images at adjacent moments corresponding to all the image pairs; during the generation of the motion field sequence, the sequence motion consistency constraint is optimized to enhance the continuity of the motion field in the time dimension.

[0071] The respiratory motion estimation method provided in the embodiment of the present application constructs a sequence motion consistency constraint with the help of the idea of ​​motion compound decomposition, decomposes the single-step estimation for large-amplitude motion into a gradual compound of small-amplitude motions between several adjacent moments, and improves the smoothness and credibility of dense motion estimation of image sequences in the time dimension. The small-amplitude motion between two frames of images at adjacent moments is jointly realized by a rigid registration network guided by automatic detection of sparse key points and a deformation densification network guided by multi-source structured features. The former's rigid registration network automatically detects and extracts a set of sparse key points that are effective for the registration task from the image sequence and calculates the rigid transformation component. The latter's deformation densification network receives the image after rigid registration, models the spatial structure at the sparse points, the residual displacement and the deep semantic multi-source structured features as guiding information to predict the dense elastic deformation component. Finally, the motion field between two frames of images at adjacent moments is composed of a composite of rigid components and elastic components.

[0072] The motion field sequence is composed of the motion fields between two frames of images at many adjacent moments. The motion field sequence generation process is optimized by the sequence motion consistency constraint, which enhances the continuity of the motion field in the time dimension and improves the prediction accuracy of motion estimation. The entire system uses a three-dimensional ultrasound image sequence as input, and the entire optimization process does not require any supervisory information.

[0073] The method provided in the embodiment of the present application fully considers the difference between the poor image quality of ultrasound images and the large-scale motion caused by respiratory movement, and improves the modeling accuracy of the motion field for large-scale motion; on the premise of reducing the requirements for data standardization as much as possible, it fully models the temporal characteristics of the image sequence, explores the sequence dependence of the motion field, and further improves the accuracy and continuity of the motion field.

[0074] The respiratory motion estimation method provided in the embodiments of the present application is described in detail below.

[0075] like Figure 2 As shown, the training and implementation process of the method may include the following steps:

[0076] Step 1: Build a 2D ultrasound sequence acquisition system to acquire raw ultrasound slice data; for example, liver ultrasound image data included in public data can be obtained.

[0077] Step 2: Use the ultrasound reconstruction algorithm to reconstruct the collected original 2D ultrasound slice data to obtain the corresponding 3D ultrasound image sequence, and perform image preprocessing operations such as cropping, resampling, normalization, and histogram matching, and then divide the data into training sets, validation sets, and test sets. In the specific implementation, an ultrasound image sequence image data set is constructed, and the training set and test set are divided. In order to ensure that the generalization of the invention method is fully verified, experimental verification is carried out on public data sets and clinical self-collected data sets, and preprocessing operations such as size cropping, intensity normalization, and histogram equalization are performed on each original data set. The preprocessed data set is divided into training data and test data in a certain proportion.

[0078] Step 3: The 3D ultrasound image sequence is combined into two adjacent frames, each combination includes a floating image and a fixed image, and sent to the motion estimation framework. First, the key point sets corresponding to the two images to be registered are extracted, namely the floating image key point set and the fixed image key point set. Among them, the image that needs to be mapped and transformed is called the floating image, and the image that remains unchanged during the registration process is called the fixed image or reference image.

[0079] The sparse-dense motion estimation framework with sequence motion consistency constraints is constructed, as shown in the following figure. Figure 3 As shown in the figure, the whole framework consists of a rigid registration network guided by sparse key point automatic detection and a deformation densification network guided by multi-source structured features. The rigid registration network guided by sparse key point automatic detection consists of a key point extraction network and a rigid transformation matrix solver, and the deformation densification network guided by multi-source structured features consists of a guided feature generator, a guided feature fusion module, and a velocity field generation module.

[0080] The entire framework receives a complete 3D ultrasound image sequence as input and outputs a motion field sequence consisting of the motion fields between two images of adjacent respiratory states. Specifically, each pair of images to be estimated is fed into the motion estimation framework to obtain the corresponding motion field, and the motion field between images that are far apart in the respiratory cycle is obtained by compounding the motion fields between several adjacent images one by one.

[0081] Step 4: Calculate the optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transform the floating image and the corresponding key point set according to the rigid transformation matrix to obtain the rigid transformation result;

[0082] Step 5: The sparse fixed image key point set corresponding to the fixed image is used as the control point to generate the spatial position, semantic features and residual displacement multi-source guidance vector, and then sent to the feature generation module including the deep learning 3D network PointNet for feature modeling to extract the corresponding multi-source structured guidance features;

[0083] Step 6: Combine the multi-source structured guidance features with the image pairs to be registered after the initial rigid transformation, and predict the dense deformation field between images under the guidance of sparse points as the elastic deformation result through the velocity field prediction module, and then combine the elastic deformation result with the rigid transformation result obtained after the initial rigid transformation matrix transformation to obtain the motion field between images at adjacent moments corresponding to each image pair, that is, obtain multiple small motion fields.

[0084] Step 7: Model the motion field between two images with large differences in breathing states and motion differences as a composite of several small motion fields between adjacent images, and construct a sequence motion consistency constraint based on this and add it to the overall loss function.

[0085] Step 8: The final network output obtains the sequence motion field corresponding to the input image sequence, and calculates various loss functions to iteratively optimize the complete motion estimation network. Train the network according to the training set data until convergence, and save the network model parameters. Try different training strategies and hyperparameter combinations, and finally determine a set of optimal training strategies and hyperparameter combinations by designing ablation experiments.

[0086] Step 9: In the testing phase, the trained network model parameters are imported to achieve fast and accurate motion field estimation based on the input image pairs to be registered.

[0087] To verify the effectiveness of the method provided in the embodiment of the present application, extensive experimental verification was carried out on the 3D liver ultrasound public dataset CLUST and the clinical self-collected dataset. In addition, in order to quantitatively compare the advantages and disadvantages of the method provided in the present application with the existing methods, 6 different registration algorithms LSD, KeyMorph, VoxelMorph, VXM_Prob, DeepSim, GraphRegNet and 3 different motion estimation algorithms DSD, DeepTag, SequenceMorph were selected for comparative experiments.

[0088] The experimental results can be found in Figure 4 , Figure 5 , Figure 6 .

[0089] Figure 4 The comparison results of the registration visualization effects of different registration algorithms on the CLUST dataset are shown in the figure. The white box represents the enlarged area. (a) Floating image; (b) VoxelMorph; (c) VoxelMorph_Prob; (d) LSD; (e) GraphRegNet; (f) DeepSim; (g) KeyMorph; (h) SCSDME; (i) Fixed image.

[0090] Figure 5The vascular distance error mapping of the motion estimation effect of different motion estimation algorithms on the CLUST dataset is shown in Figure 2. (a) The images in the middle row represent the original 3D ultrasound image sequence in the same respiratory cycle, and the images from 1 to 6 represent the unidirectional respiratory motion from the end of exhalation to the end of inspiration. The end-inspiratory EI image is set as the fixed image, and the images in other states in the respiratory cycle are aligned to the end-inspiratory EI image, and the distance map is obtained based on the vascular labels in the first two images.

[0091] In the figure, dark colors represent smaller distances between blood vessels, while light colors represent larger distances between blood vessels. The remaining rows represent the alignment effects of different motion estimation methods on data under different breathing states: (b) DSD; (c) DeepTag; (d) SequenceMorph and (e) SCSDME.

[0092] Figure 6 The quantitative evaluation results of the registration effects of all methods on the CLUST public dataset are shown in Figure 1. The SCSDME proposed in this application is evaluated against six registration methods in (a1-2), and three motion estimation methods are compared in (b1-2). The numerical range of each coordinate axis in the radar chart is the same, ranging from 0 to 1. The specific value of SCSDME (red) and the best performance of the compared methods are marked with corresponding colors.

[0093] As shown in the various result graphs, compared with the existing registration algorithms, the method SCSDME provided in the embodiment of the present application can achieve the best alignment effect and the most accurate deformation. Compared with LSD and KeyMorph based solely on key point registration, SCSDME deformation is more sufficient and detailed, which is mainly due to the dense deformation field prediction function provided by the deformation densification network guided by multi-source structured features, which can diffuse the displacement information at sparse positions to the entire image, driving better alignment of other areas in the image.

[0094] Compared with the VoxelMorph, VoxelMorph_Prob and DeepSim methods based solely on image registration, SCSDME has better matching effects in key structural areas such as blood vessels, with fewer abnormal and erroneous deformations. This is mainly due to the fact that the rigid registration network guided by automatic detection of sparse key points can extract key points that are beneficial to image alignment and closer to significant areas such as blood vessels. Propagating the multi-source structural features of these key points as guiding information into dense deformation field modeling improves the visibility of key structures during the deformation process to a certain extent.

[0095] Compared with the existing motion estimation algorithms, the SCSDME provided by the embodiment of the present application can achieve better motion estimation effects under the influence of different amplitudes of respiratory motion, especially the alignment effect of the blood vessel area is the best. In contrast, the motion estimation effects of other comparison methods will degrade to a certain extent and the error will increase as the respiratory amplitude increases.

[0096] In summary, the respiratory motion estimation method provided by the present application adopts an unsupervised sparse-dense motion estimation framework with sequence motion consistency constraints, which can quickly and accurately realize the motion estimation of the liver ultrasound image sequence under the influence of respiratory motion. The motion field between adjacent respiratory state images is accurately predicted through a sparse-dense coarse-to-fine registration strategy guided by sparse points, and a rigid registration network guided by automatic detection of sparse key points is designed to automatically detect sparse key points from the image in an unsupervised manner. A deformation densification network guided by multi-source structured features is constructed to predict the dense elastic deformation component of the motion between images. Combining the ideas of motion decomposition and conformity, it is proposed to construct a sequence motion consistency constraint based on a sequence of small motions between several adjacent respiratory states, strengthen the continuity of motion in the time flow, and further improve the accuracy of motion estimation.

[0097] See also Figure 7 , the embodiment of the present application may also provide a respiratory motion estimation device, such as Figure 7 As shown, for performing the above-mentioned respiratory motion estimation method, the device may include:

[0098] A data acquisition unit 701 is used to acquire original ultrasound slice data;

[0099] An image sequence reconstruction unit 702 is used to reconstruct the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0100] The motion estimation unit 703 is used to combine the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and send the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0101] The motion estimation model is used to perform the following operations:

[0102] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0103] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0104] The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features;

[0105] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a dense deformation field between two frames of images at adjacent moments under sparse point guidance is predicted through a velocity field prediction module as an elastic deformation result;

[0106] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0107] The motion field sequence is obtained by compounding the motion fields between the images at adjacent moments corresponding to all the image pairs; during the generation of the motion field sequence, the sequence motion consistency constraint is optimized to enhance the continuity of the motion field in the time dimension.

[0108] The embodiment of the present application may also provide a respiratory motion estimation device, the device comprising a processor and a memory:

[0109] The memory is used to store program code and transmit the program code to the processor;

[0110] The processor is used to execute the steps of the above-mentioned respiratory motion estimation method according to the instructions in the program code.

[0111] like Figure 8 As shown, a respiratory motion estimation device provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0112] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.

[0113] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the respiratory motion estimation method.

[0114] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0115] Obtaining raw ultrasound slice data;

[0116] Reconstructing the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence;

[0117] Combining the 3D ultrasound image sequence in pairs according to two frames of images at adjacent moments to form a plurality of image pairs, and sending the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence;

[0118] The motion estimation model is used to perform the following operations:

[0119] Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set;

[0120] Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result;

[0121] The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features;

[0122] The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a dense deformation field between two frames of images at adjacent moments under sparse point guidance is predicted through a velocity field prediction module as an elastic deformation result;

[0123] Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair;

[0124] The motion field sequence is obtained by compounding the motion fields between the images at adjacent moments corresponding to all the image pairs; during the generation of the motion field sequence, the sequence motion consistency constraint is optimized to enhance the continuity of the motion field in the time dimension.

[0125] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0126] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0127] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0128] Of course, it should be noted that Figure 8 The structure shown does not constitute a limitation on the respiratory motion estimation device in the embodiment of the present application. In actual applications, the respiratory motion estimation device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0129] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the steps of the above-mentioned respiratory motion estimation method.

[0130] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0131] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0132] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A respiratory motion estimation method, characterized in that: include: Obtaining raw ultrasound slice data; Reconstructing the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence; Combining the 3D ultrasound image sequence in pairs according to two frames of images at adjacent moments to form a plurality of image pairs, and sending the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence; The motion estimation model is used to perform the following operations: Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set; Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result; The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features; The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a dense deformation field between two frames of images at adjacent moments under sparse point guidance is predicted through a velocity field prediction module as an elastic deformation result; Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair; The motion field sequence is obtained by compounding the motion fields between the images at adjacent moments corresponding to all the image pairs; during the generation of the motion field sequence, the sequence motion consistency constraint is optimized to enhance the continuity of the motion field in the time dimension.

2. The respiratory motion estimation method according to claim 1, characterized in that: The original ultrasound slice data were acquired using the constructed 2D ultrasound sequence acquisition system.

3. The respiratory motion estimation method according to claim 1, characterized in that: After the 3D ultrasound image sequence is subjected to image preprocessing, the 3D ultrasound image sequence is combined in pairs according to two frames of images at adjacent moments to form a plurality of image pairs; the image preprocessing at least includes cropping, resampling, normalization, and histogram matching.

4. The respiratory motion estimation method according to claim 1, characterized in that: The original ultrasound slice data includes liver ultrasound image data with large motion differences caused by respiratory motion.

5. The respiratory motion estimation method according to claim 1, characterized in that: The feature generation module includes a deep learning 3D network PointNet.

6. A respiratory motion estimation device, characterized in that: Used to perform the respiratory motion estimation method according to any one of claims 1 to 5, the device comprising: A data acquisition unit, used for acquiring original ultrasonic slice data; An image sequence reconstruction unit, configured to reconstruct the original ultrasound slice data using an ultrasound reconstruction algorithm to obtain a corresponding 3D ultrasound image sequence; A motion estimation unit, configured to combine the 3D ultrasound image sequence into a plurality of image pairs according to two frames of images at adjacent moments, and send the plurality of image pairs to a motion estimation model so that the motion estimation model outputs a corresponding motion field sequence; The motion estimation model is used to perform the following operations: Extracting key points of the floating image and the fixed image contained in each of the image pairs through a sparse key point extraction network to obtain a floating image key point set and a fixed image key point set; Calculating an optimal spatial rigid transformation matrix between images in the form of a closed solution according to the floating image key point set and the fixed image key point set, and transforming the floating image and the floating image key point set according to the rigid transformation matrix to obtain a rigid transformation result; The fixed image key point set is used as a control point to generate a multi-source guidance vector including spatial position, semantic features and residual displacement, and is sent to a feature generation module for feature modeling and extraction to obtain corresponding multi-source structured guidance features; The multi-source structured guidance features and the rigid transformation results are combined through a guidance feature fusion module, and a dense deformation field between two frames of images at adjacent moments under sparse point guidance is predicted through a velocity field prediction module as an elastic deformation result; Combining the rigid transformation result with the elastic deformation result to obtain a motion field between images at adjacent moments corresponding to the image pair; The motion field sequence is obtained by compounding the motion fields between the images at adjacent moments corresponding to all the image pairs; during the generation of the motion field sequence, the sequence motion consistency constraint is optimized to enhance the continuity of the motion field in the time dimension.

7. A respiratory motion estimation device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the respiratory motion estimation method according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the respiratory motion estimation method according to any one of claims 1-5.

Images