Virtual-to-real field adaptive ultrasound and CT body registration method, device and system based on physical guidance

Abstract

The invention discloses a virtual-to-real field adaptive ultrasound and CT body registration method, device and system based on physical guidance. The method comprises the following steps: performing CT imaging on a target object to obtain a CT body image; a virtual-to-real field adaptive technology is utilized to obtain an adaptive virtual physical ultrasonic image of a target object, and the modal difference between the target object and CT is reduced; the CT body image and the self-adaptive virtual physical ultrasonic image are input into a preset slice-to-body registration module, a two-way CNN structure is adopted, feature extraction is conducted on the CT body image and the self-adaptive virtual physical ultrasonic image, and features extracted in the two ways are fused; and the orientation information of the self-adaptive virtual physical ultrasonic image in the CT body image is predicted based on the fused features by using a multi-layer perceptron. Based on self-supervised learning and virtuality-to-reality field adaptive technology, the invention provides an ultrasonic image and CT registration model which does not need to use tracking equipment and is rapid and good in robustness.

Description

Technical Field

[0001] The present invention relates to the field of ultrasound and CT registration, and in particular to a virtual-to-real domain adaptive ultrasound and CT body registration method, device and system based on physical guidance. Background Art

[0002] Ultrasound technology uses the difference in sound wave reflection in different tissues to form images, providing real-time imaging guidance for interventional surgery. However, due to the limitation of ultrasound imaging range, ultrasound cannot provide a detailed whole-body view compared with preoperative CT. Therefore, during actual interventional surgery, doctors need to continuously establish a mental mapping between intraoperative ultrasound images and preoperative CT images, which places an excessive psychological burden on doctors.

[0003] Developers are trying to accurately register ultrasound images with CT images to provide real-time and comprehensive detailed images to guide interventional surgery and reduce the psychological burden on doctors. If ultrasound images are registered with CT images, the real-time guidance of ultrasound and the global details of CT can be combined to reduce the time required for interventional surgery and improve surgical results.

[0004] However, the acquisition time, equipment and imaging principles of ultrasound images and CT images are different. There are huge differences between ultrasound images and CT images in dimension, modality and coordinate system. Traditional machine learning methods are difficult to locate the pose (i.e., position and posture) of ultrasound images in CT images in real time and robustly. In addition, the huge modal differences between ultrasound images and CT images and the lack of sufficient data lead to insufficient accuracy and generalization of intelligent models.

[0005] Although the image transformation method can make up for the insufficient data volume and modality differences, it is difficult to directly transform CT images and ultrasound images. In addition, there is a large domain difference between the images obtained after virtual transformation and the real images, which makes it difficult to obtain accurate CT and ultrasound fusion effects. The method based on image conversion and then posture estimation lacks robustness in real scenes.

[0006] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present application. It does not necessarily belong to the prior art of the present application, nor does it necessarily provide technical guidance. In the absence of clear evidence that the above content has been disclosed before the filing date of the present application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0007] The purpose of the present invention is to provide a fast and robust ultrasound image and CT registration model without the need for tracking equipment based on self-supervised learning and virtual-to-real domain adaptation technology.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A virtual-to-real domain adaptive ultrasound and CT volume registration method based on physical guidance comprises the following steps:

[0010] Performing CT imaging on the target object to obtain a CT volume image;

[0011] Using virtual-to-real domain adaptive technology, an adaptive virtual physical ultrasound image of the target object is obtained, and the adaptive virtual physical ultrasound image reduces the modality difference between itself and CT compared with real ultrasound;

[0012] The CT volume image and the adaptive virtual physical ultrasound image are input into a preset slice-to-volume registration module, wherein the slice-to-volume registration module adopts a dual-path CNN structure, wherein one path is configured to perform feature extraction on the CT volume image, and the other path is configured to perform feature extraction on the adaptive virtual physical ultrasound image, and the features extracted from the dual paths are fused; the slice-to-volume registration module is also configured with a multi-layer perceptron, which predicts the position information of the adaptive virtual physical ultrasound image within the CT volume image based on the fused features.

[0013] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the adaptive virtual physical ultrasound image is obtained by the following method:

[0014] Cropping the CT volume image to obtain a plurality of CT slice data;

[0015] The CT slices are mapped physically to generate the first virtual physical ultrasound image I PHY_CT ;

[0016] By using the virtual-to-real domain adaptation technology, a first adaptive virtual physical ultrasound image I corresponding to the first virtual physical ultrasound image is obtained after domain adaptation. PHY_CT '.

[0017] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the CT physical mapping is defined as:

[0018] E i (d) = R i (d)+B i (d), where d is the sensor depth, E i (d) represents the mapping of the ultrasonic rays emitted by the sensor at position d, R i (d) represents the reflected energy from the tissue interface in the body, B i (d) represents the backscattered energy;

[0019] in, Among them, I i(d) = e -αd , which represents the energy attenuation caused by the reflection of the tissue interface in the body, e represents the natural constant, α represents the attenuation coefficient, and Z i (d) represents the reflection coefficient, P(d) represents the point scattering function, G(d) represents the boundary, Represents the cross product operation;

[0020] in, represents the texture, which is generated by a random Gaussian process controlled by the speckle distribution parameters μ0, μ1 and δ0.

[0021] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the adaptive virtual physical ultrasound image is obtained by the following method:

[0022] The conditional diffusion model is used to learn in the spatial domain of real ultrasound to generate a second virtual physical ultrasound image I PHY_US ;

[0023] By using the virtual-to-real domain adaptation technology, a second adaptive virtual physical ultrasound image I corresponding to the second virtual physical ultrasound image is obtained after domain adaptation. PHY_US '.

[0024] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the conditional diffusion model defines a Markov chain in which random noise is gradually integrated into the data, which is called a forward process;

[0025] Sampling from the data distribution corresponding to the true data x0, the forward process adds Gaussian noise to the samples in T time steps:

[0026]

[0027] Where T is the diffusion time step, represents the process of adding Gaussian noise to the image at time t and the image at time t-1, βt represents t constants for controlling the noise step size or learning variance, t is a time between 1 and T steps, N represents a non-negative integer set, x0 represents a virtual image at time 0, q(x) represents a virtual image at a non-zero time, and x t represents the virtual image at time t, x t-1 represents the virtual image at time t-1, I represents the diagonal covariance matrix, It means that when x0 is given, x1,x2,...,x T The joint probability distribution of , ∏ is a continuous product operation.

[0028] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, x is sampled from x0 by the following formula: t : Among them, q(x t |x0) means sampling x from x0 t , let α t =1-β t ,and

[0029] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the second virtual physical ultrasound image I is generated by learning the inverse process and moving backward with the parameterized diagonal Gaussian transform. PHY_US :

[0030] Among them, p θ (x t-1 |x t ) is the sampling distribution of the unconditionally guided inverse process, μ θ (x t ,t) represents the mean of the conditional distribution obtained by network training, The value is Empirical parameters of

[0031] If there is an additional guiding signal y, the sampling distribution transforms to:

[0032] in, is the sampling distribution of the inverse process with an additional pilot signal y, is the normalization constant, represents the conditional random sampling process guided by the classifier, represents the time step x t-1 The conditional y classifier for .

[0033] Further, according to any one of the above-mentioned technical solutions or a combination of multiple technical solutions, by utilizing the alignment between the spatial feature map and the features of the corresponding spatial position, the generated second virtual physical ultrasound image I is guided. PHY_US Shares similar structure with the original real ultrasound image:

[0034] Among them, x′ t represents the real ultrasound image at time t, F(x t ,x′ t ,t) represents the virtual image at time t, It represents the image encoder E I 'j-th layer spatial feature, C j H represents the number of image feature channels at the jth layer of the image encoder,j represents the image feature height of the jth layer of the image encoder, W j Represents the image feature width of the j-th layer of the image encoder.

[0035] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the virtual-to-real domain adaptive technology converts the first virtual physical ultrasound image into the first adaptive virtual physical ultrasound image in the following manner:

[0036] Define the low frequency amplitude cutoff window M β ;

[0037] The conditional diffusion model is used to learn in the spatial domain of real ultrasound to generate a second virtual physical ultrasound image I PHY_US ; With the second virtual physical ultrasound image I PHY_US As the target domain image, the first virtual physical ultrasound image I PHY_CT For the source domain image, the low-frequency mapping from the source domain to the target domain is defined as:

[0038] I s→t =F -1 ([M β ·F a (x t )+(1-M β )·F a (x s ),F p (x s )]), where F -1 () is the inverse Fourier algorithm, F a (x t ) represents the target domain amplitude obtained by Fourier transforming the target domain image, F a (x s ) represents the source domain amplitude obtained by Fourier transforming the source domain image, F p (x s ) represents the source domain phase obtained by Fourier transforming the source domain image;

[0039] The virtual image obtained by low-frequency mapping is used as the first adaptive virtual physical ultrasound image I PHY_CT '.

[0040] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the virtual-to-real domain adaptive technology converts the second virtual physical ultrasound image into the second adaptive virtual physical ultrasound image in the following manner:

[0041] Define the low frequency amplitude cutoff window M β ;

[0042] Acquire a real ultrasound image as the target domain image, and use the second virtual physical ultrasound image I PHY_US For the source domain image, the low-frequency mapping from the source domain to the target domain is defined as:

[0043] I s→t =F -1 ([M β ·F a (x t )+(1-M β )·F a (x s ),F p (x s )]), where F -1 () is the inverse Fourier algorithm, F a (x t ) represents the target domain amplitude obtained by Fourier transforming the target domain image, F a (x s ) represents the source domain amplitude obtained by Fourier transforming the source domain image, F p (x s ) represents the source domain phase obtained by Fourier transforming the source domain image;

[0044] The virtual image obtained by low-frequency mapping is used as the second adaptive virtual physical ultrasound image I PHY_US '.

[0045] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the low-frequency amplitude cutoff window M β is defined as:

[0046] M β (h,w)=1 (h,w) ∈[-βH:βH,-βW:βW]

[0047] Among them, β∈(0,1), h represents the height of the window, w represents the width of the window, H represents the image feature height, and W represents the image feature width.

[0048] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the ultrasound and CT volume registration method provided by the present invention also includes:

[0049] A domain discriminator is used to determine whether the virtual physical ultrasound image comes from the CT domain or the ultrasound domain. The domain discriminator is expressed as:

[0050] G d =Sigmoid(u T G f (x)+z), where G drepresents the output of the domain discriminator, Sigmoid represents the activation function, G f (x) represents feature extraction, u T represents the transpose of the feature vector, and z represents the noise sampled from the latent space;

[0051] Definition i is the domain label of the i-th sample, and the discriminant loss of the domain discriminator is defined as:

[0052] Among them, L d is the discriminative loss, G f (x i ) represents the feature extraction result of the i-th sample.

[0053] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the slice-to-volume registration module adopts self-supervisory constraints, and the constraints include:

[0054] Among them, L POS is the pose consistency loss function, represents the spatial offset predicted by the slice-to-volume registration module, p t Indicates the actual spatial offset, represents the spatial rotation predicted by the slice-to-volume registration module, p r Indicates the actual amount of rotation in space;

[0055] And / or, extracting a slice in the CT volume image using the posture predicted by the slice-to-volume registration module, subject to the following constraints: Among them, L SIM is the image consistency loss function, N represents the number of extracted slices, represents the image predicted by the slice-to-volume registration module, y i Represents the image of the i-th slice extracted.

[0056] According to another aspect of the present invention, an ultrasound and CT volume registration device is provided, comprising a slice-to-volume registration module, which uses the ultrasound and CT volume registration method as described above to complete registration training.

[0057] According to another aspect of the present invention, there is provided an ultrasound-based surgical navigation system, comprising an ultrasound imaging device, a CT imaging device, a display, and the ultrasound and CT volume registration device as described above, wherein the CT imaging device is configured to acquire a CT volume image, and the ultrasound imaging device is configured to acquire a real ultrasound image;

[0058] The CT volume registration device is configured to predict the position information of the real ultrasound image within the CT volume image;

[0059] The display is configured to display the registration information output by the CT volume registration device.

[0060] The beneficial effects brought by the technical solution provided by the present invention are as follows:

[0061] a. Ultrasound images and CT images are fused through deep learning models, eliminating the need for tracking equipment and reducing costs;

[0062] b. Generate a large amount of virtual data through self-supervised learning and diffusion model-based image conversion technology, reduce the modality difference between ultrasound images and CT images, and build a real-time, wide-range and accurate ultrasound-CT registration model;

[0063] c. Through domain adaptation technology, the domain difference between the generated virtual image and the real image is reduced, so that the model has better registration effect and robustness in real scenes;

[0064] d. The virtual-to-real domain adaptation technology is used to improve the generation quality of physical ultrasound and reduce the domain difference between physical ultrasound and real scene ultrasound. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A schematic diagram of a physically guided virtual-to-real domain adaptive ultrasound and CT volume registration architecture provided for an exemplary embodiment of the present invention;

[0067] Figure 2 A schematic flow chart of a method for adaptive virtual-to-real domain ultrasound and CT volume registration based on physical guidance provided by an exemplary embodiment of the present invention;

[0068] Figure 3 A schematic diagram of a framework of an ultrasound-based surgical navigation system provided for an exemplary embodiment of the present invention;

[0069] Figure 4 Visualization results of the effects of different registration methods provided by the embodiments of the present invention;

[0070] Figure 5 The S provided in the embodiment of the present invention 3 Virtual ultrasound renderings of VR models;

[0071] Figure 6 A schematic diagram showing the comparison between the registration method provided in an embodiment of the present invention and the registration method of in vitro optical positioning tracking in clinical data. DETAILED DESCRIPTION

[0072] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0073] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0074] In one embodiment of the present invention, a physically guided virtual-to-real domain adaptive ultrasound and CT volume registration method is provided, and the architecture module for realizing the registration is as follows: Figure 1 As shown in Figure 1, the architecture includes a physical US projection module, a physical US generation module, a synthetic-to-real domain adaptation module, and a slice-to-volume registration module (SVR module). The above four modules are collectively referred to as S 3 VR module, that is, the implementation subject of the virtual to real domain adaptive ultrasound and CT volume registration method based on physical guidance provided in this embodiment is S 3 VR module.

[0075] The input of the Physical US Projection module is the CT slice (CT Slice) cut from the CT volume (CT Volume). The Physical US Projection module generates a first virtual physical ultrasound image I by physically mapping the CT slice. PHY_CT ;

[0076] The input of the Physical US Generation module is the real ultrasound image and the initial image x T , using the conditional diffusion model to learn in the spatial domain of real ultrasound to generate the second virtual physical ultrasound image I PHY_US ;

[0077] The input of the Synthetic-to-Real Domain Adaptation module is the first virtual physical ultrasound image I PHY_CT , then use the virtual to real domain adaptive technology to output I PHY_CT The first adaptive virtual physical ultrasound image I after corresponding domain adaptation PHY_CT '; If the input is the second virtual physical ultrasound image I PHY_US , then use the virtual to real domain adaptive technology to output I PHY_US The second adaptive virtual physical ultrasound image I after corresponding domain adaptation PHY_US ';

[0078] The input of the SVR module is the CT volume image and the adaptive virtual physical ultrasound image (I PHY_CT ' and / or I PHY_US '), the SVR module adopts a dual-path CNN structure, one of which is configured to extract features from the CT volume image, and the other is configured to extract features from the adaptive virtual physical ultrasound image, and fuse the features extracted from the dual paths; the SVR module is also configured with a multi-layer perceptron, which predicts the position information of the adaptive virtual physical ultrasound image in the CT volume image based on the fused features, specifically six degrees of freedom (6DoF, 6Degree of Freedom), including the movement freedom and rotation freedom along the three axes of x, y, and z.

[0079] Based on the above framework, the virtual-to-real domain adaptive ultrasound and CT volume registration method based on physical guidance provided by the embodiment of the present invention includes the following steps: Figure 2 The process shown:

[0080] Performing CT imaging on the target object to obtain a CT volume image;

[0081] Using virtual-to-real domain adaptive technology, an adaptive virtual physical ultrasound image of the target object is obtained, and the adaptive virtual physical ultrasound image reduces the modality difference between itself and CT compared with real ultrasound;

[0082] The CT volume image and the adaptive virtual physical ultrasound image are input into a preset slice-to-volume registration module (SVRmodule). The SVR module adopts a dual-path CNN structure, wherein one path is configured to extract features from the CT volume image, and the other path is configured to extract features from the adaptive virtual physical ultrasound image, and the features extracted from the dual paths are fused. The SVR module is also configured with a multilayer perceptron (MLP for short), which predicts the position information of the adaptive virtual physical ultrasound image within the CT volume image based on the fused features, thereby completing the registration between the ultrasound and CT volume in the virtual to real domain adaptively based on physical guidance.

[0083] First embodiment

[0084] First, we randomly select posture parameters in the liver area of ​​CT scans, cut out multiple slices as input data, and predict the posture through the slices to form a self-supervised learning framework. Then, based on the improvement of ray tracing and ultrasound echo generation formula, we generate the first virtual physical ultrasound image I by physical mapping the CT slices. PHY_CT .

[0085] The adaptive virtual physical ultrasound image in this embodiment is obtained through the Physical US Projection module and the Synthetic-to-Real Domain Adaptation module in sequence, wherein the CT physical mapping made by the Physical US Projection module is defined as:

[0086] E i (d) = R i (d)+B i (d), where d is the sensor depth, E i (d) represents the mapping of the ultrasonic rays emitted by the sensor at position d, R i (d) represents the reflected energy from the tissue interface in the body, B i (d) represents the backscattered energy;

[0087] in, Among them, I i (d) = e -αd , which represents the energy attenuation caused by the reflection of the tissue interface in the body, e represents the natural constant, α represents the attenuation coefficient, and Z i(d) represents the reflection coefficient, P(d) represents the point scattering function, G(d) represents the boundary, Represents the cross product operation;

[0088] in, represents the texture, which is generated by a random Gaussian process controlled by the speckle distribution parameters μ0, μ1 and δ0.

[0089] The Synthetic-to-Real Domain Adaptation module converts the first virtual physical ultrasound image into a first adaptive virtual physical ultrasound image by:

[0090] Considering that the amplitude of low-frequency components mainly contains information about low-level image features, which can usually understand the overall or macroscopic changes in brightness, intensity distribution, contrast, and texture intensity, this method helps to coordinate the domain conversion between virtual and real images. PHY_CT and I PHY_US The frequency domain mapping is performed between. The Fourier transform F of a two-dimensional grayscale single channel image I is:

[0091] This is achieved by using the FFT algorithm. Through iFFT, the phase and amplitude of the image can be mapped back to the image space;

[0092] For the region of β∈(0,1), there is a low-frequency amplitude cutoff window M β , whose value is 0, and is defined as: M β (h,w)=1 (h,w) ∈[-βH:βH,-βW:βW]; where h represents the height of the window, w represents the width of the window, H represents the image feature height, and W represents the image feature width.

[0093] Define the low frequency amplitude cutoff window M β ; Use the conditional diffusion model to learn in the spatial domain of real ultrasound to generate a second virtual physical ultrasound image I PHY_US ; With the second virtual physical ultrasound image I PHY_US As the target domain image, the first virtual physical ultrasound image I PHY_CT For the source domain image, the low-frequency mapping from the source domain to the target domain is defined as:

[0094] I s→t =F -1 ([M β ·F a (x t )+(1-M β )·F a (x s ),Fp (x s )]), where F -1 () is the inverse Fourier algorithm, F a (x t ) represents the target domain amplitude obtained by Fourier transforming the target domain image, F a (x s ) represents the source domain amplitude obtained by Fourier transforming the source domain image, F p (x s ) represents the source domain phase obtained by Fourier transforming the source domain image;

[0095] The virtual image obtained by low-frequency mapping is used as the first adaptive virtual physical ultrasound image I PHY_CT '.

[0096] That is, in order to reduce the domain difference between the synthetic image and the real image, we introduce the low-frequency amplitude of the real ultrasound into I PHY_US , and further I PHY_US The low frequency amplitude is introduced into I PHY_CT , combined with domain discriminator to achieve domain adaptation.

[0097] Then, the adaptive virtual physical ultrasound image is integrated into the SVR module together with the CT volume to facilitate the pose regression model training. Finally, the training process utilizes a two-stream loss function to simultaneously consider the pose estimation accuracy and image similarity, forming a closed-loop self-supervised learning paradigm.

[0098] By using virtual to real domain adaptation technology, the generation quality of physical ultrasound is improved, and the domain difference between physical ultrasound and real scene ultrasound is reduced.

[0099] Second embodiment

[0100] The adaptive virtual physical ultrasound image in this embodiment is obtained through the Physical US Generation module and the Synthetic-to-Real Domain Adaptation module in sequence, wherein the conditional diffusion model used in the Physical US Generation module defines a Markov chain in which random noise is gradually integrated into the data, which is called the forward process;

[0101] Sampling from the data distribution corresponding to the true data x0, the forward process adds Gaussian noise to the samples in T time steps:

[0102]

[0103] Where T is the diffusion time step, represents the process of adding Gaussian noise to the image at time t and the image at time t-1, βt represents t constants for controlling the noise step size or learning variance, t is a time between 1 and T steps, N represents a non-negative integer set, x0 represents a virtual image at time 0, q(x) represents a virtual image at a non-zero time, and x t represents the virtual image at time t, x t-1 represents the virtual image at time t-1, I represents the diagonal covariance matrix, It means that when x0 is given, x1,x2,...,x T The joint probability distribution of , ∏ is a continuous product operation.

[0104] Further, we sample x from x0 by the following formula: t : Among them, q(x t |x0) means sampling x from x0 t , let α t =1-β t ,and

[0105] The second virtual physical ultrasound image I is generated by learning the inverse process and moving backward with the parameterized diagonal Gaussian transform. PHY_US :

[0106] Among them, p θ (x t-1 |x t ) is the sampling distribution of the unconditionally guided inverse process, μ θ (x t ,t) represents the mean of the conditional distribution obtained by network training, The value is Empirical parameters of

[0107] If there is an additional guiding signal y, the sampling distribution transforms to:

[0108] in, is the sampling distribution of the inverse process with an additional pilot signal y, is the normalization constant, represents the conditional random sampling process guided by the classifier, represents the time step x t-1 The conditional y classifier for .

[0109] By utilizing the alignment between the spatial feature map and the features at the corresponding spatial position, the generated second virtual physical ultrasound image I is guided PHY_US Shares similar structure with the original real ultrasound image:

[0110] Among them, x′ t represents the real ultrasound image at time t, F(x t ,x′ t ,t) represents the virtual image at time t, It represents the image encoder E I 'j-th layer spatial feature, C j H represents the number of image feature channels at the jth layer of the image encoder, j represents the image feature height of the jth layer of the image encoder, W j Represents the image feature width of the j-th layer of the image encoder.

[0111] The Synthetic-to-Real Domain Adaptation module converts the second virtual physical ultrasound image into a second adaptive virtual physical ultrasound image by:

[0112] Define the low frequency amplitude cutoff window M β ;

[0113] Acquire a real ultrasound image as the target domain image, and use the second virtual physical ultrasound image I PHY_US For the source domain image, the low-frequency mapping from the source domain to the target domain is defined as:

[0114] I s→t =F -1 ([M β ·F a (x t )+(1-M β )·F a (x s ),F p (x s )]), where F -1 () is the inverse Fourier algorithm, F a (x t ) represents the target domain amplitude obtained by Fourier transforming the target domain image, F a (x s ) represents the source domain amplitude obtained by Fourier transforming the source domain image, F p (x s ) represents the source domain phase obtained by Fourier transforming the source domain image;

[0115] The virtual image obtained by low-frequency mapping is used as the second adaptive virtual physical ultrasound image I PHY_US '.

[0116] In the first and second embodiments, the low-frequency amplitude cutoff window M β is defined as:

[0117] Mβ (h,w)=1 (h,w) ∈[-βH:βH,-βW:βW]

[0118] Among them, β∈(0,1), h represents the height of the window, w represents the width of the window, H represents the image feature height, and W represents the image feature width.

[0119] The ultrasound and CT volume registration method provided in the embodiment of the present invention also includes:

[0120] A domain discriminator is used to determine whether the virtual physical ultrasound image comes from the CT domain or the ultrasound domain. The domain discriminator is expressed as:

[0121] G d =Sigmoid(u T G f (x)+z), where G d represents the output of the domain discriminator, Sigmoid represents the activation function, G f (x) represents feature extraction, u T represents the transpose of the feature vector, and z represents the noise sampled from the latent space;

[0122] Definition i is the domain label of the i-th sample, and the discriminant loss of the domain discriminator is defined as:

[0123] Among them, L d is the discriminative loss, G f (x i ) represents the feature extraction result of the i-th sample.

[0124] In addition to the discriminative loss, one or both of the following pose consistency loss and image consistency loss can also be defined:

[0125] The slice-to-volume registration module adopts self-supervisory constraints, and the constraints include:

[0126] Among them, L POS is the pose consistency loss function, represents the spatial offset predicted by the slice-to-volume registration module, p t Indicates the actual spatial offset, represents the spatial rotation predicted by the slice-to-volume registration module, p r Indicates the actual amount of rotation in space;

[0127] And / or, extracting a slice in the CT volume image using the posture predicted by the slice-to-volume registration module, subject to the following constraints: Among them, L SIMis the image consistency loss function, N represents the number of extracted slices, represents the image predicted by the slice-to-volume registration module, y i Represents the image of the i-th slice extracted.

[0128] Considering the diversity of internal tissue structures of different patients, the SVR module proposed in this embodiment is equipped with a unique retrained MLP for each patient. The MLP head is initially set according to the average weight obtained in the training phase, and then fine-tuned and optimized using ultrasound obtained from the scans of new patients to mitigate differences between patients and improve accuracy. The dual-stream CNN structure extracts features from ultrasound images and CT respectively. Since ultrasound images are two-dimensional imaging, two-dimensional convolution is first used to increase the dimension of ultrasound images, and the features of the two are fused through splicing and convolution modules. Finally, the 6DoF of ultrasound in CT is predicted through the MLP layer.

[0129] During the training phase, the public BTCV dataset was used in the process of physical ultrasound mapping, which labeled various abdominal tissues; during the physical ultrasound generation process, ultrasound images of the subcostal, intercostal and upper abdominal areas of multiple volunteers were collected through an ultrasound probe integrated with an optical positioning and tracking system.

[0130] In the testing phase, a dual-modal abdominal body membrane is used as the test object, and only two types of data are input: the first is the CT slice and its posture when cropped, and the second is the ultrasound image. When the model retrains the MLP layer, the first type of data is input. After the physical ultrasound is virtually generated, the SVR module receives three inputs: CT volume, virtual physical ultrasound, and a set of posture parameters, and fine-tunes the training MLP layer. In the verification phase of real ultrasound, the second type of data is input to predict the posture of ultrasound in CT.

[0131] Three evaluation indicators are used to evaluate and compare the performance of the model: the transmission error (TE) between the predicted and true poses, the rotation error (RE) between the predicted and true poses, and the normalized cross-correlation (NCC) between the input CT image and the image sampled from the predicted pose. The smaller the TE and RE values ​​are, the closer the ground truth and the predicted pose are to each other, which may indicate a more accurate prediction; the larger the NCC value is, the higher the similarity between the two images is, which means that the prediction of the ultrasound position is more accurate.

[0132] In the comparative evaluation, S 3 The VR model is compared with the traditional iterative image registration method. The baseline method uses LC2 The image similarity was evaluated using a metric and iteratively optimized using the Powell optimizer. In the field of image transformation-based registration methods, the PSSS (Patient-Specific Self-supervised) method was used (previously applied to the registration task between X-ray and CT). As a supplement, we also considered a unimodal SVR method, PlaneInVol (Learning to Map 2D Ultrasound Images into 3D Space with Minimal Human Annotation), which uses an inter-layer attention mechanism to identify the relationship between ultrasound slices, and a cross-modal 3D image registration method, AttentionReg (Cross-modal Attention for MRI and UltrasoundVolume Registration), which promotes deep feature interactions between 3D ultrasound and CT through an attention mechanism.

[0133] The evaluation indexes of this embodiment compared with the traditional registration method are shown in Table 1:

[0134] Table 1 compares with traditional methods

[0135]

[0136]

[0137] From Table 1, it can be seen that the traditional method with range initialization produces the best registration effect. However, after the range is expanded, its performance decreases significantly, especially under the interference of elastic deformation, the traditional method fails to register obviously. The method of the present invention does not require range initialization, and can achieve real-time registration, and has better robustness in the case of breathing elasticity.

[0138] Direct CT to ultrasound conversion method PSSS and cross-modality S 3 The evaluation indicators of VR model comparison are shown in Table 2:

[0139] Table 2 Comparison of PSSS and cross-modal SVR methods

[0140]

[0141] Figure 4 Visualize the effects of different registration methods: Figure 4 The first row of sub-images (a) are ultrasound images, the first row of sub-images (b) are CT slice image labels, the first row of sub-images (c) are the registration effects of the PSSS+SVR module, and the first row of sub-images (d) are S3 The registration effect of the VR+SVR module. The first row of sub-images (e) are S 3 The registration effect of VR+PlaneInVol module. The first row of sub-images (f) are S 3 Figure 2. Registration effect of VR+AttentionReg module. In the first row, the difference is not obvious in the predicted CT slices. In the fused image of ultrasound thermal map and CT image shown in the second row, the difference between ultrasound and predicted CT can be directly observed, reflecting the registration effect. The arrows in sub-images (b) and (c) in the first row indicate the difference between the comparison method and the label image of the original image, and the arrows in sub-images (b), (d), (e) and (f) in the second row indicate the difference between the comparison method and the label image in the fused image.

[0142] From Table 2 and Figure 4 It can be seen that the method of directly converting CT into ultrasound, which is also a method of generating virtual data for training ultrasound and CT registration models through self-supervision, does not perform well. The method provided in this embodiment reduces the modal difference between CT and ultrasound, and narrows the field difference between virtual images and real images, achieving better performance. In addition, the present invention can replace the SVR module with other carefully designed posture regression networks to achieve better results.

[0143] Compared with the existing cross-modal SVR method, the present invention adopts a self-supervised learning method to alleviate the problem of data shortage; physical ultrasound is generated from CT and ultrasound images respectively through physical mapping and diffusion model, which greatly reduces the modal difference between ultrasound and CT; the domain adaptation technology adopted reduces the domain difference between virtual and real images; by solving these three points, an accurate and robust ultrasound and CT registration model is constructed.

[0144] Compared with ultrasound three-dimensional reconstruction and re-registration, the present invention does not require long-term ultrasound scanning and reconstruction of the patient during surgery, thus reducing the intraoperative time and eliminating cumulative errors.

[0145] For ablation research, supervised training means that each ultrasound has its pose label in CT space during training to train the SVR module. Unsupervised training uses the proposed self-supervised method to train the SVR module. Table 3 shows the effects of supervised / unsupervised training of SVR modules corresponding to different modules:

[0146] Table 3 Effects of supervised / unsupervised training of SVR modules

[0147]

[0148] Figure 5 Shows a visualization of S 3 Virtual ultrasound effect of VR model, including: Figure 5 Sub-image (a) is the labeled slice of the CT abdomen annotation, sub-image (b) is the real ultrasound image, and sub-image (c) is the first virtual physical ultrasound image I PHY_CT , sub-image (d) is the second virtual physical ultrasound image I PHY_US , sub-image (e) is the first adaptive virtual physical ultrasound image I PHY_CT ', sub-image (f) is the second adaptive virtual physical ultrasound image I PHY_US '; By comparison, I PHY_CT 'The modal difference with the real ultrasound image is compared with the real CT image, I PHY_CT Smaller, I PHY_US 'The modal difference with the real CT image is compared with the real ultrasound image, I PHY_US Smaller.

[0149] Ablation experiments were performed on each module: when only the physical ultrasound mapping process was used, the significant modality difference between the real ultrasound and the physical ultrasound led to poor registration performance, while when the physical ultrasound mapping and physical ultrasound generation were combined, the registration performance was significantly improved. The introduction of virtual-to-real guided by frequency domain signals further enhanced the model, because the low-frequency signals from the real ultrasound were transferred to the physical ultrasound, making the generated physical ultrasound more representative of the real ultrasound. In addition, the addition of the domain discriminator made the virtual physical ultrasound closer to the real ultrasound, thereby further reducing the modality difference between ultrasound and CT. In addition, the data requirements of the model were studied. When only 500 input images were provided, the pose fitting model performed poorly, while the results were similar when 1000 and 2000 images were provided. Therefore, it can be inferred that 1000 random poses are required for model fine-tuning on different patients to obtain better fitting results.

[0150] Figure 6 The comparison between the registration method of this embodiment and the registration method of in vitro optical positioning tracking in clinical data is shown, wherein: Figure 6 Sub-figure (a) is the preoperative planning CT image, sub-figure (b) is the post-ablation verification CT image, sub-figure (c) is the CT section image obtained by optical tracking, and sub-figure (d) is the S 3 The image of the CT section predicted by the VR model, sub-image (e) is the real ultrasound image. Preoperative planning includes cross-layer needle insertion; after the ablation surgery, the ablation site is checked by CT scanning to determine the accuracy of the puncture. Comparing sub-image (c) with sub-image (d), it can be seen that the ultrasound and CT volume registration method provided in this embodiment can achieve an effect comparable to that of the CT section obtained by optical tracking without the use of an in vitro optical positioning and tracking device. Compared with in vitro calibration and registration, the present invention does not require expensive optical positioning devices, is not affected by external occlusions and can reduce costs.

[0151] In one embodiment of the present invention, an ultrasound and CT volume registration device is provided, comprising a slice-to-volume registration module, which uses the ultrasound and CT volume registration method as described above to complete registration training;

[0152] The embodiment also provides an ultrasound-based surgical navigation system, such as Figure 3 As shown, the surgical navigation system includes an ultrasound imaging device, a CT imaging device, a display, and the ultrasound and CT volume registration device as described above, wherein the CT imaging device is configured to acquire a CT volume image, and the ultrasound imaging device is configured to acquire a real ultrasound image;

[0153] The CT volume registration device is configured to predict the position information of the real ultrasound image within the CT volume image;

[0154] The display is configured to display the registration information output by the CT volume registration device.

[0155] Specifically, the CT imaging device acquires a CT volume image in the preoperative examination stage, the ultrasound imaging device acquires a real ultrasound image during the operation, and the SVR Module of the ultrasound and CT volume registration device preliminarily uses the CT volume data and the corresponding first adaptive virtual physical ultrasound image I PHY_CT ' / Second Adaptive Virtual Physical Ultrasound Image I PHY_US ' to complete the training.

[0156] The SVR Module for performing ultrasound and CT volume registration in the ultrasound and CT volume registration device provided in this embodiment implements registration training by utilizing the ultrasound and CT volume registration method based on physical guidance and adaptive virtual to real domain provided in the above embodiment, so that the SVR Module has a function of predicting the orientation of the ultrasound image in the CT volume image. The contents of the embodiment of the ultrasound and CT volume registration method based on physical guidance and adaptive virtual to real domain are incorporated into the embodiment of the surgical navigation system by full reference herein and will not be repeated herein.

[0157] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0158] The above is only a specific implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A physically guided virtual-to-real domain adaptive ultrasound and CT volume registration method, characterized in that: The following steps are involved: Performing CT imaging on the target object to obtain a CT volume image; Using virtual-to-real domain adaptive technology, an adaptive virtual physical ultrasound image of the target object is obtained, and the adaptive virtual physical ultrasound image reduces the modality difference between itself and CT compared with real ultrasound; The CT volume image and the adaptive virtual physical ultrasound image are input into a preset slice-to-volume registration module, wherein the slice-to-volume registration module adopts a dual-path CNN structure, wherein one path is configured to perform feature extraction on the CT volume image, and the other path is configured to perform feature extraction on the adaptive virtual physical ultrasound image, and the features extracted from the dual paths are fused; the slice-to-volume registration module is also configured with a multi-layer perceptron, which predicts the position information of the adaptive virtual physical ultrasound image within the CT volume image based on the fused features.

2. The ultrasound and CT volume registration method according to claim 1, characterized in that: The adaptive virtual physical ultrasound image is obtained by: Cropping the CT volume image to obtain a plurality of CT slice data; The CT slices are mapped physically to generate the first virtual physical ultrasound image I PHY_CT ; By using the virtual-to-real domain adaptation technology, a first adaptive virtual physical ultrasound image I corresponding to the first virtual physical ultrasound image is obtained after domain adaptation. PHY_CT '.

3. The ultrasound and CT volume registration method according to claim 2, characterized in that: The CT physical map is defined as: E i (d) = R i (d)+B i (d), where d is the sensor depth, E i (d) represents the mapping of the ultrasonic rays emitted by the sensor at position d, R i (d) represents the reflected energy from the tissue interface in the body, B i (d) represents the backscattered energy; in, Among them, I i (d) = e -αd , which represents the energy attenuation caused by the reflection of the tissue interface in the body, e represents the natural constant, α represents the attenuation coefficient, and Z i (d) represents the reflection coefficient, P(d) represents the point scattering function, G(d) represents the boundary, Represents the cross product operation; in, represents the texture, which is generated by a random Gaussian process controlled by the speckle distribution parameters μ0, μ1 and δ0.

4. The ultrasound and CT volume registration method according to claim 1, characterized in that: The adaptive virtual physical ultrasound image is obtained by: The conditional diffusion model is used to learn in the spatial domain of real ultrasound to generate a second virtual physical ultrasound image I PHY_US ; By using the virtual-to-real domain adaptation technology, a second adaptive virtual physical ultrasound image I corresponding to the second virtual physical ultrasound image is obtained after domain adaptation. PHY_US '.

5. The ultrasound and CT volume registration method according to claim 4, characterized in that: The conditional diffusion model defines a Markov chain in which random noise is gradually incorporated into the data, called the forward process; Sampling from the data distribution corresponding to the true data x0, the forward process adds Gaussian noise to the samples in T time steps: Where T is the diffusion time step, represents the process of adding Gaussian noise to the image at time t and the image at time t-1, βt represents t constants for controlling the noise step size or learning variance, t is a time between 1 and T steps, N represents a non-negative integer set, x0 represents the virtual image at time 0, x t represents the virtual image at time t, x t-1 represents the virtual image at time t-1, I represents the diagonal covariance matrix, It means that when x0 is given, x1,x2,...,x T The joint probability distribution of , ∏ is a continuous product operation.

6. The ultrasound and CT volume registration method according to claim 5, characterized in that: Sample x from x0 by the following formula t : Among them, q(x t |x0) means sampling x from x0 t , let α t =1-β t ,and 7. The ultrasound and CT volume registration method according to claim 5, characterized in that: The second virtual physical ultrasound image I is generated by learning the inverse process and moving backward with the parameterized diagonal Gaussian transform. PHY_US : Among them, p θ (x t-1 |x t ) is the sampling distribution of the unconditionally guided inverse process, μ θ (x t ,t) represents the mean of the conditional distribution obtained by network training, The value is Empirical parameters of If there is an additional guiding signal y, the sampling distribution transforms to: in, is the sampling distribution of the inverse process with an additional pilot signal y, is the normalization constant, represents the conditional random sampling process guided by the classifier, represents the time step x t-1 The conditional y classifier for .

8. The ultrasound and CT volume registration method according to any one of claims 5 to 7, characterized in that: By utilizing the alignment between the spatial feature map and the features at the corresponding spatial position, the generated second virtual physical ultrasound image I is guided PHY_US Shares similar structure with the original real ultrasound image: Among them, x′ t represents the real ultrasound image at time t, F(x t ,x′ t ,t) represents the virtual image at time t, It represents the image encoder E I 'j-th layer spatial feature, C j H represents the number of image feature channels at the jth layer of the image encoder, j represents the image feature height of the jth layer of the image encoder, W j Represents the image feature width of the j-th layer of the image encoder.

9. The ultrasound and CT volume registration method according to claim 2, characterized in that: The virtual-to-real domain adaptation technology converts the first virtual physical ultrasound image into the first adaptive virtual physical ultrasound image in the following manner: Define the low frequency amplitude cutoff window M β ; The conditional diffusion model is used to learn in the spatial domain of real ultrasound to generate a second virtual physical ultrasound image I PHY_US ; With the second virtual physical ultrasound image I PHY_US As the target domain image, the first virtual physical ultrasound image I PHY_CT For the source domain image, the low-frequency mapping from the source domain to the target domain is defined as: I s→t =F -1 ([M β ·F a (x t )+(1-M β )·F a (x s ),F p (x s )]), where F -1 ( ) is the inverse Fourier algorithm, F a (x t ) represents the target domain amplitude obtained by Fourier transforming the target domain image, F a (x s ) represents the source domain amplitude obtained by Fourier transforming the source domain image, F p (x s ) represents the source domain phase obtained by Fourier transforming the source domain image; The virtual image obtained by low-frequency mapping is used as the first adaptive virtual physical ultrasound image I PHY_CT '.

10. The ultrasound and CT volume registration method according to claim 4, characterized in that: The virtual-to-real domain adaptation technology converts the second virtual physical ultrasound image into a second adaptive virtual physical ultrasound image by: Define the low frequency amplitude cutoff window M β ; Acquire a real ultrasound image as the target domain image, and use the second virtual physical ultrasound image I PHY_US For the source domain image, the low-frequency mapping from the source domain to the target domain is defined as: I s→t =F -1 ([M β ·F a (x t )+(1-M β )·F a (x s ),F p (x s )]), where F -1 () is the inverse Fourier algorithm, F a (x t ) represents the target domain amplitude obtained by Fourier transforming the target domain image, F a (x s ) represents the source domain amplitude obtained by Fourier transforming the source domain image, F p (x s ) represents the source domain phase obtained by Fourier transforming the source domain image; The virtual image obtained by low-frequency mapping is used as the second adaptive virtual physical ultrasound image I PHY_US '.

11. The ultrasound and CT volume registration method according to claim 9 or 10, characterized in that: The low frequency amplitude cutoff window M β is defined as: M β (h,w)=1 (h,w) ∈[-βH:βH,-βW:βW] Among them, β∈(0,1), h represents the height of the window, w represents the width of the window, H represents the image feature height, and W represents the image feature width.

12. The ultrasound and CT volume registration method according to any one of claims 1 to 7 or claim 9 or 10, characterized in that: Also includes: A domain discriminator is used to determine whether the virtual physical ultrasound image comes from the CT domain or the ultrasound domain. The domain discriminator is expressed as: G d =Sigmoid(u T G f (x)+z), where G d represents the output of the domain discriminator, Sigmoid represents the activation function, G f (x) represents feature extraction, u T represents the transpose of the feature vector, and z represents the noise sampled from the latent space; Definition i is the domain label of the i-th sample, and the discriminant loss of the domain discriminator is defined as: Among them, L d is the discriminative loss, G f (x i ) represents the feature extraction result of the i-th sample.

13. The ultrasound and CT volume registration method according to any one of claims 1 to 7 or claim 9 or 10, characterized in that: The slice-to-volume registration module adopts self-supervisory constraints, and the constraints include: Among them, L POS is the pose consistency loss function, represents the spatial offset predicted by the slice-to-volume registration module, p t Indicates the actual spatial offset, represents the spatial rotation predicted by the slice-to-volume registration module, p r Indicates the actual amount of rotation in space; And / or, extracting a slice in the CT volume image using the posture predicted by the slice-to-volume registration module, subject to the following constraints: Among them, L SIM is the image consistency loss function, N represents the number of extracted slices, represents the image predicted by the slice-to-volume registration module, y i Represents the image of the i-th slice extracted.

14. An ultrasound and CT volume registration device, characterized in that: It comprises a slice-to-volume registration module, which uses the ultrasound and CT volume registration method according to any one of claims 1 to 13 to complete the registration training.

15. An ultrasound-based surgical navigation system, characterized in that: The device comprises an ultrasound imaging device, a CT imaging device, a display, and the ultrasound and CT volume registration device according to claim 14, wherein the CT imaging device is configured to obtain a CT volume image, and the ultrasound imaging device is configured to obtain a real ultrasound image; The CT volume registration device is configured to predict the position information of the real ultrasound image within the CT volume image; The display is configured to display the registration information output by the CT volume registration device.

Images