Multi-mode offset image fusion reconstruction method based on conditional diffusion and related device
Through the multimode offset image fusion reconstruction method based on conditional diffusion, the feature offset and brightness spatial distribution problems caused by different acquisition times of CT and MRI images are solved, and the accurate fusion and three-dimensional reconstruction of the images are achieved, and the image is improved The quality and interpretability of the fusion.
Patent Information
- Application Number
- CN202411095527.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The prior art is difficult to effectively solve the problem of feature shifts and uneven brightness spatial distribution caused by different acquisition times of CT and MRI images, which affects the accuracy and quality of image fusion.
Using a multimode offset image fusion reconstruction method based on conditional diffusion, migrating MRI images are generated through unconditional diffusion sampling, and combined with state space feature extraction and collaborative task decoder, feature extraction and decoding operations are performed to generate and reconstruct CT images and tissue deformation fields. and MRI segment images to solve the problems of feature offset and uneven brightness distribution.
Accurate fusion segmentation and three-dimensional reconstruction of CT and MRI images are realized, which improves the quality and interpretability of image fusion, and solves the problems of feature offset and uneven brightness distribution.
Smart Images

Figure CN118628372B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a multi-mode offset image fusion reconstruction method based on conditional diffusion and a related device. Background Art
[0002] The clinical diagnosis and treatment process generates a large amount of medical imaging data, which provides a new opportunity for the improvement of medical imaging computer vision technology. However, most current studies focus on exploring the semantic recognition of individual medical images such as CT, MR, and ultrasound images. The deep learning frameworks used are mostly for individual CT or MR images, and cannot fuse the features of the two to achieve more accurate segmentation. The digestive tract and surrounding areas contain many tissues and organs, such as blood vessels, bile and pancreatic ducts, intestines, solid organs, etc. It is difficult for the same medical image to simultaneously take into account the boundary development requirements of different tissues and organs. Multimodal image fusion segmentation provides an effective way to achieve the segmentation and reconstruction of many tissues and organs around the surgical goal. However, the image features of CT, MR, etc. are significantly different. The training of some mainstream methods such as generative adversarial networks (GANs) is unstable, lacks interpretability, and is prone to mode collapse, which seriously affects the quality of generated samples.
[0003] Current image fusion technologies are mostly aimed at the fusion of visible light images and infrared images. Medical image fusion has attracted extensive discussion in the image fusion community due to its complexity and application significance. Medical image fusion can assist diagnosis and treatment by fusing multiple medical imaging modalities to accurately detect abnormal parts. Many methods have been designed recently to address the challenges brought by medical image fusion, and generative models have been widely used to model the distribution of fused images and achieve satisfactory batch fusion effects. Among them, the GAN network based on generative adversarial networks is dominant. The literature designs a dual adversarial fusion network TarDAL, which enables the fusion network with a generator and dual discriminators to learn from the differences while seeking common ground while reserving differences, thereby retaining the structural information of infrared targets and the texture details of visible light, and showing obvious advantages in efficiently processing challenging scenes. In order to make up for the defects of optical sensor imaging, some literature proposes a learnable feature adaptation network MSIS, which studies the complementarity of infrared and visible light images to improve scene perception in terms of visual information and viewing range, and achieves achievable and excellent stitching results. Recently, the transformer model based on the attention mechanism has aroused extensive discussion in the community due to its powerful information processing capabilities. Some literature has studied the image fusion function under this model and proposed the CDDFuse framework. Through the dual-branch Transformer-CNN architecture and the related driving loss function, the modality-specific features and shared features are extracted, the cross-modal feature modeling problem is solved, and positive fusion results are achieved in multiple fusion tasks.
[0004] The GAN network based on generative adversarial network has achieved excellent image generation results. However, the training of the GAN network is unstable, lacks interpretability, and is prone to mode collapse, which seriously affects the quality of the generated samples. In addition, due to the different acquisition times of medical images, there is often a large feature offset between the features of different images. The relatively limited feature field of view of existing methods makes it difficult to accurately correct the offset and fuse the features of different tissues. In medical images, there are often large differences between CT and MR images, and the brightness of the two is unevenly distributed in space. Existing technologies are difficult to solve this problem well. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a multi-mode offset image fusion reconstruction method based on conditional diffusion and related devices, which solves the problem of feature offset between CT and MRI images caused by different acquisition times and the problem of uneven brightness spatial distribution in the difference between CT and MRI images.
[0006] In order to solve the above technical problems, an embodiment of the present invention provides a multi-mode offset image fusion reconstruction method based on conditional diffusion, the method comprising:
[0007] Obtaining an original CT image, and performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migrated MRI image;
[0008] Inputting the migration MRI image and the original MRI image into a state space feature extraction module for feature extraction processing to obtain fusion feature data;
[0009] Inputting the fused feature data into a collaborative task decoder for decoding operation to generate a reconstructed CT image, a tissue deformation field and an MRI segmentation image, wherein the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder, and the reconstructed CT image is used to calculate the cyclic consistency loss with the original CT image;
[0010] A three-dimensional reconstruction process is performed based on the tissue deformation field and the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue.
[0011] Optionally, performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migrated MRI image includes:
[0012] gradually adding Gaussian noise to the original CT image to convert the original CT image into a Gaussian signal, and performing denoising processing on the Gaussian signal by using a denoising fraction matching method to generate a denoised image;
[0013] The probability of the original CT image is used as a priori condition to perform migration processing on the denoised image through likelihood correction optimization to generate a migrated MRI image.
[0014] Optionally, the likelihood correction optimization is to use a maximum expectation algorithm to calculate the similarity between the original CT image and the migrated MRI image; the maximum expectation algorithm includes two steps, E-step and M-step;
[0015] Among them, E-step is the calculation result of the conditional expectation of updating the latent variable x so as to perform the maximization step in M-step, and its calculation formula is as follows:
[0016] ;
[0017] ;
[0018] in, represents the expected value of its random variable; Represents the row index, ranging from 1 to H, where H is a constant; Represents the column index, ranging from 1 to W, where W is a constant; represents the de-drying diffusion step length; represents the observed value of the data point in the i-th row and j-th column in the difference between the migrated MRI image and the original CT image; represents the observed value of the data point in the i-th row and j-th column in the difference between the original CT image and the migrated MRI image; Represents the parameters of a normal distribution, used to describe data points and relationship; is a latent variable, representing the parameters of a normal distribution, used to describe the data points distribution.
[0019] Optionally, the step of inputting the migration MRI image and the original MRI image into a state space feature extraction module for feature extraction processing to obtain fusion feature data includes:
[0020] Inputting the migration MRI image and the original MRI image into the state space feature extraction module and performing feature extraction processing through N layers of VSS long-distance feature encoding to obtain a first extracted feature;
[0021] Performing feature patch fusion processing on the first extracted features to obtain first fused features;
[0022] The first fused feature is further subjected to feature extraction processing through N layers of VSS long-range feature encoding to obtain a second extracted feature;
[0023] The second extracted features are subjected to feature patch fusion processing to form fused feature data.
[0024] Optionally, the step of inputting the fused feature data into a collaborative task decoder for decoding to generate a reconstructed CT image, a tissue deformation field and an MRI segmentation image includes:
[0025] Inputting the fused feature data into the collaborative task decoder, and performing CT image reconstruction processing in a cyclic reconstruction decoder in the collaborative task decoder to obtain a reconstructed CT image;
[0026] Extracting and processing the offset information of the original CT image and the original MRI image in the deformation estimation decoder in the collaborative task decoder to obtain a tissue deformation field;
[0027] The blood vessel and tissue segmentation processing of the original MRI image is performed in the tissue segmentation decoder in the collaborative task decoder to obtain an MRI segmentation image.
[0028] Optionally, after obtaining the reconstructed CT image, the method further includes:
[0029] The node parameters of the state space feature extraction module are corrected based on the calculated cycle consistent loss of the reconstructed CT image and the original CT image.
[0030] Optionally, performing three-dimensional reconstruction processing based on the tissue deformation field and the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue includes:
[0031] A three-dimensional reconstruction process is performed based on the offset information in the tissue deformation field and the blood vessels and tissues in the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue.
[0032] In addition, an embodiment of the present invention further provides a multi-mode offset image fusion reconstruction device based on conditional diffusion, the device comprising:
[0033] Migration generation module: used for obtaining the original CT image, and performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migration MRI image;
[0034] Feature extraction module: used for inputting the migration MRI image and the original MRI image into the state space feature extraction module for feature extraction processing to obtain fusion feature data;
[0035] Decoding generation module: used for inputting the fused feature data into the collaborative task decoder for decoding operation, generating reconstructed CT images, tissue deformation fields and MRI segmentation images, wherein the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder, and the reconstructed CT image is used for calculating the cyclic consistency loss with the original CT image;
[0036] Three-dimensional reconstruction module: used for performing three-dimensional reconstruction processing based on the tissue deformation field and the MRI segmentation image to obtain a three-dimensional anatomical reconstruction image of the tissue.
[0037] In addition, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.
[0038] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above methods are implemented.
[0039] In an embodiment of the present invention, a multi-image fusion segmentation method generated by unconditional diffusion sampling satisfies the natural image prior and limits the similarity with the source image through likelihood correction; modeling natural images using a probability diffusion model can better generate details that are difficult to control with a manually designed loss function, thereby producing a visually perceptible image; at the same time, the problem of feature offset caused by the different acquisition times of CT and MRI images and the problem of uneven spatial distribution of brightness in the difference between CT and MRI images are specifically solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 is a flow chart of a multi-mode offset image fusion reconstruction method based on conditional diffusion in an embodiment of the present invention;
[0042] Figure 2 Schematic diagram of the structure of a multi-mode offset image fusion and reconstruction device based on conditional diffusion in an embodiment of the present invention;
[0043] Figure 3 It is a schematic diagram of the structure of the cloud server in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] For example, please refer to Figure 1 , Figure 1 It is a flowchart of a multi-mode offset image fusion reconstruction method based on conditional diffusion in an embodiment of the present invention.
[0046] like Figure 1 As shown, a multi-mode offset image fusion reconstruction method based on conditional diffusion, the method comprising:
[0047] S11: obtaining an original CT image, and performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migrated MRI image;
[0048] In the specific implementation process of the present invention, the migration processing of the original CT image based on unconditional diffusion sampling to generate a migration MRI image includes: gradually adding Gaussian noise to the original CT image, converting the original CT image into a Gaussian signal, and using a denoising score matching method to denoise the Gaussian signal to generate a denoised image; using the probability of the original CT image as a priori condition to migrate the denoised image through likelihood correction optimization to generate a migration MRI image.
[0049] Furthermore, the likelihood correction optimization is to use a maximum expectation algorithm to calculate the similarity between the original CT image and the migrated MRI image; the maximum expectation algorithm includes two steps, E-step and M-step;
[0050] Among them, E-step is the calculation result of the conditional expectation of updating the latent variable x so as to perform the maximization step in M-step, and its calculation formula is as follows:
[0051] ;
[0052] ;
[0053] in, represents the expected value of its random variable; Represents the row index, ranging from 1 to H, where H is a constant; Represents the column index, ranging from 1 to W, where W is a constant; represents the de-drying diffusion step length; represents the observed value of the data point in the i-th row and j-th column in the difference between the migrated MRI image and the original CT image; represents the observed value of the data point in the i-th row and j-th column in the difference between the original CT image and the migrated MRI image; Represents the parameters of a normal distribution, used to describe data points and relationship; is a latent variable, representing the parameters of a normal distribution, used to describe the data points distribution.
[0054] Specifically, we first need to obtain the original CT image , and then generate the new modality initial target domain image, i.e. the initial migration MRI image, through unconditional diffusion sampling .
[0055] Specifically, firstly, given the original CT image Gradually add Gaussian noise to transform the clean sample into an almost Gaussian signal, and then generate .
[0056] Then the original CT image The probability is used as a priori condition to keep more information in the source image through likelihood correction, that is, Generates something closer to of , and finally obtain the migration MRI image; the likelihood correction optimization uses the EM algorithm module to calculate the maximum similarity x between the source image and the migration image to maximize the brightness space of the source image. This optimization problem can be transformed into a maximum likelihood inference problem. The EM algorithm includes two steps, E-step and M-step. The E-step updates the calculation result of the conditional expectation of the latent variable x so as to perform the maximization step in the M-step. The calculation formula is as follows:
[0057] ;
[0058] ;
[0059] in, represents the expected value of its random variable; Represents the row index, ranging from 1 to H, where H is a constant; Represents the column index, ranging from 1 to W, where W is a constant; represents the de-drying diffusion step length; represents the observed value of the data point in the i-th row and j-th column in the difference between the migrated MRI image and the original CT image; represents the observed value of the data point in the i-th row and j-th column in the difference between the original CT image and the migrated MRI image; Represents the parameters of a normal distribution, used to describe data points and relationship; is a latent variable, representing the parameters of a normal distribution, used to describe the data points distribution.
[0060] First, the conditional expectation of the latent variable is calculated based on the current latent variable and the observed data. This involves calculating the Q function, which is the expected log-likelihood function. Based on the calculated conditional expectation, the value of the latent variable x is updated to maximize the Q function, thereby obtaining a better estimate. The E-step is iterated multiple times until the value of the latent variable x converges to the optimal solution. In each iteration, the conditional expectation of the latent variable x is updated, and the latent variable is updated based on the new conditional expectation value.
[0061] After calculating the conditional expectation of the latent variable x in the E-step, the goal of the M-step is to maximize the likelihood function and find the model parameters that maximize the likelihood function, that is, to use the semi-quadratic segmentation algorithm to find the minimum value of x. Depend on After likelihood correction, the first step of M-step is based on The semi-quadratic segmentation algorithm is used to iteratively solve the minimization problem and find the optimal k value to update the parameter k. The second step is to use the semi-quadratic segmentation algorithm to iteratively solve the minimization problem based on k and find the optimal u value. The parameter u can be returned to modify and update the parameter k. The third step is to combine the x, parameter k and ; Use the semi-quadratic partitioning algorithm to iteratively solve the minimization problem to find the optimal Through the iterative process of M-step, the estimated results of model parameters can be gradually optimized, thereby improving the model's fitting ability and prediction accuracy. M-step and E-step are performed alternately until the algorithm converges, and the optimal model parameters and latent variable estimation results are obtained. Estimated , after the final denoising step, the migration MRI image is generated .
[0062] S12: inputting the migration MRI image and the original MRI image into a state space feature extraction module for feature extraction processing to obtain fusion feature data;
[0063] In the specific implementation process of the present invention, the migration MRI image and the original MRI image are input into the state-space feature extraction module for feature extraction processing to obtain fused feature data, including: inputting the migration MRI image and the original MRI image into the state-space feature extraction module for feature extraction processing through N layers of VSS long-distance feature encoding to obtain a first extracted feature; performing feature patch fusion processing on the first extracted feature to obtain a first fused feature; performing feature extraction processing on the first fused feature again through N layers of VSS long-distance feature encoding to obtain a second extracted feature; performing feature patch fusion processing on the second extracted feature to form fused feature data.
[0064] Specifically, after migrating the original CT image to a unified migrated MRI image, image feature fusion is required. Due to the different image acquisition times, there is often a large feature offset between the features of different images, and this is even more true for the digestive tract adjacent to the lungs. In order to ensure the accurate fusion and segmentation of different image features, an unsupervised collaborative image feature adaptive fusion architecture based on conditional diffusion migration is designed. The framework takes the migrated MRI image and the original MRI image migrated to the unified image domain as input, and synchronously extracts features of different images in a shared encoder; after inputting the shared encoder, it first passes through N layers of VSS long-distance feature encoding, and the extracted feature patches are fused, and then pass through N layers of VSS long-distance feature encoding. The features extracted after patch fusion form fused feature data.
[0065] S13: inputting the fused feature data into a collaborative task decoder for decoding operation to generate a reconstructed CT image, a tissue deformation field and an MRI segmentation image, wherein the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder, and the reconstructed CT image is used to calculate the cyclic consistency loss with the original CT image;
[0066] In the specific implementation process of the present invention, the fused feature data is input into the collaborative task decoder for decoding operation to generate a reconstructed CT image, a tissue deformation field and an MRI segmentation image, including: inputting the fused feature data into the collaborative task decoder, and performing CT image reconstruction processing in a cyclic reconstruction decoder in the collaborative task decoder to obtain a reconstructed CT image; performing offset information extraction processing between the original CT image and the original MRI image in the deformation estimation decoder in the collaborative task decoder to obtain a tissue deformation field; performing blood vessel and tissue segmentation processing of the original MRI image in the tissue segmentation decoder in the collaborative task decoder to obtain an MRI segmentation image.
[0067] Furthermore, after obtaining the reconstructed CT image, the method further includes: returning a correction process of the node parameters of the state space feature extraction module based on the calculated cycle consistent loss between the reconstructed CT image and the original CT image.
[0068] Specifically, the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder; that is, the fused feature data is input into the collaborative task decoder for decoding operation, and the CT image reconstruction processing is performed in the cyclic reconstruction decoder in the collaborative task decoder to obtain a reconstructed CT image; the offset information of the original CT image and the original MRI image is extracted in the deformation estimation decoder in the collaborative task decoder to obtain a tissue deformation field; the blood vessel and tissue segmentation processing of the original MRI image is performed in the tissue segmentation decoder in the collaborative task decoder to obtain an MRI segmentation image.
[0069] At the same time, the node parameters of the state space feature extraction module are corrected according to the calculation cycle consistency loss of the reconstructed CT image and the original CT image.
[0070] The fused feature data is input into the collaborative task decoder, namely, a cyclic reconstruction decoder to reconstruct the CT image. The reconstruction process designs a cycle consistency loss and uses the original CT image to provide a cycle consistency constraint for the reverse reconstruction; another deformation estimation decoder extracts the offset information of the CT and MRI images and outputs the tissue deformation field to avoid the segmentation reconstruction results with inconsistent offsets; there is also a tissue segmentation decoder to perform blood vessel and tissue segmentation on the original MRI image; by sharing the feature encoder, the architecture can simultaneously transform the image appearance and extract domain-invariant features for the segmentation task.
[0071] In order to solve the problem that the existing feature extraction module has limited field of view due to the dimension of three-dimensional images, we designed a feature codec based on VSSMs, built it in the encoder and decoder, and made full use of the characteristics of VSSMs to establish long-distance dependencies and maintain linear complexity to improve the field of view of feature extraction; the encoder and decoder are composed of multiple VSS blocks. After the data is input into VSS, it first passes through the initial linear embedding layer and the normalization layer, and the output is divided into two information streams. One stream passes through a deep convolution layer and then enters the core SS2D module through the Silu activation function; in the SS2D module, the input continuous space (such as the coordinate space of the image) is first divided into a set of discrete grids, and then two-dimensional sampling is performed through multiple levels of convolution and pooling operations, and finally a discretization operation is performed, that is, the sampled values are discretized into a quantitative value space, which is convenient for subsequent processing and calculation. The output of the SS2D module passes through a normalization layer and is then added to the output of another information stream, which undergoes a linear transformation and Silu activation. Finally, the fused features undergo a linear transformation and are fused with the initial state features to obtain the final output. The SS2D module helps map high-dimensional continuous features to low-dimensional discrete representations, facilitating efficient calculations and more accurate feature extraction.
[0072] S14: Perform three-dimensional reconstruction processing based on the tissue deformation field and the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue.
[0073] In the specific implementation process of the present invention, the three-dimensional reconstruction processing is performed based on the tissue deformation field and the MRI segmentation image to obtain a three-dimensional anatomical reconstructed image of the tissue, including: performing three-dimensional reconstruction processing based on the offset information in the tissue deformation field and the blood vessels and tissues in the MRI segmentation image to obtain a three-dimensional anatomical reconstructed image of the tissue.
[0074] Specifically, by performing three-dimensional reconstruction processing on the offset information in the tissue deformation field and the blood vessels and tissues in the MRI segmentation image, a three-dimensional anatomical reconstruction image of the tissue can be obtained.
[0075] In an embodiment of the present invention, a multi-image fusion segmentation method generated by unconditional diffusion sampling satisfies the natural image prior and limits the similarity with the source image through likelihood correction; modeling natural images using a probability diffusion model can better generate details that are difficult to control with a manually designed loss function, thereby producing a visually perceptible image; at the same time, the problem of feature offset caused by the different acquisition times of CT and MRI images and the problem of uneven spatial distribution of brightness in the difference between CT and MRI images are specifically solved.
[0076] For example 2, please refer to Figure 2 , Figure 2It is a schematic diagram of the structural composition of a multi-mode offset image fusion and reconstruction device based on conditional diffusion in an embodiment of the present invention.
[0077] like Figure 2 As shown, a multi-mode offset image fusion reconstruction device based on conditional diffusion, the device comprises:
[0078] Migration generation module 21: used for obtaining the original CT image, and performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migration MRI image;
[0079] In the specific implementation process of the present invention, the migration processing of the original CT image based on unconditional diffusion sampling to generate a migration MRI image includes: gradually adding Gaussian noise to the original CT image, converting the original CT image into a Gaussian signal, and using a denoising score matching method to denoise the Gaussian signal to generate a denoised image; using the probability of the original CT image as a priori condition to migrate the denoised image through likelihood correction optimization to generate a migration MRI image.
[0080] Furthermore, the likelihood correction optimization is to use a maximum expectation algorithm to calculate the similarity between the original CT image and the migrated MRI image; the maximum expectation algorithm includes two steps, E-step and M-step;
[0081] Among them, E-step is the calculation result of the conditional expectation of updating the latent variable x so as to perform the maximization step in M-step, and its calculation formula is as follows:
[0082] ;
[0083] ;
[0084] in, represents the expected value of its random variable; Represents the row index, ranging from 1 to H, where H is a constant; Represents the column index, ranging from 1 to W, where W is a constant; represents the de-drying diffusion step length; represents the observed value of the data point in the i-th row and j-th column in the difference between the migrated MRI image and the original CT image; represents the observed value of the data point in the i-th row and j-th column in the difference between the original CT image and the migrated MRI image; Represents the parameters of a normal distribution, used to describe data points and relationship; is a latent variable, representing the parameters of a normal distribution, used to describe the data points distribution.
[0085] Specifically, we first need to obtain the original CT image , and then generate the new modality initial target domain image, i.e. the initial migration MRI image, through unconditional diffusion sampling .
[0086] Specifically, firstly, given the original CT image Gradually add Gaussian noise to transform the clean sample into an almost Gaussian signal, and then generate .
[0087] Then the original CT image The probability is used as a priori condition to keep more information in the source image through likelihood correction, that is, Generates something closer to of , and finally obtain the migration MRI image; the likelihood correction optimization uses the EM algorithm module to calculate the maximum similarity x between the source image and the migration image to maximize the brightness space of the source image. This optimization problem can be transformed into a maximum likelihood inference problem. The EM algorithm includes two steps, E-step and M-step. The E-step updates the calculation result of the conditional expectation of the latent variable x so as to perform the maximization step in the M-step. The calculation formula is as follows:
[0088] ;
[0089] ;
[0090] in, represents the expected value of its random variable; Represents the row index, ranging from 1 to H, where H is a constant; Represents the column index, ranging from 1 to W, where W is a constant; represents the de-drying diffusion step length; represents the observed value of the data point in the i-th row and j-th column in the difference between the migrated MRI image and the original CT image; represents the observed value of the data point in the i-th row and j-th column in the difference between the original CT image and the migrated MRI image; Represents the parameters of a normal distribution, used to describe data points and relationship; is a latent variable, representing the parameters of a normal distribution, used to describe the data points distribution.
[0091] First, the conditional expectation of the latent variable is calculated based on the current latent variable and the observed data. This involves calculating the Q function, which is the expected log-likelihood function. Based on the calculated conditional expectation, the value of the latent variable x is updated to maximize the Q function, thereby obtaining a better estimate. The E-step is iterated multiple times until the value of the latent variable x converges to the optimal solution. In each iteration, the conditional expectation of the latent variable x is updated, and the latent variable is updated based on the new conditional expectation value.
[0092] After calculating the conditional expectation of the latent variable x in the E-step, the goal of the M-step is to maximize the likelihood function and find the model parameters that maximize the likelihood function, that is, to use the semi-quadratic segmentation algorithm to find the minimum value of x. Depend on After likelihood correction, the first step of M-step is based on The semi-quadratic segmentation algorithm is used to iteratively solve the minimization problem and find the optimal k value to update the parameter k. The second step is to use the semi-quadratic segmentation algorithm to iteratively solve the minimization problem based on k and find the optimal u value. The parameter u can be returned to modify and update the parameter k. The third step is to combine the x, parameter k and ; Use the semi-quadratic partitioning algorithm to iteratively solve the minimization problem to find the optimal Through the iterative process of M-step, the estimated results of model parameters can be gradually optimized, thereby improving the model's fitting ability and prediction accuracy. M-step and E-step are performed alternately until the algorithm converges, and the optimal model parameters and latent variable estimation results are obtained. Estimated , after the final denoising step, the migration MRI image is generated .
[0093] Feature extraction module 22: used for inputting the migration MRI image and the original MRI image into the state space feature extraction module for feature extraction processing to obtain fusion feature data;
[0094] In the specific implementation process of the present invention, the migration MRI image and the original MRI image are input into the state-space feature extraction module for feature extraction processing to obtain fused feature data, including: inputting the migration MRI image and the original MRI image into the state-space feature extraction module for feature extraction processing through N layers of VSS long-distance feature encoding to obtain a first extracted feature; performing feature patch fusion processing on the first extracted feature to obtain a first fused feature; performing feature extraction processing on the first fused feature again through N layers of VSS long-distance feature encoding to obtain a second extracted feature; performing feature patch fusion processing on the second extracted feature to form fused feature data.
[0095] Specifically, after migrating the original CT image to a unified migrated MRI image, image feature fusion is required. Due to the different image acquisition times, there is often a large feature offset between the features of different images, and this is even more true for the digestive tract adjacent to the lungs. In order to ensure the accurate fusion and segmentation of different image features, an unsupervised collaborative image feature adaptive fusion architecture based on conditional diffusion migration is designed. The framework takes the migrated MRI image and the original MRI image migrated to the unified image domain as input, and synchronously extracts features of different images in a shared encoder; after inputting the shared encoder, it first passes through N layers of VSS long-distance feature encoding, and the extracted feature patches are fused, and then pass through N layers of VSS long-distance feature encoding. The features extracted after patch fusion form fused feature data.
[0096] Decoding generation module 23: used for inputting the fused feature data into the collaborative task decoder for decoding operation, generating a reconstructed CT image, a tissue deformation field and an MRI segmentation image, wherein the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder, and the reconstructed CT image is used for calculating the cyclic consistency loss with the original CT image;
[0097] In the specific implementation process of the present invention, the fused feature data is input into the collaborative task decoder for decoding operation to generate a reconstructed CT image, a tissue deformation field and an MRI segmentation image, including: inputting the fused feature data into the collaborative task decoder, and performing CT image reconstruction processing in a cyclic reconstruction decoder in the collaborative task decoder to obtain a reconstructed CT image; performing offset information extraction processing between the original CT image and the original MRI image in the deformation estimation decoder in the collaborative task decoder to obtain a tissue deformation field; performing blood vessel and tissue segmentation processing of the original MRI image in the tissue segmentation decoder in the collaborative task decoder to obtain an MRI segmentation image.
[0098] Furthermore, after obtaining the reconstructed CT image, the method further includes: returning a correction process of the node parameters of the state space feature extraction module based on the calculated cycle consistent loss between the reconstructed CT image and the original CT image.
[0099] Specifically, the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder; that is, the fused feature data is input into the collaborative task decoder for decoding operation, and the CT image reconstruction processing is performed in the cyclic reconstruction decoder in the collaborative task decoder to obtain a reconstructed CT image; the offset information of the original CT image and the original MRI image is extracted in the deformation estimation decoder in the collaborative task decoder to obtain a tissue deformation field; the blood vessel and tissue segmentation processing of the original MRI image is performed in the tissue segmentation decoder in the collaborative task decoder to obtain an MRI segmentation image.
[0100] At the same time, the node parameters of the state space feature extraction module are corrected according to the calculation cycle consistency loss of the reconstructed CT image and the original CT image.
[0101] The fused feature data is input into the collaborative task decoder, namely, a cyclic reconstruction decoder to reconstruct the CT image. The reconstruction process designs a cycle consistency loss and uses the original CT image to provide a cycle consistency constraint for the reverse reconstruction; another deformation estimation decoder extracts the offset information of the CT and MRI images and outputs the tissue deformation field to avoid the segmentation reconstruction results with inconsistent offsets; there is also a tissue segmentation decoder to perform blood vessel and tissue segmentation on the original MRI image; by sharing the feature encoder, the architecture can simultaneously transform the image appearance and extract domain-invariant features for the segmentation task.
[0102] In order to solve the problem that the existing feature extraction module has limited field of view due to the dimension of three-dimensional images, we designed a feature codec based on VSSMs, built it in the encoder and decoder, and made full use of the characteristics of VSSMs to establish long-distance dependencies and maintain linear complexity to improve the field of view of feature extraction; the encoder and decoder are composed of multiple VSS blocks. After the data is input into VSS, it first passes through the initial linear embedding layer and the normalization layer, and the output is divided into two information streams. One stream passes through a deep convolution layer and then enters the core SS2D module through the Silu activation function; in the SS2D module, the input continuous space (such as the coordinate space of the image) is first divided into a set of discrete grids, and then two-dimensional sampling is performed through multiple levels of convolution and pooling operations, and finally a discretization operation is performed, that is, the sampled values are discretized into a quantitative value space, which is convenient for subsequent processing and calculation. The output of the SS2D module passes through a normalization layer and is then added to the output of another information stream, which undergoes a linear transformation and Silu activation. Finally, the fused features undergo a linear transformation and are fused with the initial state features to obtain the final output. The SS2D module helps map high-dimensional continuous features to low-dimensional discrete representations, facilitating efficient calculations and more accurate feature extraction.
[0103] The three-dimensional reconstruction module 24 is used to perform three-dimensional reconstruction processing based on the tissue deformation field and the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue.
[0104] In the specific implementation process of the present invention, the three-dimensional reconstruction processing is performed based on the tissue deformation field and the MRI segmentation image to obtain a three-dimensional anatomical reconstructed image of the tissue, including: performing three-dimensional reconstruction processing based on the offset information in the tissue deformation field and the blood vessels and tissues in the MRI segmentation image to obtain a three-dimensional anatomical reconstructed image of the tissue.
[0105] Specifically, by performing three-dimensional reconstruction processing on the offset information in the tissue deformation field and the blood vessels and tissues in the MRI segmentation image, a three-dimensional anatomical reconstruction image of the tissue can be obtained.
[0106] In an embodiment of the present invention, a multi-image fusion segmentation method generated by unconditional diffusion sampling satisfies the natural image prior and limits the similarity with the source image through likelihood correction; modeling natural images using a probability diffusion model can better generate details that are difficult to control with a manually designed loss function, thereby producing a visually perceptible image; at the same time, the problem of feature offset caused by the different acquisition times of CT and MRI images and the problem of uneven spatial distribution of brightness in the difference between CT and MRI images are specifically solved.
[0107] An embodiment of the present invention provides a computer-readable storage medium, on which an application is stored, and when the program is executed by a processor, a multi-mode offset image fusion reconstruction method of any one of the above embodiments is implemented. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (EraSable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that can store or transmit information in a readable form by a device (for example, a computer, a mobile phone), which can be a read-only memory, a disk or an optical disk, etc.
[0108] An embodiment of the present invention further provides a computer application program, which runs on a computer and is used to execute the multi-mode offset image fusion reconstruction method of any one of the above embodiments.
[0109] also, Figure 3 It is a schematic diagram of the structure of the cloud server in an embodiment of the present invention.
[0110] The embodiment of the present invention further provides an electronic device, such as Figure 3 The electronic device includes a processor 302, a memory 303, an input unit 304, a display unit 305 and other devices. Those skilled in the art will understand that Figure 3 The device structure device shown does not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 303 can be used to store the application 301 and various functional modules, and the processor 302 runs the application 301 stored in the memory 303, thereby executing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both internal and external memories. The internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory may include a hard disk, a floppy disk, a ZIP disk, a U disk, a magnetic tape, etc. The memory disclosed in the present invention includes but is not limited to these types of memories. The memory disclosed in the present invention is only used as an example and not as a limitation.
[0111] The input unit 304 is used to receive the input of signals and keywords input by the user. The input unit 304 may include a touch panel and other input devices. The touch panel can collect the user's touch operations on or near it (such as the user's operation on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program; other input devices may include but are not limited to one or more of a physical keyboard, a function key (such as a play control key, a switch key, etc.), a trackball, a mouse, a joystick, etc. The display unit 305 can be used to display the information input by the user or the information provided to the user and various menus of the terminal device. The display unit 305 can be in the form of a liquid crystal display, an organic light emitting diode, etc. The processor 302 is the control center of the terminal device, which uses various interfaces and lines to connect various parts of the entire device, and executes various functions and processes data by running or executing software programs and / or modules stored in the memory 303, and calling data stored in the memory.
[0112] As an embodiment, the electronic device includes: one or more processors 302, a memory 303, and one or more applications 301, wherein the one or more applications 301 are stored in the memory 303 and are configured to be executed by the one or more processors 302, and the one or more applications 301 are configured to execute the multi-mode offset image fusion reconstruction method in any one of the above-mentioned embodiments.
[0113] In an embodiment of the present invention, a multi-image fusion segmentation method generated by unconditional diffusion sampling satisfies the natural image prior and limits the similarity with the source image through likelihood correction; modeling natural images using a probability diffusion model can better generate details that are difficult to control with a manually designed loss function, thereby producing a visually perceptible image; at the same time, the problem of feature offset caused by the different acquisition times of CT and MRI images and the problem of uneven spatial distribution of brightness in the difference between CT and MRI images are specifically solved.
[0114] In addition, the above is a detailed introduction to a multi-mode offset image fusion reconstruction method based on conditional diffusion and related devices provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A multi-mode offset image fusion reconstruction method based on conditional diffusion, characterized in that: The method comprises: Obtaining an original CT image, and performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migrated MRI image; Inputting the migration MRI image and the original MRI image into a state space feature extraction module for feature extraction processing to obtain fusion feature data; Inputting the fused feature data into a collaborative task decoder for decoding operation to generate a reconstructed CT image, a tissue deformation field and an MRI segmentation image, wherein the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder, and the reconstructed CT image is used to calculate the cyclic consistency loss with the original CT image; Performing three-dimensional reconstruction processing based on the tissue deformation field and the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue; The step of performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migrated MRI image includes: gradually adding Gaussian noise to the original CT image, converting the original CT image into a Gaussian signal, and denoising the Gaussian signal using a denoising fraction matching method to generate a denoised image; The denoised image is migrated by likelihood correction optimization using the probability of the original CT image as a priori condition to generate a migrated MRI image; The likelihood correction optimization is to use the maximum expectation algorithm to calculate the similarity between the original CT image and the migrated MRI image; The step of inputting the migration MRI image and the original MRI image into the state space feature extraction module for feature extraction processing to obtain fusion feature data includes: Inputting the migration MRI image and the original MRI image into the state space feature extraction module and performing feature extraction processing through N layers of VSS long-distance feature encoding to obtain a first extracted feature; Performing feature patch fusion processing on the first extracted features to obtain first fused features; The first fused feature is further subjected to feature extraction processing through N layers of VSS long-range feature encoding to obtain a second extracted feature; The second extracted features are subjected to feature patch fusion processing to form fused feature data.
2. The multi-mode offset image fusion reconstruction method according to claim 1, characterized in that: The maximum expectation algorithm includes two steps: E-step and M-step; Among them, E-step is the calculation result of the conditional expectation of updating the latent variable x so as to perform the maximization step in M-step, and its calculation formula is as follows: Where E represents the expected value of its random variable; i represents the row index, ranging from 1 to H, H is a constant; j represents the column index, ranging from 1 to W, W is a constant; t represents the denoising diffusion step size; x ij represents the observed value of the data point in the i-th row and j-th column in the difference between the migrated MRI image and the original CT image; ij represents the observed value of the data point in the i-th row and j-th column in the difference between the original CT image and the migrated MRI image; m ij Represents the parameters of a normal distribution, used to describe the data point y ij and x ij relationship; ij is a latent variable, representing the parameters of a normal distribution, used to describe the data point x ij distribution.
3. The multi-mode offset image fusion reconstruction method according to claim 1, characterized in that: The step of inputting the fused feature data into the collaborative task decoder for decoding to generate a reconstructed CT image, a tissue deformation field and an MRI segmentation image comprises: Inputting the fused feature data into the collaborative task decoder, and performing CT image reconstruction processing in a cyclic reconstruction decoder in the collaborative task decoder to obtain a reconstructed CT image; Extracting and processing the offset information of the original CT image and the original MRI image in the deformation estimation decoder in the collaborative task decoder to obtain a tissue deformation field; The blood vessel and tissue segmentation processing of the original MRI image is performed in the tissue segmentation decoder in the collaborative task decoder to obtain an MRI segmentation image.
4. The multi-mode offset image fusion reconstruction method according to claim 3, characterized in that: After obtaining the reconstructed CT image, the method further includes: The node parameters of the state space feature extraction module are corrected based on the calculated cycle consistent loss of the reconstructed CT image and the original CT image.
5. The multi-mode offset image fusion reconstruction method according to claim 1, characterized in that: The three-dimensional reconstruction process is performed based on the tissue deformation field and the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue, including: A three-dimensional reconstruction process is performed based on the offset information in the tissue deformation field and the blood vessels and tissues in the MRI segmented image to obtain a three-dimensional anatomical reconstruction image of the tissue.
6. A multi-mode offset image fusion reconstruction device based on conditional diffusion, characterized in that: The device comprises: Migration generation module: used for obtaining the original CT image, and performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migration MRI image; Feature extraction module: used for inputting the migration MRI image and the original MRI image into the state space feature extraction module for feature extraction processing to obtain fusion feature data; Decoding generation module: used for inputting the fused feature data into the collaborative task decoder for decoding operation, generating reconstructed CT images, tissue deformation fields and MRI segmentation images, wherein the collaborative task decoder includes a cyclic reconstruction decoder, a deformation estimation decoder and a tissue segmentation decoder, and the reconstructed CT image is used for calculating the cyclic consistency loss with the original CT image; A three-dimensional reconstruction module: used for performing three-dimensional reconstruction processing based on the tissue deformation field and the MRI segmentation image to obtain a three-dimensional anatomical reconstruction image of the tissue; The step of performing migration processing on the original CT image based on unconditional diffusion sampling to generate a migrated MRI image includes: gradually adding Gaussian noise to the original CT image, converting the original CT image into a Gaussian signal, and denoising the Gaussian signal using a denoising fraction matching method to generate a denoised image; The denoised image is migrated by likelihood correction optimization using the probability of the original CT image as a priori condition to generate a migrated MRI image; The likelihood correction optimization is to use the maximum expectation algorithm to calculate the similarity between the original CT image and the migrated MRI image; The step of inputting the migration MRI image and the original MRI image into the state space feature extraction module for feature extraction processing to obtain fusion feature data includes: Inputting the migration MRI image and the original MRI image into the state space feature extraction module and performing feature extraction processing through N layers of VSS long-distance feature encoding to obtain a first extracted feature; Performing feature patch fusion processing on the first extracted features to obtain first fused features; The first fused feature is further subjected to feature extraction processing through N layers of VSS long-range feature encoding to obtain a second extracted feature; The second extracted features are subjected to feature patch fusion processing to form fused feature data.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
