Three-dimensional CT image synthesis method and system based on consistency diffusion model
Through the three-dimensional CT image synthesis method based on the consistency diffusion model, the pre-trained medical report encoding model and variational autoencoder are used to generate high-fidelity 3-dimensional CT images, which solves the problem of insufficient data volume and improves the accuracy of the model and Medical imaging research ability.
Patent Information
- Application Number
- CN202411182646.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, the data amount of three-dimensional CT imaging and medical report paired data is insufficient, which limits the accuracy of the model and the analytical ability of medical experts.
Using a three-dimensional CT image synthesis method based on a consistent diffusion model, a model including a pre-trained medical report coding model, a consistent diffusion model and a variational autoencoder are built by obtaining the data set paired by a three-dimensional CT image and a medical report. The preprocessed data set is trained to generate high-fidelity three-dimensional CT images.
The problem of insufficient pairing data of three-dimensional CT imaging and medical reports is solved, the accuracy of the model and medical imaging research capabilities are improved, and the use of medical reports is allowed to generate three-dimensional CT images of designated cases.
Smart Images

Figure CN118967944A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of three-dimensional CT image generation, and in particular relates to a three-dimensional CT image synthesis method and system based on a consistent diffusion model. Background Art
[0002] Diffusion models have made great progress in the field of image and video generation, providing great convenience for designers. Currently, there are many challenges in the field of computational medicine. The lack of 3D CT images and paired medical reports specifically for training limits the accuracy of the model and also reduces the analysis capabilities of medical experts.
[0003] Diffusion models are famous for their high quality, diversity, and training stability in generated images. The emergence of diffusion models makes it possible to generate 3D CT images using medical reports. Medical reports usually include key medical information such as detailed pathological information of patients and location of lesions. Using this information as conditional input can provide sufficient generation conditions for diffusion models. Currently, researchers have begun to apply diffusion models to the field of medical image segmentation, but few have applied diffusion models to the field of medical image generation. Generating high-fidelity 3D CT images through diffusion models can solve the current problem of insufficient 3D CT image data. Training the model based on the generated 3D CT images can help improve the accuracy of the model. Summary of the invention
[0004] The purpose of the present invention is to solve the problem of insufficient data volume of three-dimensional CT images and medical report pairing data that can be used for model training, and to propose a three-dimensional CT image synthesis method and system based on a consistent diffusion model.
[0005] The technical solution adopted by the present invention to solve the above technical problems is:
[0006] According to one aspect of the present invention, a three-dimensional CT image synthesis method based on a consistent diffusion model comprises the following steps:
[0007] Step S1, obtaining a data set of three-dimensional CT images and medical reports, and preprocessing the medical reports in the obtained data set to obtain a preprocessed three-dimensional CT image and medical report paired data set;
[0008] Step S2, building a model including a pre-trained medical report encoding model, a consistency diffusion model, and a variational autoencoder;
[0009] Step S3, training the constructed model using the preprocessed three-dimensional CT image and medical report paired data set;
[0010] Step S4, obtaining a medical report for synthesizing a three-dimensional CT image, preprocessing the obtained medical report, and then using a word segmenter to output a vector form corresponding to the preprocessed medical report;
[0011] The output of the word segmenter is used as the input of the pre-trained medical report encoding model to obtain the text vector of the medical report; then an initial Gaussian noise vector is randomly sampled, and the sampled Gaussian noise vector and the text vector of the medical report are input into the consistency diffusion model for the reverse denoising process, and the output of the reverse denoising process is used as the input of the decoding part of the variational autoencoder, and the synthesized three-dimensional CT image is output through the decoding part of the VAE.
[0012] According to another aspect of the present invention, a three-dimensional CT image synthesis system based on a consistent diffusion model includes a data set acquisition module, a data set preprocessing module and a neural network model;
[0013] A data set acquisition module, used to acquire a data set of paired three-dimensional CT images and medical reports;
[0014] A data set preprocessing module is used to preprocess the medical reports in the acquired data set to obtain a preprocessed three-dimensional CT image and medical report paired data set;
[0015] The neural network model includes a pre-trained medical report encoding model, a consistency diffusion model, and a variational autoencoder;
[0016] And the neural network model is trained using the preprocessed data set, and the training process is:
[0017] Step S21: Processing the medical report
[0018] Use a tokenizer to mark the beginning of the medical report with a [CLS] tag, and mark adjacent sentences with a [SEP] tag as a sentence separator. Then, tokenize each sentence, that is, divide each sentence into several words. Finally, map the processed sentences to integer indices and output them in the form of vectors.
[0019] Step S22: using the processed vector corresponding to the medical report as the input of the pre-trained medical report coding model, and performing feature extraction through the pre-trained medical report coding model to output the text vector corresponding to the medical report; specifically:
[0020] Project the vectors output by the tokenizer into the subspaces of each attention head:
[0021]
[0022] Among them, X is the sentence vector converted by the word segmenter, Q iis the Query vector of the subspace of the i-th attention head, K i is the Key vector of the subspace of the i-th attention head, V i is the Value vector of the subspace of the i-th attention head, and is the weight matrix;
[0023] According to Q i , K i and V i Calculate the attention score of the i-th attention head i :
[0024]
[0025] The superscript T represents transposition, d k is the characteristic dimension;
[0026] Concatenate the attention scores of all attention heads and map the concatenated vectors:
[0027] Z text =Concat(head1,head2,...,head h )W o
[0028] Among them, Z text is the text vector corresponding to the medical report, W o is the projection matrix;
[0029] Step S23, the three-dimensional CT image is subjected to the encoding part of the variational autoencoder to obtain a vector representation of the three-dimensional CT image; the vector representation of the three-dimensional CT image is then used as the input of the forward denoising process of the consistent diffusion model, and the three-dimensional CT image is subjected to noise processing through the forward denoising process of the consistent diffusion model to convert the three-dimensional CT image into a noise vector;
[0030] Step S24, using the text vector obtained in step S22 and the noise vector obtained in step S23 as inputs of the reverse denoising process of the consistency diffusion model, and obtaining a noise-free image vector x′0 through the reverse denoising process;
[0031] The specific process of step S24 is:
[0032] Step S241, initializing the number of denoising modules t=T;
[0033] Step S242: Use the text vector obtained in step S22 and the noise vector obtained in step S23 as the input of the t-th denoising module to obtain the output x′ of the t-th denoising module. t-1 ;
[0034] Step S243, determine whether t=1 is satisfied;
[0035] If satisfied, then end;
[0036] If not satisfied, continue to step S244;
[0037] Step S244: The output x′ of the t-th denoising module t-1 The text vector obtained in step S22 is used as the input of the t-1 denoising module to obtain the output x′ of the t-1 denoising module. t-2 , then let t = t-1;
[0038] Then return to step S243;
[0039] Step S25, calculating the loss function value according to the inverse denoising result x′0 and the three-dimensional CT image corresponding to the medical report, and stopping the training until the maximum epoch is reached or the maximum tolerance is reached, thereby obtaining a trained model;
[0040] The trained neural network model is used to synthesize 3D CT images. The specific process is as follows:
[0041] Obtain a medical report for synthesizing a three-dimensional CT image, pre-process the obtained medical report, and then use a word segmenter to output a vector form corresponding to the pre-processed medical report;
[0042] The output of the tokenizer is used as the input of the pre-trained medical report encoding model to obtain the text vector of the medical report;
[0043] Then randomly sample an initial Gaussian noise vector, input the sampled Gaussian noise vector and the text vector of the medical report into the consistency diffusion model for the reverse denoising process, use the output of the reverse denoising process as the input of the decoding part of the variational autoencoder, and output the synthesized three-dimensional CT image through the decoding part of the VAE.
[0044] The beneficial effects of the present invention are:
[0045] The model of the present invention includes a pre-trained medical report encoding model, a consistency diffusion model and a variational autoencoder, and the combined model is trained. The trained model and the public medical report can be used to generate a three-dimensional CT image corresponding to the medical report, which can solve the problem of insufficient paired data between public three-dimensional CT images and medical reports. At the same time, it allows the use of medical reports to generate three-dimensional CT images of specified cases, thereby enhancing the research capabilities of computational imaging in the field of three-dimensional CT images. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1It is a framework diagram of a three-dimensional CT image synthesis method based on a consistent diffusion model of the present invention;
[0047] Figure 2 It is a flow chart of a three-dimensional CT image synthesis method based on a consistent diffusion model of the present invention;
[0048] Figure 3 It is a flowchart of joint training and prediction based on pre-trained medical report encoding model, consistency diffusion model and variational autoencoder;
[0049] Among them, the process in the green box represents the training structure diagram of the model, and the process in the red box represents the prediction structure diagram of the model;
[0050] Figure 4 It is the structural diagram of the denoising module;
[0051] Figure 5 This is the block diagram of the Cross-Slice Control unit;
[0052] Figure 6 This is the result of three-dimensional CT image synthesis. DETAILED DESCRIPTION
[0053] Specific implementation method 1: Combination Figure 1 and Figure 2 The present embodiment describes a three-dimensional CT image synthesis method based on a consistent diffusion model, and the method specifically includes the following steps:
[0054] Step S1, obtaining a data set of three-dimensional CT images and medical reports, and preprocessing the medical reports in the obtained data set to obtain a preprocessed three-dimensional CT image and medical report paired data set;
[0055] Step S2, building a model including a pre-trained medical report encoding model, a consistency diffusion model and a variational autoencoder (VAE);
[0056] Step S3, training the constructed model using the preprocessed three-dimensional CT image and medical report paired data set;
[0057] Step S4: The prediction process of the model is as follows: Figure 2 As shown. Obtain a medical report for synthesizing a three-dimensional CT image, pre-process the obtained medical report, and then use a word segmenter to output a vector form corresponding to the pre-processed medical report;
[0058] The output of the tokenizer is used as the input of the pre-trained medical report encoding model to obtain the text vector of the medical report;
[0059] Then randomly sample an initial Gaussian noise vector, input the sampled Gaussian noise vector and the text vector of the medical report into the consistency diffusion model for the reverse denoising process, use the output of the reverse denoising process as the input of the decoding part of the variational autoencoder, and output the synthesized 3D CT image through the decoding part of the VAE, as shown in Figure 6 shown.
[0060] The variational autoencoder of the present invention is an unsupervised learning model, which is mainly used for feature learning and dimensionality reduction. It consists of two parts: an encoder and a decoder. The encoder maps the input data to a low-dimensional latent space, and the decoder remaps the latent space representation back to the original data space. The encoder can map the input data x to a latent vector space z, in which the data is represented as a Gaussian distribution parameter. The decoder can map the latent vector space z back to the original data space to generate a sample x' similar to the input data x.
[0061] Specific implementation method 2: This implementation method is a further limitation of specific implementation method 1, and the medical reports in the acquired data set are preprocessed, specifically: the medical reports are translated into English.
[0062] In the present invention, after downloading the existing data set, three-dimensional CT images with good imaging quality and medical report pairing data are manually selected. The translation tools that can be used include but are not limited to GPT-4.
[0063] Specific implementation method three: Combination Figure 3 This embodiment is a further limitation of the second embodiment, wherein the model is trained using a data set of pre-processed three-dimensional CT images paired with medical reports, and the specific training process is as follows:
[0064] Step S21: Processing the medical report
[0065] Use a tokenizer to mark the beginning of the medical report with a [CLS] tag, and mark adjacent sentences with a [SEP] tag as a sentence separator. Then, tokenize each sentence, that is, divide each sentence into several words. Finally, map the processed sentences to integer indices and output them in the form of vectors.
[0066] Step S22: using the processed vector corresponding to the medical report as the input of the pre-trained medical report encoding model, and performing feature extraction through the pre-trained medical report encoding model to output a text vector corresponding to the medical report;
[0067] Step S23, the three-dimensional CT image is subjected to the encoding part of the variational autoencoder to obtain a vector representation of the three-dimensional CT image; the vector representation of the three-dimensional CT image is then used as the input of the forward denoising process of the consistent diffusion model, and the three-dimensional CT image is subjected to noise processing through the forward denoising process of the consistent diffusion model to convert the three-dimensional CT image into a noise vector;
[0068] The forward noise adding process is:
[0069]
[0070] Among them, β t represents the noise increment added at the tth time step, β t Between (0,1), ∈ is an independent Gaussian noise sampled from a standard normal distribution, x t-1 represents the noisy image obtained at the t-1th time step, x t represents the noisy image obtained at the t-th time step, t = 1, 2, ..., T, T is the total number of time steps of noise addition; by continuously repeating the noise addition process, the input 3D CT image is transformed into a Gaussian noise that obeys the normal distribution;
[0071] Step S24, using the text vector obtained in step S22 and the noise vector obtained in step S23 as inputs of the reverse denoising process of the consistency diffusion model, and obtaining a noise-free image vector x′0 through the reverse denoising process;
[0072] The specific process of step S24 is:
[0073] Step S241, initializing the number of denoising modules t=T;
[0074] Step S242: Use the text vector obtained in step S22 and the noise vector obtained in step S23 as the input of the t-th denoising module to obtain the output x′ of the t-th denoising module. t-1 ;
[0075] Step S243, determine whether t=1 is satisfied;
[0076] If satisfied, then end;
[0077] If not satisfied, continue to step S244;
[0078] Step S244: The output x′ of the t-th denoising module t-1 The text vector obtained in step S22 is used as the input of the t-1 denoising module to obtain the output x′ of the t-1 denoising module. t-2 , then let t = t-1;
[0079] Then return to step S243;
[0080] Step S25, calculate the loss function value according to the reverse denoising result x′0 and the three-dimensional CT image corresponding to the medical report, adjust the model parameters according to the loss function value by backpropagation, and stop training when the maximum epoch is reached or the maximum tolerance is reached to obtain a trained model.
[0081] The process from step S21 to step S24 is performed for each pair of data in the training set until the maximum epoch is reached or the maximum tolerance is reached, and the training is stopped.
[0082] The hyperparameters set during training are: the number of training rounds is set to 1000, the tolerance is set to 50 (when the indicator no longer improves after 50 consecutive rounds of verification on the verification set, the training is stopped), the training batch size is set to 32, and the learning rate is 0.00005.
[0083] The consistency diffusion model is divided into a forward process and a reverse process. Forward process: Through the formula Gradually add noise to the image, where β t represents the noise intensity at time t, ∈ is an independent Gaussian noise sampled from a standard normal distribution, x t-1 Represents the image at the previous moment. By repeating the noise adding process, the input 3D CT image is transformed into a Gaussian noise that obeys the normal distribution. Reverse process: Use the medical report encoder to extract the features of the input text to form a high-dimensional feature vector, and input the text feature vector and the noise image into the denoising module of the diffusion model. After t iterations of denoising, the model obtains the predicted image. The mean square error between the predicted image and the true image is calculated through the loss function and the model is optimized through back propagation.
[0084] Specific implementation method 4: This implementation method is a further limitation of specific implementation method 3, wherein the vector corresponding to the processed medical report is used as the input of the pre-trained medical report encoding model, and the pre-trained medical report encoding model is used to perform feature extraction to output the text vector corresponding to the medical report; specifically:
[0085] Project the vectors output by the tokenizer into the subspaces of each attention head:
[0086]
[0087] Among them, X is the sentence vector converted by the word segmenter, Q i is the Query vector of the subspace of the i-th attention head, K i is the Key vector of the subspace of the i-th attention head, V i is the Value vector of the subspace of the i-th attention head, and are different weight matrices;
[0088] According to Q i , K i and V i Calculate the attention score of the i-th attention head i :
[0089]
[0090] The superscript T represents transposition, d k is the characteristic dimension;
[0091] Concatenate the attention scores of all attention heads and map the concatenated vector to the output dimension of the attention head:
[0092] Z text =Concat(head1,head2,...,head h )W o
[0093] Among them, Z text is the text vector corresponding to the medical report, W o is a trainable projection matrix.
[0094] The encoder structure adopted by the pre-trained medical report encoding model of the present invention is a Transformer structure, and a multi-head attention mechanism is used.
[0095] Specific implementation method five: Combination Figure 4 This embodiment is a further limitation of the fourth embodiment, wherein the denoising module includes a first downsampling unit to an Nth downsampling unit, a Mid Block, and a first upsampling unit to an Nth upsampling unit; wherein:
[0096] The input of the denoising module first passes through the first downsampling unit to the Nth downsampling unit in sequence, and then the output of the Nth downsampling unit is used as the input of the MidBlock, and the output of the MidBlock then passes through the first upsampling unit to the Nth upsampling unit in sequence;
[0097] The output of the first down-sampling unit is used as the input of the Nth up-sampling unit through a skip connection (another part of the input of the Nth up-sampling unit comes from the output of the N-1th up-sampling unit), the output of the second down-sampling unit is used as the input of the N-1th up-sampling unit through a skip connection (another part of the input of the N-1th up-sampling unit comes from the output of the N-2th up-sampling unit), the output of the third down-sampling unit is used as the input of the N-2th up-sampling unit through a skip connection (another part of the input of the N-2th up-sampling unit comes from the output of the N-3th up-sampling unit), and so on, the output of the N-1th down-sampling unit is used as the input of the 2nd up-sampling unit through a skip connection (another part of the input of the 2nd up-sampling unit comes from the output of the 1st up-sampling unit);
[0098] The first downsampling unit includes a 2D CNN subunit and a volume perception subunit, and the structures of the second downsampling unit to the Nth downsampling unit and the first upsampling unit to the Nth upsampling unit are the same as the structure of the first downsampling unit;
[0099] Mid Block includes four sub-modules, and the structures of the second sub-module, the third sub-module and the fourth sub-module are the same as the structure of the first sub-module;
[0100] The first submodule includes a 2D CNN subunit, a volume perception subunit, a Cross-Slice Control subunit, and a Cross Attention subunit.
[0101] During the downsampling process, the spatial dimension will gradually decrease, and the high-resolution information in the feature map will gradually be lost to obtain deeper semantic information. Before the image vector is input into the first downsampling unit, an initial convolution is performed to change the number of channels to 32, which lays the foundation for subsequent downsampling and deeper feature extraction. Assume that the input image vector is x∈R b×c×f×h×w , where b represents the batch number, c represents the number of input image channels, f represents the number of input image slices, and h and w represent the input image height and width respectively. After an initial convolution, the image vector dimension becomes x∈R b×32×f×h×w , and then input to the downsampling unit. In the downsampling unit, after passing through a 2D CNN, the height and width of the image vector dimension are converted to the original The number of channels becomes twice as much as before, which can be expressed as Then it is processed by the volume perception subunit, and the processing result is input to the next downsampling unit. In the upsampling unit, the image vector after compression and feature extraction by the downsampling unit is restored to its original dimension. Each time it passes through an upsampling unit, the h and w of the input vector are doubled, and the number of channels is reduced to the original Until the dimension of the image vector becomes x∈R b×32×f×h×w Then, after another convolution, the image vector dimension is restored to x∈R b×c×f×h×w .
[0102] Moreover, the features of the downsampling stage are directly passed to the upsampling stage through jump connections, so that feature information at the same level can interact with each other, helping the model to generate more delicate and accurate images in the final reconstruction stage.
[0103] Specific implementation method six: Combination Figure 4 and Figure 5 This embodiment is a further limitation of the fifth embodiment, and the working process of the Mid Block is as follows:
[0104] Step 1: Use the output of the Nth downsampling unit as the input of the Mid Block, that is, use the output of the Nth downsampling unit as the input of the 2D CNN subunit in the first submodule;
[0105] The working process of the 2D CNN subunit is:
[0106] The 3D CT image input by the 2D CNN subunit is represented as a 5-dimensional tensor x∈R b×c×f×h×w , where b represents the batch number, c represents the channel number, f represents the slice number, h represents the height of the 3D CT image, and w represents the width of the 3D CT image; the number of slices f of the tensor is merged into the batch dimension b, and the merged tensor dimension is x∈R (b×f)×c×h×w ;
[0107] The 2D CNN subunit is used to process the tensor after the dimension is merged to extract the features of each slice. After the slice feature extraction is completed, the tensor format is resized to x∈R b×c×f×h×w ;
[0108] Step 2: Use the output of the 2D CNN subunit in the first submodule as the input of the volume perception subunit in the first submodule, where the volume perception subunit includes M Cross-Frame Attention blocks.
[0109] The working process of the volume perception subunit is:
[0110] Step 21: Use sinusoidal position coding to encode the position of each slice in the three-dimensional CT image. The sinusoidal position coding volume perception module can learn the relative relationship between each slice in the three-dimensional CT, such as the position relationship between slices;
[0111] Step 22, initialize the count m of the Cross-Frame Attention block to 1;
[0112] Step 23: Merge the spatial dimensions of the feature tensors output by the 2D CNN subunits into the batch dimension to obtain a feature tensor Z∈R consisting of the feature tensors of each slice after merging the dimensions. (b×h×w)×c×f , where the feature tensor of the jth slice can be expressed as z j ∈R (b×h×w)×c ;
[0113] Step 24: Use the dimension-merged feature tensor Z as the input of the mth Cross-Frame Attention block;
[0114] Step 25. Multiply the input feature tensor by the weight matrix W in the Cross-Frame Attention block Q Generate the Q vector by multiplying the input feature tensor by the weight matrix W K Generate K vectors, multiply the input features by the weight matrix W V Generate V vector. The dimensions of the generated Q, K, and V vectors are all (b×h×w)×f×d k , where d k is the characteristic dimension;
[0115] Then perform weighted summation based on the Q, K and V vectors to obtain the output feature tensor Z out :
[0116]
[0117] Step 26: Determine whether the count m of the Cross-Frame Attention block is less than M;
[0118] If m is less than M, then set m=m+1 and change Z out As the input of the next Cross-Frame Attention block, return to step 25;
[0119] If m is equal to M, the final weighted summation feature tensor Z is obtained out ;
[0120] Step 3: The feature tensor Z output by the volume perception subunit of the first submodule out The position encoding result is used as the input of the Cross-Slice Control subunit of the first submodule, and the vector representation of each slice is output through the Cross-Slice Control subunit;
[0121] The working process of the Cross-Slice Control subunit is:
[0122] For the jth slice, the feature tensor Zout The feature of the j-th slice in the self-attention mechanism obtains the Q corresponding to the j-th slice j , K j and V j , the feature tensor Z out The feature of the j-1th slice in the self-attention mechanism obtains the Q corresponding to the j-1th slice j-1 , K j-1 and V j-1 ;
[0123] Then the vector of the jth slice represents x j for:
[0124]
[0125] In the self-attention mechanism, K and V vectors usually store semantic information, and Q vectors usually store spatial information. The present invention improves the consistency of generated semantics by retaining the K and V vectors corresponding to the previous slice, and uses the Q vector corresponding to the current slice to retain the spatial information of each slice.
[0126] Step 4: Use the text vector and the vector representation of each slice output by the Cross-Slice Control subunit of the first submodule as the input of the Cross Attention subunit of the first submodule, and use the output of the Cross Attention subunit as the output of the first submodule;
[0127] The working process of the Cross Attention subunit is:
[0128] Perform a linear transformation on the input text vector and convert the input text vector into K t 、V t Vector, linearly transform the output of the Cross-Slice Control subunit, and convert the output of the Cross-Slice Control subunit into the Q of the image through linear transformation ct Vector, then
[0129]
[0130] Among them, Z ct Represents the image feature vector that incorporates text semantic information;
[0131] Step 5: Use the output of the first submodule and the text vector as the input of the second submodule, use the output of the second submodule and the text vector as the input of the third submodule, use the output of the third submodule and the text vector as the input of the fourth submodule, and use the output of the fourth submodule as the output of Mid Block.
[0132] Specific implementation method 7: This implementation method is a further limitation of specific implementation method 6, and the loss function is:
[0133]
[0134] Among them, x0 represents the real input image, f text represents the text vector corresponding to the input medical report, θ is the parameter of the consistency diffusion model, x t represents the output of the forward noise addition process, G(x t ,f text ; θ) represents the output of the inverse denoising process, q(x0,f text ) represents x0 and f text This part introduces the conditional generated text feature information, allowing the model to refer to these features during the generation process. t |x0) means that given x0, x is generated. t This part introduces the noise processing process, which enables the model to gradually restore the original data during the denoising process. L is the loss function, Represents the joint distribution q(x0,f text ) and the conditional distribution q(x t |x0). By performing expected calculations on these distributions, all possible samples can be considered during training, thereby improving the generalization ability of the model.
[0135] Indicates that given the real data x0 and the text vector f text The goal is to minimize the mean square error between the generated image and the real image. The model parameters are updated by calculating the loss function. This step uses the gradient descent optimization algorithm to update the model parameters layer by layer through back propagation so that the loss function value is reduced.
[0136] Specific implementation eight: A three-dimensional CT image synthesis system based on a consistent diffusion model described in this implementation includes a data set acquisition module, a data set preprocessing module and a neural network model;
[0137] A data set acquisition module, used to acquire a data set of paired three-dimensional CT images and medical reports;
[0138] A data set preprocessing module is used to preprocess the medical reports in the acquired data set to obtain a preprocessed three-dimensional CT image and medical report paired data set;
[0139] The neural network model includes a pre-trained medical report encoding model, a consistency diffusion model, and a variational autoencoder (VAE);
[0140] And the neural network model is trained using the preprocessed data set, and the training process is:
[0141] Step S21: Processing the medical report
[0142] Use a tokenizer to mark the beginning of the medical report with a [CLS] tag, and mark adjacent sentences with a [SEP] tag as a sentence separator. Then, tokenize each sentence, that is, divide each sentence into several words. Finally, map the processed sentences to integer indices and output them in the form of vectors.
[0143] Step S22: using the processed vector corresponding to the medical report as the input of the pre-trained medical report coding model, and performing feature extraction through the pre-trained medical report coding model to output the text vector corresponding to the medical report; specifically:
[0144] Project the vectors output by the tokenizer into the subspaces of each attention head:
[0145]
[0146] Among them, X is the sentence vector converted by the word segmenter, Q i is the Query vector of the subspace of the i-th attention head, K i is the Key vector of the subspace of the i-th attention head, V i is the Value vector of the subspace of the i-th attention head, and are different weight matrices;
[0147] According to Q i , K i and V i Calculate the attention score of the i-th attention head i :
[0148]
[0149] The superscript T represents transposition, d k is the characteristic dimension;
[0150] Concatenate the attention scores of all attention heads and map the concatenated vector to the output dimension of the attention head:
[0151] Z text =Concat(head1,head2,...,headh )W o
[0152] Among them, Z text is the text vector corresponding to the medical report, W o is a trainable projection matrix;
[0153] Step S23, the three-dimensional CT image is subjected to the encoding part of the variational autoencoder to obtain a vector representation of the three-dimensional CT image; the vector representation of the three-dimensional CT image is then used as the input of the forward denoising process of the consistent diffusion model, and the three-dimensional CT image is subjected to noise processing through the forward denoising process of the consistent diffusion model to convert the three-dimensional CT image into a noise vector;
[0154] The forward noise adding process is:
[0155]
[0156] Among them, β t represents the noise increment added at the tth time step, β t Between (0,1), ∈ is an independent Gaussian noise sampled from a standard normal distribution, x t-1 represents the noisy image obtained at the t-1th time step, x t represents the noisy image obtained at the t-th time step, t = 1, 2, ..., T, T is the total number of time steps of noise addition; by continuously repeating the noise addition process, the input 3D CT image is transformed into a Gaussian noise that obeys the normal distribution;
[0157] Step S24, using the text vector obtained in step S22 and the noise vector obtained in step S23 as inputs of the reverse denoising process of the consistency diffusion model, and obtaining a noise-free image vector x′0 through the reverse denoising process;
[0158] The specific process of step S24 is:
[0159] Step S241, initializing the number of denoising modules t=T;
[0160] Step S242: Use the text vector obtained in step S22 and the noise vector obtained in step S23 as the input of the t-th denoising module to obtain the output x′ of the t-th denoising module. t-1 ;
[0161] Step S243, determine whether t=1 is satisfied;
[0162] If satisfied, then end;
[0163] If not satisfied, continue to step S244;
[0164] Step S244: The output x′ of the t-th denoising module t-1 The text vector obtained in step S22 is used as the input of the t-1 denoising module to obtain the output x′ of the t-1 denoising module. t-2 , then let t = t-1;
[0165] Then return to step S243;
[0166] Step S25, calculating the loss function value according to the inverse denoising result x′0 and the three-dimensional CT image corresponding to the medical report, and stopping the training until the maximum epoch is reached or the maximum tolerance is reached, thereby obtaining a trained model;
[0167] The trained neural network model is used to synthesize 3D CT images. The specific process is as follows:
[0168] Obtain a medical report for synthesizing a three-dimensional CT image, pre-process the obtained medical report, and then use a word segmenter to output a vector form corresponding to the pre-processed medical report;
[0169] The output of the tokenizer is used as the input of the pre-trained medical report encoding model to obtain the text vector of the medical report;
[0170] Then randomly sample an initial Gaussian noise vector, input the sampled Gaussian noise vector and the text vector of the medical report into the consistency diffusion model for the reverse denoising process, use the output of the reverse denoising process as the input of the decoding part of the variational autoencoder, and output the synthesized three-dimensional CT image through the decoding part of the VAE.
[0171] Specific implementation method 9: This implementation method is a further limitation of specific implementation method 8, wherein the medical reports in the acquired data set are preprocessed, specifically: the medical reports are translated into English.
[0172] Specific implementation method ten: This implementation method is a further limitation of specific implementation method eight, wherein the denoising module includes a first downsampling unit to an Nth downsampling unit, a Mid Block, and a first upsampling unit to an Nth upsampling unit; wherein:
[0173] The input of the denoising module first passes through the first downsampling unit to the Nth downsampling unit in sequence, and then the output of the Nth downsampling unit is used as the input of the MidBlock, and the output of the MidBlock then passes through the first upsampling unit to the Nth upsampling unit in sequence;
[0174] The output of the first down-sampling unit is used as the input of the Nth up-sampling unit through a skip connection (another part of the input of the Nth up-sampling unit comes from the output of the N-1th up-sampling unit), the output of the second down-sampling unit is used as the input of the N-1th up-sampling unit through a skip connection (another part of the input of the N-1th up-sampling unit comes from the output of the N-2th up-sampling unit), the output of the third down-sampling unit is used as the input of the N-2th up-sampling unit through a skip connection (another part of the input of the N-2th up-sampling unit comes from the output of the N-3th up-sampling unit), and so on, the output of the N-1th down-sampling unit is used as the input of the 2nd up-sampling unit through a skip connection (another part of the input of the 2nd up-sampling unit comes from the output of the 1st up-sampling unit);
[0175] The first downsampling unit includes a 2D CNN subunit and a volume perception subunit, and the structures of the second downsampling unit to the Nth downsampling unit and the first upsampling unit to the Nth upsampling unit are the same as the structure of the first downsampling unit;
[0176] Mid Block includes four sub-modules, and the structures of the second, third and fourth sub-modules are the same as that of the first sub-module; the first sub-module includes 2D CNN sub-unit, volume perception sub-unit, Cross-Slice Control sub-unit and Cross Attention sub-unit;
[0177] The working process of the Mid Block is as follows:
[0178] Step 1: Use the output of the Nth downsampling unit as the input of the Mid Block, that is, use the output of the Nth downsampling unit as the input of the 2D CNN subunit in the first submodule;
[0179] The working process of the 2D CNN subunit is:
[0180] The 3D CT image input by the 2D CNN subunit is represented as a 5-dimensional tensor x∈R b×c×f×h×w , where b represents the batch number, c represents the channel number, f represents the slice number, h represents the height of the 3D CT image, and w represents the width of the 3D CT image; the number of slices f of the tensor is merged into the batch dimension b, and the merged tensor dimension is x∈R (b×f)×c×h×w ;
[0181] The 2D CNN subunit is used to process the tensor after the dimension is merged to extract the features of each slice. After the slice feature extraction is completed, the tensor format is resized to x∈R b×c×f×h×w ;
[0182] Step 2: Use the output of the 2D CNN subunit in the first submodule as the input of the volume perception subunit in the first submodule, where the volume perception subunit includes M Cross-Frame Attention blocks.
[0183] The working process of the volume perception subunit is:
[0184] Step 21: Use sinusoidal position coding to encode the position of each slice in the three-dimensional CT image. The sinusoidal position coding volume perception module can learn the relative relationship between each slice in the three-dimensional CT, such as the position relationship between slices;
[0185] Step 22, initialize the count m of the Cross-Frame Attention block to 1;
[0186] Step 23: Merge the spatial dimensions of the feature tensors output by the 2D CNN subunits into the batch dimension to obtain a feature tensor Z∈R consisting of the feature tensors of each slice after merging the dimensions. (b×h×w)×c×f , where the feature tensor of the jth slice can be expressed as z j ∈R (b×h×w)×c ;
[0187] Step 24: Use the dimension-merged feature tensor Z as the input of the mth Cross-Frame Attention block;
[0188] Step 25. Multiply the input feature tensor by the weight matrix W in the Cross-Frame Attention block Q Generate the Q vector by multiplying the input feature tensor by the weight matrix W K Generate K vectors, multiply the input features by the weight matrix W V Generate V vector. The dimensions of the generated Q, K, and V vectors are all (b×h×w)×f×d k , where d k is the characteristic dimension;
[0189] Then perform weighted summation based on the Q, K and V vectors to obtain the output feature tensor Z out :
[0190]
[0191] Step 26: Determine whether the count m of the Cross-Frame Attention block is less than M;
[0192] If m is less than M, then let m = m + 1 and set Z out As the input of the next Cross-Frame Attention block, return to step 25;
[0193] If m is equal to M, the final weighted summation feature tensor Z is obtained out ;
[0194] Step 3: The feature tensor Z output by the volume perception subunit of the first submodule out The position encoding result is used as the input of the Cross-Slice Control subunit of the first submodule, and the vector representation of each slice is output through the Cross-Slice Control subunit;
[0195] The working process of the Cross-Slice Control subunit is:
[0196] For the jth slice, the feature tensor Z out The feature of the j-th slice in the self-attention mechanism obtains the Q corresponding to the j-th slice j , K j and V j , the feature tensor Z out The feature of the j-1th slice in the self-attention mechanism obtains the Q corresponding to the j-1th slice j-1 , K j-1 and V j-1 ;
[0197] Then the vector of the jth slice represents x j for:
[0198]
[0199] In the self-attention mechanism, K and V vectors usually store semantic information, and Q vectors usually store spatial information. The present invention improves the consistency of generated semantics by retaining the K and V vectors corresponding to the previous slice, and uses the Q vector corresponding to the current slice to retain the spatial information of each slice.
[0200] Step 4: Use the text vector and the vector representation of each slice output by the Cross-Slice Control subunit of the first submodule as the input of the Cross Attention subunit of the first submodule, and use the output of the Cross Attention subunit as the output of the first submodule;
[0201] The working process of the Cross Attention subunit is:
[0202] Perform a linear transformation on the input text vector and convert the input text vector into K t 、V tVector, linearly transform the output of the Cross-Slice Control subunit, and convert the output of the Cross-Slice Control subunit into the Q of the image through linear transformation ci Vector, then
[0203]
[0204] Among them, Z ct Represents the image feature vector that incorporates text semantic information;
[0205] Step 5: Use the output of the first submodule and the text vector as the input of the second submodule, use the output of the second submodule and the text vector as the input of the third submodule, use the output of the third submodule and the text vector as the input of the fourth submodule, and use the output of the fourth submodule as the output of Mid Block.
[0206] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A 3D CT image synthesis method based on consistent diffusion model, It is characterized in that The method specifically comprises the following steps: Step S1, obtaining a data set of three-dimensional CT images and medical reports, and preprocessing the medical reports in the obtained data set to obtain a preprocessed three-dimensional CT image and medical report paired data set; Step S2, building a model including a pre-trained medical report encoding model, a consistency diffusion model, and a variational autoencoder; Step S3, training the constructed model using the preprocessed three-dimensional CT image and medical report paired data set; Step S4, obtaining a medical report for synthesizing a three-dimensional CT image, preprocessing the obtained medical report, and then using a word segmenter to output a vector form corresponding to the preprocessed medical report; The output of the word segmenter is used as the input of the pre-trained medical report encoding model to obtain the text vector of the medical report; then an initial Gaussian noise vector is randomly sampled, and the sampled Gaussian noise vector and the text vector of the medical report are input into the consistency diffusion model for the reverse denoising process, and the output of the reverse denoising process is used as the input of the decoding part of the variational autoencoder, and the synthesized three-dimensional CT image is output through the decoding part of the VAE.
2. A three-dimensional CT image synthesis method based on a consistent diffusion model according to claim 1, It is characterized in that The preprocessing of the medical reports in the acquired data set specifically includes: translating the medical reports into English.
3. The three-dimensional CT image synthesis method based on the consistent diffusion model according to claim 1, It is characterized in that The model is trained using the pre-processed three-dimensional CT image and medical report paired data set. The specific training process is as follows: Step S21: Processing the medical report Use a tokenizer to mark the beginning of the medical report with a [CLS] tag, and mark adjacent sentences with a [SEP] tag as a sentence separator. Then, tokenize each sentence, that is, divide each sentence into several words. Finally, map the processed sentences to integer indices and output them in the form of vectors. Step S22: using the processed vector corresponding to the medical report as the input of the pre-trained medical report encoding model, and performing feature extraction through the pre-trained medical report encoding model to output a text vector corresponding to the medical report; Step S23, the three-dimensional CT image is subjected to the encoding part of the variational autoencoder to obtain a vector representation of the three-dimensional CT image; the vector representation of the three-dimensional CT image is then used as the input of the forward denoising process of the consistent diffusion model, and the three-dimensional CT image is subjected to noise processing through the forward denoising process of the consistent diffusion model to convert the three-dimensional CT image into a noise vector; Step S24: Use the text vector obtained in step S22 and the noise vector obtained in step S23 as inputs to the reverse denoising process of the consistency diffusion model, and obtain a noise-free image vector x′ through the reverse denoising process. 0 ; The specific process of step S24 is: Step S241, initializing the number of denoising modules t=T; Step S242: Use the text vector obtained in step S22 and the noise vector obtained in step S23 as the input of the t-th denoising module to obtain the output x′ of the t-th denoising module. t-1 ; Step S243, determine whether t=1 is satisfied; If satisfied, then end; If not satisfied, continue to step S244; Step S244: The output x′ of the t-th denoising module t-1 The text vector obtained in step S22 is used as the input of the t-1 denoising module to obtain the output x′ of the t-1 denoising module. t-2 , then let t = t-1; Then return to step S243; Step S25: Based on the reverse denoising result x′ 0 The loss function value is calculated based on the three-dimensional CT image corresponding to the medical report until the maximum epoch is reached or the maximum tolerance is reached, and the training is stopped to obtain a trained model.
4. The method for synthesizing three-dimensional CT images based on a consistent diffusion model according to claim 3, It is characterized in that The process of using the vector corresponding to the processed medical report as the input of the pre-trained medical report encoding model, and performing feature extraction through the pre-trained medical report encoding model to output the text vector corresponding to the medical report is as follows: Project the vectors output by the tokenizer into the subspaces of each attention head: Among them, X is the sentence vector converted by the word segmenter, Q i is the Query vector of the subspace of the i-th attention head, K i is the Key vector of the subspace of the i-th attention head, V i is the Value vector of the subspace of the i-th attention head, and is the weight matrix; According to Q i , K i and V i Calculate the attention score of the i-th attention head i : The superscript T represents transposition, d k is the characteristic dimension; Concatenate the attention scores of all attention heads and map the concatenated vectors: Z text =Concat(head 1 ,head 2 ,……,head h )W o Among them, Z text is the text vector corresponding to the medical report, W o is the projection matrix.
5. The three-dimensional CT image synthesis method based on the consistent diffusion model according to claim 4, It is characterized in that The denoising module includes the first down-sampling unit to the Nth down-sampling unit, the Mid Block, and the first up-sampling unit to the Nth up-sampling unit; wherein: The input of the denoising module first passes through the first downsampling unit to the Nth downsampling unit in sequence, and then the output of the Nth downsampling unit is used as the input of the MidBlock, and the output of the MidBlock then passes through the first upsampling unit to the Nth upsampling unit in sequence; The output of the first down-sampling unit is used as the input of the Nth up-sampling unit through a skip connection, the output of the second down-sampling unit is used as the input of the N-1th up-sampling unit through a skip connection, the output of the third down-sampling unit is used as the input of the N-2th up-sampling unit through a skip connection, and so on, the output of the N-1th down-sampling unit is used as the input of the 2nd up-sampling unit through a skip connection; The first downsampling unit includes a 2D CNN subunit and a volume perception subunit, and the structures of the second downsampling unit to the Nth downsampling unit and the first upsampling unit to the Nth upsampling unit are the same as the structure of the first downsampling unit; Mid Block includes four sub-modules, and the structures of the second sub-module, the third sub-module and the fourth sub-module are the same as the structure of the first sub-module; The first submodule includes a 2D CNN subunit, a volume perception subunit, a Cross-Slice Control subunit, and a Cross Attention subunit.
6. A three-dimensional CT image synthesis method based on a consistent diffusion model according to claim 5, It is characterized in that The working process of the Mid Block is as follows: Step 1: Use the output of the Nth downsampling unit as the input of the Mid Block, that is, use the output of the Nth downsampling unit as the input of the 2D CNN subunit in the first submodule; The working process of the 2D CNN subunit is: The 3D CT image input by the 2D CNN subunit is represented as a 5-dimensional tensor x∈R b×c×f×h×w , where b represents the batch number, c represents the channel number, f represents the slice number, h represents the height of the 3D CT image, and w represents the width of the 3D CT image; the number of slices f of the tensor is merged into the batch dimension b, and the merged tensor dimension is x∈R (b×f)×c×h×w ; The 2D CNN subunit is used to process the tensor after the dimension is merged to extract the features of each slice. After the slice feature extraction is completed, the tensor format is resized to x∈R b×v×f×h×w ; Step 2: Use the output of the 2D CNN subunit in the first submodule as the input of the volume perception subunit in the first submodule, where the volume perception subunit includes M Cross-Frame Attention blocks. The working process of the volume perception subunit is: Step 21, using sinusoidal position coding to encode the position of each slice in the three-dimensional CT image; Step 22, initialize the count m of the Cross-Frame Attention block to 1; Step 23: Merge the spatial dimensions of the feature tensors output by the 2D CNN subunits into the batch dimension to obtain a feature tensor Z∈R consisting of the feature tensors of each slice after merging the dimensions. (b×h×w)×c×f ; Step 24: Use the dimension-merged feature tensor Z as the input of the mth Cross-Frame Attention block; Step 25. Multiply the input feature tensor by the weight matrix W in the Cross-Frame Attention block Q Generate the Q vector by multiplying the input feature tensor by the weight matrix W K Generate K vectors, multiply the input features by the weight matrix W V Generate V vector. The dimensions of the generated Q, K, and V vectors are all (b×h×w)×f×d k , where d k is the characteristic dimension; Then perform weighted summation based on the Q, K and V vectors to obtain the output feature tensor Z out : Step 26: Determine whether the count m of the Cross-Frame Attention block is less than M; If m is less than M, then set m=m+1 and change Z out As the input of the next Cross-Frame Attention block, return to step 25; If m is equal to M, the final weighted summation feature tensor Z is obtained out ; Step 3: The feature tensor Z output by the volume perception subunit of the first submodule out The position encoding result is used as the input of the Cross-Slice Control subunit of the first submodule, and the vector representation of each slice is output through the Cross-Slice Control subunit; The working process of the Cross-Slice Control subunit is: For the jth slice, the feature tensor Z out The feature of the j-th slice in the self-attention mechanism obtains the Q corresponding to the j-th slice j , K j and V j , the feature tensor Z out The feature of the j-1th slice in the self-attention mechanism obtains the Q corresponding to the j-1th slice j-1 , K j-1 and V j-1 ; Then the vector of the jth slice represents x j for: Step 4: Use the text vector and the vector representation of each slice output by the Cross-Slice Control subunit of the first submodule as the input of the Cross Attention subunit of the first submodule, and use the output of the Cross Attention subunit as the output of the first submodule; The working process of the Cross Attention subunit is: Perform a linear transformation on the input text vector and convert the input text vector into K t 、V t Vector, linearly transform the output of the Cross-Slice Control subunit, and convert the output of the Cross-Slice Control subunit into the Q of the image through linear transformation ct Vector, then Among them, Z ct Represents the image feature vector that incorporates text semantic information; Step 5: Use the output of the first submodule and the text vector as the input of the second submodule, use the output of the second submodule and the text vector as the input of the third submodule, use the output of the third submodule and the text vector as the input of the fourth submodule, and use the output of the fourth submodule as the output of Mid Block.
7. The method for synthesizing three-dimensional CT images based on a consistent diffusion model according to claim 6, It is characterized in that The loss function is: Among them, x 0 represents the real input image, f text represents the text vector corresponding to the input medical report, θ is the parameter of the consistency diffusion model, x t represents the output of the forward noise addition process, G(x t ,f text ; θ) represents the output of the inverse denoising process, q(x 0 ,f text ) represents x 0 and f text The joint distribution of q(x t |x 0 ) means that at a given x 0 Under the condition of t The conditional distribution of , L is the loss function, Denotes the joint distribution q(x 0 ,f text ) and the conditional distribution q(x t |x 0 )’s expected value.
8. A 3D CT image synthesis system based on a consistent diffusion model. It is characterized in that The system includes a data set acquisition module, a data set preprocessing module and a neural network model; A data set acquisition module, used to acquire a data set of paired three-dimensional CT images and medical reports; A data set preprocessing module is used to preprocess the medical reports in the acquired data set to obtain a preprocessed three-dimensional CT image and medical report paired data set; The neural network model includes a pre-trained medical report encoding model, a consistency diffusion model, and a variational autoencoder; And the neural network model is trained using the preprocessed data set, and the training process is: Step S21: Processing the medical report Use a tokenizer to mark the beginning of the medical report with a [CLS] tag, and mark adjacent sentences with a [SEP] tag as a sentence separator. Then, tokenize each sentence, that is, divide each sentence into several words. Finally, map the processed sentences to integer indices and output them in the form of vectors. Step S22: using the processed vector corresponding to the medical report as the input of the pre-trained medical report coding model, and performing feature extraction through the pre-trained medical report coding model to output the text vector corresponding to the medical report; specifically: Project the vectors output by the tokenizer into the subspaces of each attention head: Among them, X is the sentence vector converted by the word segmenter, Q i is the Query vector of the subspace of the i-th attention head, K i is the Key vector of the subspace of the i-th attention head, V i is the Value vector of the subspace of the i-th attention head, and is the weight matrix; According to Q i , K i and V i Calculate the attention score of the i-th attention head i : The superscript T represents transposition, d k is the characteristic dimension; Concatenate the attention scores of all attention heads and map the concatenated vectors: Z text =Concat(head 1 ,head 2 ,……,head h )W o Among them, Z text is the text vector corresponding to the medical report, W o is the projection matrix; Step S23, the three-dimensional CT image is subjected to the encoding part of the variational autoencoder to obtain a vector representation of the three-dimensional CT image; the vector representation of the three-dimensional CT image is then used as the input of the forward denoising process of the consistent diffusion model, and the three-dimensional CT image is subjected to noise processing through the forward denoising process of the consistent diffusion model to convert the three-dimensional CT image into a noise vector; Step S24: Use the text vector obtained in step S22 and the noise vector obtained in step S23 as inputs to the reverse denoising process of the consistency diffusion model, and obtain a noise-free image vector x′ through the reverse denoising process. 0 ; The specific process of step S24 is: Step S241, initializing the number of denoising modules t=T; Step S242: Use the text vector obtained in step S22 and the noise vector obtained in step S23 as the input of the t-th denoising module to obtain the output x′ of the t-th denoising module. t-1 ; Step S243, determine whether t=1 is satisfied; If satisfied, then end; If not satisfied, continue to step S244; Step S244: The output x′ of the t-th denoising module t-1 The text vector obtained in step S22 is used as the input of the t-1 denoising module to obtain the output x′ of the t-1 denoising module. t-2 , then let t = t-1; Then return to step S243; Step S25: Based on the reverse denoising result x′ 0 And the loss function value is calculated based on the three-dimensional CT image corresponding to the medical report, and the training is stopped when the maximum epoch is reached or the maximum tolerance is reached to obtain a trained model; The trained neural network model is used to synthesize 3D CT images. The specific process is as follows: Obtain a medical report for synthesizing a three-dimensional CT image, pre-process the obtained medical report, and then use a word segmenter to output a vector form corresponding to the pre-processed medical report; The output of the tokenizer is used as the input of the pre-trained medical report encoding model to obtain the text vector of the medical report; Then randomly sample an initial Gaussian noise vector, input the sampled Gaussian noise vector and the text vector of the medical report into the consistency diffusion model for the reverse denoising process, use the output of the reverse denoising process as the input of the decoding part of the variational autoencoder, and output the synthesized three-dimensional CT image through the decoding part of the VAE.
9. A three-dimensional CT image synthesis system based on a consistent diffusion model according to claim 8, It is characterized in that The preprocessing of the medical reports in the acquired data set specifically includes: translating the medical reports into English.
10. A three-dimensional CT image synthesis system based on a consistent diffusion model according to claim 8, It is characterized in that The denoising module includes the first down-sampling unit to the Nth down-sampling unit, the Mid Block, and the first up-sampling unit to the Nth up-sampling unit; wherein: The input of the denoising module first passes through the first downsampling unit to the Nth downsampling unit in sequence, and then the output of the Nth downsampling unit is used as the input of the MidBlock, and the output of the MidBlock then passes through the first upsampling unit to the Nth upsampling unit in sequence; The output of the first down-sampling unit is used as the input of the Nth up-sampling unit through a skip connection, the output of the second down-sampling unit is used as the input of the N-1th up-sampling unit through a skip connection, the output of the third down-sampling unit is used as the input of the N-2th up-sampling unit through a skip connection, and so on, the output of the N-1th down-sampling unit is used as the input of the 2nd up-sampling unit through a skip connection; The first downsampling unit includes a 2D CNN subunit and a volume perception subunit, and the structures of the second downsampling unit to the Nth downsampling unit and the first upsampling unit to the Nth upsampling unit are the same as the structure of the first downsampling unit; Mid Block includes four sub-modules, and the structures of the second, third and fourth sub-modules are the same as that of the first sub-module; the first sub-module includes 2D CNN sub-unit, volume perception sub-unit, Cross-Slice Control sub-unit and Cross Attention sub-unit; The working process of the Mid Block is as follows: Step 1: Use the output of the Nth downsampling unit as the input of the Mid Block, that is, use the output of the Nth downsampling unit as the input of the 2D CNN subunit in the first submodule; The working process of the 2D CNN subunit is: The 3D CT image input by the 2D CNN subunit is represented as a 5-dimensional tensor x∈R b×c×f×h×w , where b represents the batch number, c represents the channel number, f represents the slice number, h represents the height of the 3D CT image, and w represents the width of the 3D CT image; the number of slices f of the tensor is merged into the batch dimension b, and the merged tensor dimension is x∈R (b×f)×c×h×w ; The 2D CNN subunit is used to process the tensor after the dimension is merged to extract the features of each slice. After the slice feature extraction is completed, the tensor format is resized to x∈R b×c×f×h×w ; Step 2: Use the output of the 2D CNN subunit in the first submodule as the input of the volume perception subunit in the first submodule, where the volume perception subunit includes M Cross-Frame Attention blocks. The working process of the volume perception subunit is: Step 21, using sinusoidal position coding to encode the position of each slice in the three-dimensional CT image; Step 22, initialize the count m of the Cross-Frame Attention block to 1; Step 23: Merge the spatial dimensions of the feature tensors output by the 2D CNN subunits into the batch dimension to obtain a feature tensor Z∈R consisting of the feature tensors of each slice after merging the dimensions. (b×h×w)×c×f ; Step 24: Use the dimension-merged feature tensor Z as the input of the mth Cross-Frame Attention block; Step 25. Multiply the input feature tensor by the weight matrix W in the Cross-Frame Attention block Q Generate the Q vector by multiplying the input feature tensor by the weight matrix W K Generate K vectors, multiply the input features by the weight matrix W V Generate V vector. The dimensions of the generated Q, K, and V vectors are all (b×h×w)×f×d k , where d k is the characteristic dimension; Then perform weighted summation based on the Q, K and V vectors to obtain the output feature tensor Z out : Step 26: Determine whether the count m of the Cross-Frame Attention block is less than M; If m is less than M, then set m=m+1 and change Z out As the input of the next Cross-Frame Attention block, return to step 25; If m is equal to M, the final weighted summation feature tensor Z is obtained out ; Step 3: The feature tensor Z output by the volume perception subunit of the first submodule out The position encoding result is used as the input of the Cross-Slice Control subunit of the first submodule, and the vector representation of each slice is output through the Cross-Slice Control subunit; The working process of the Cross-Slice Control subunit is: For the jth slice, the feature tensor Z out The feature of the j-th slice in the self-attention mechanism obtains the Q corresponding to the j-th slice j , K j and V j , the feature tensor Z out The feature of the j-1th slice in the self-attention mechanism obtains the Q corresponding to the j-1th slice j-1 , K j-1 and V j-1 ; Then the vector of the jth slice represents x j for: Step 4: Use the text vector and the vector representation of each slice output by the Cross-Slice Control subunit of the first submodule as the input of the Cross Attention subunit of the first submodule, and use the output of the Cross Attention subunit as the output of the first submodule; The working process of the Cross Attention subunit is: Perform a linear transformation on the input text vector and convert the input text vector into K t 、V t Vector, linearly transform the output of the Cross-Slice Control subunit, and convert the output of the Cross-Slice Control subunit into the Q of the image through linear transformation ct Vector, then Among them, Z ct Represents the image feature vector that incorporates text semantic information; Step 5: Use the output of the first submodule and the text vector as the input of the second submodule, use the output of the second submodule and the text vector as the input of the third submodule, use the output of the third submodule and the text vector as the input of the fourth submodule, and use the output of the fourth submodule as the output of Mid Block.
