Medical image generation method and device based on bimodal fusion, equipment and medium

Abstract

The invention discloses a medical image generation method and device based on bimodal fusion, equipment and a medium, and the method comprises the steps: respectively extracting the visual features of a medical image and the semantic features of text description through an image encoder and a text encoder; mapping the two types of features to a shared semantic space by adopting comparative learning to realize cross-modal alignment; a first path captures hierarchical semantic information through a convolutional neural network, a second path retains local significant features through maximum pooling, and two paths of outputs are fused layer by layer to construct spatial context features; projecting the cross-modal features into a spatial feature map through a multi-layer perceptron, splicing the spatial feature map with coding features, and inputting the spliced spatial feature map into a decoder for up-sampling reconstruction; and performing joint optimization on the comparison loss and the structural similarity loss to realize end-to-end training. And finally, the SSIM index of the generated medical image is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image generation, and in particular to a medical image generation method, device, equipment and medium based on bimodal fusion. Background Art

[0002] Currently, in medical image generation models, generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs) have been widely used in generating medical images. However, these techniques have limited performance when dealing with medical data with long-tailed distributions. The scarcity of rare disease case data greatly limits the training and generation capabilities of deep learning models, and existing generative models, especially when combining image and text information, have deficiencies, lacking the correlation between text and image, reducing the generation quality and consistency of rare case images. Therefore, there is an urgent need for a method that can improve the SSIM index of medical images generated by medical image generation models. Summary of the Invention

[0003] Embodiments of the present invention provide a medical image generation method, device, equipment and medium based on bimodal fusion, aiming to solve the problem of low quality of medical images generated by medical image generation models in the prior art.

[0004] In a first aspect, an embodiment of the present invention provides a medical image generation method based on bimodal fusion, including:

[0005] Obtain a medical image and its corresponding text description, and perform standardized preprocessing to obtain a preprocessed image and a text sequence;

[0006] Extract image features from the preprocessed image through an image encoder, extract text features from the text sequence through a text encoder, and map and fuse the image features and the text features into a shared semantic space through contrastive learning to obtain cross-modal features;

[0007] Extract multi-scale features of the medical image through a dual-path encoder to obtain encoded features;

[0008] Fuse the cross-modal features with the encoded features to obtain decoder input features, and generate a target image through a decoder according to the decoder input features;

[0009] Construct a joint loss function using the target image to jointly optimize the network parameters of the image encoder, the text encoder, the dual-path encoder and the decoder.

[0010] In a second aspect, an embodiment of the present invention further provides a medical image generation device based on bimodal fusion, which is used to execute the medical image generation method based on bimodal fusion as described above.

[0011] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to execute the steps of the above-mentioned medical image generation method based on dual-modal fusion.

[0012] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions that can implement the steps of the above-mentioned medical image generation method based on dual-modal fusion when executed by a processor.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0014] In the technical solution of the present invention, the visual features of medical images and the semantic features of text descriptions are respectively extracted by an image encoder and a text encoder, and contrastive learning is used to map the two types of features to a shared semantic space to achieve cross-modal alignment; a dual-path encoder is constructed to perform multi-scale feature extraction on the medical image: the first path captures hierarchical semantic information through a convolutional neural network, and the second path retains local significant features through max pooling, and the outputs of the two paths are fused layer by layer to construct spatial context features; the cross-modal features are projected into a spatial feature map through a multi-layer perceptron, and after being concatenated with the encoded features, they are input into a decoder for upsampling and reconstruction; the contrastive loss and the structural similarity loss are jointly optimized to achieve end-to-end training. Finally, the SSIM index of the generated medical image is improved. Description of the Drawings

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a flowchart of the medical image generation method based on dual-modal fusion provided by the present invention;

[0017] Figure 2 It is a first sub-flowchart of the medical image generation method based on dual-modal fusion provided by the present invention;

[0018] Figure 3 It is a second sub-flowchart of the medical image generation method based on dual-modal fusion provided by the present invention;

[0019] Figure 4 It is a third sub-flowchart of the medical image generation method based on dual-modal fusion provided by the present invention;

[0020] Figure 5 This is the fourth sub - flowchart of the medical image generation method based on bimodal fusion provided by the present invention;

[0021] Figure 6 This is the fifth sub - flowchart of the medical image generation method based on bimodal fusion provided by the present invention;

[0022] Figure 7 This is a schematic block diagram of the unit of the medical image generation device based on bimodal fusion provided by the present invention;

[0023] Figure 8 This is a schematic block diagram of the computer device provided by the embodiment of the present invention. Detailed implementation manners

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0025] It should be understood that when used in this specification and the appended claims, the terms "comprises" and "comprising" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0026] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing medical embodiments and are not intended to limit the present invention. As used in this specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0027] It should be further understood that the term " / and / " used in this specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations.

[0028] In order to solve the problem of low - quality medical images generated by the medical image generation model in the prior art, the present invention provides a medical image generation method, device, equipment, and medium based on bimodal fusion. Refer to Figures 1 to 6 , and the medical image generation method based on bimodal fusion therein includes the following steps.

[0029] S110. Obtain a medical image and its corresponding text description, and perform standardized preprocessing to obtain a preprocessed image and a text sequence;

[0030] When a medical image generation model generates a medical image for a target requirement, it is first necessary to obtain the source file for generating the medical image of the target requirement. For the method of this case, the so-called source file refers to the medical image data that can be used for model processing and the text description used by medical staff to refine the description of the image or put forward image generation requirements. Specifically, medical images are usually two-dimensional or three-dimensional images obtained by medical imaging techniques such as CT scans, MRI scans, or X-rays. The text description can be the report of the image accessory, the medical case, and the key disease description phrases in the text conclusion given by the model for medical record analysis. After that, the medical image needs to be processed so that image extraction can be performed through the encoder.

[0031] Further, referring to Figure 2 , step S110 includes:

[0032] S111. Obtain the medical image and its corresponding text description through an interaction platform;

[0033] S112. Perform format standardization, numerical standardization, and size unification processing on the medical image to obtain the preprocessed image;

[0034] S113. Perform key information extraction, word segmentation processing, and length regularization on the text description to obtain the text sequence;

[0035] The so-called interaction platform refers to the software used by medical staff to interact with the model for input and output. Through this software, medical staff can input medical images and text descriptions into the model. Specifically, medical images are stored in DICOM format, and text descriptions exist in the form of electronic reports. For convenient use, a program for extracting key disease description phrases from radiology reports is also set in front of the model to reduce the labor burden.

[0036] During the preprocessing of medical images, format standardization processing can be performed on medical images as needed. For example, convert the original DICOM file to PNG format, eliminate the manufacturer-specific coding differences, and retain the grayscale information. Numerical standardization processing can be performed on medical images as needed. For example, perform normalization and Z-score standardization. Size unification processing can be performed on medical images as needed. For example, adjust the image resolution to 224×224 through bicubic interpolation to adapt to the model input requirements. Finally, a preprocessed image can be generated.

[0037] During the preprocessing of the text description, information extraction can be performed on the text description as needed. For example, the disease description fragment (such as "periosteal hyperplasia") can be located from the radiology report, and irrelevant text can be filtered. Word segmentation can be performed on the text description as needed. For example, the BERT-base tokenizer is used to generate the WordPiece token sequence, and the [CLS] start token and [SEP] separator are added. The length of the text description can be regularized as needed. For example, the text sequence is padded or truncated to a fixed length of 64 tokens, and the insufficient part is filled with zeros. Finally, a text sequence can be generated.

[0038] S120. Extract image features from the preprocessed image through an image encoder, extract text features from the text sequence through a text encoder, and map the image features and the text features to a shared semantic space and fuse them through contrastive learning to obtain cross-modal features;

[0039] After obtaining the preprocessed image and the text sequence, image features and text features are obtained through an image encoder and a text encoder. The image features and text features referred to here are specifically the embeddings of the image and the text. Two mapping networks are trained through contrastive learning to map the embeddings of the image and the text in different dimensions to a shared semantic space. Specifically, through the image mapping network and the text mapping network, the embeddings are linearly transformed and normalized and mapped into the shared space as vectors to obtain image and text vectors of the same dimension, and then directly added to fuse the bimodal information to obtain the final input, that is, cross-modal features.

[0040] Further, referring to Figure 3 , step S120 includes:

[0041] S121. Input the preprocessed image into the image encoder for multi-level feature extraction to obtain an image embedding vector;

[0042] S122. Input the text sequence into the text encoder for context semantic encoding to obtain a text embedding vector

[0043] S123. Map the image embedding vector and the text embedding vector to a shared semantic space through a mapping network trained by contrastive learning;

[0044] S124. Perform normalization processing on the image embedding vector and the text embedding vector to obtain normalized image features and normalized text features;

[0045] S125. Add the normalized image features and the normalized text features element by element to fuse them to obtain cross-modal features.

[0046] Input the preprocessed image into the image encoder. For example, the preprocessed image is a normalized CT grayscale image of a diseased bone joint (224×224×1), and the image encoder is ResNet-50 or EfficientNet-B5. Through hierarchical convolution processing, after the feature layer and the semantic layer, the final output flattens the highest-level features into a 2048-dimensional vector as the image embedding. The convolutional layer can use dynamic hierarchical partitioning to achieve multi-resolution image hybrid data processing. Furthermore, deformable convolutions can be used to enhance the adaptability to pathological deformations by dynamically adjusting the sampling positions of the convolutional kernels.

[0047] Input the text sequence into the text encoder for context semantic encoding by the text encoder. For example, input the padded 64-token sequence, and choose to use the BERT model or BioClinicalBERT as the text editor according to the difficulty of understanding professional medical terms. The context semantic encoding mechanism can include self-attention calculation or positional encoding injection. Finally, generate the text embedding. For example, extract the 768-dimensional vector of the [CLS] token as the text embedding.

[0048] Through the image mapping network and the text mapping network trained by contrastive learning, map image and text information of different dimensions to the same shared space, and the dimension of the shared space is preset. Respectively perform linear transformation on the image embedding and the text embedding, and through L2 normalization processing or batch normalization scaling processing, obtain the normalized image features and the normalized text features, which are image and text vectors of the same dimension. Then, perform element-wise addition fusion on the bimodal information to obtain a vector, which is the cross-modal feature. Specifically, deep fusion can also be used, that is, adopt a multi-layer fusion strategy to fuse the image and text features layer by layer at different levels, rather than directly fusing at the last layer. This enables information to be fused at different abstraction levels and improves the performance of the model.

[0049] S130. Extract multi-scale features from the medical image through a dual-path encoder to obtain encoded features;

[0050] Use the medical image as the input to the dual-path encoder, perform feature extraction through the first encoding path and the second encoding path respectively, and perform element-wise addition fusion of the image semantic features and the hierarchical features at each encoding level, and finally extract the encoded features.

[0051] Further, referring to Figure 4 , step S130 includes:

[0052] S131. Use the medical image as the input to the dual-path encoder;

[0053] S132. Use a convolutional neural network to extract image semantic features layer by layer through the first encoding path of the dual-path encoder;

[0054] S133. Obtain hierarchical features by successively downsampling through the max pooling layer via the second encoding path of the dual-path encoder;

[0055] S134. At each encoding level, perform element-wise addition of the image semantic features of the current level and the hierarchical features to obtain fused encoding features.

[0056] Input the same image copy into two independent paths for processing. One encoding path uses a convolutional neural network (CNN) to extract image semantic features layer by layer. The input medical image will be processed through multiple convolutional layers, and each layer will extract different-level features, which can reflect important structural information in the image.

[0057] The other encoding path obtains hierarchical features by successively downsampling through the max pooling layer. Max pooling slides a small pooling window over the feature map of the image and selects the maximum value within each pooling window, thereby reducing the size of the image. After the pooling operation, the width and height of the image are halved, but the key information in the image is still retained. Since the hierarchical features of the other encoding path are obtained from the pooling layer, these features are usually high-level features after gradually reducing the resolution through multiple pooling layers. They can retain the global information in the image and provide an important basis for subsequent feature fusion.

[0058] On the basis of dimension alignment, the image semantic features and hierarchical features are fused at each encoding level by element-wise addition. The purpose of fusion is to combine the detailed semantic information of the image with the global information in the hierarchical features to obtain a more rich feature representation. Through element-wise addition, the two can complement each other, so that the fused features have both the information of image details and the context information of the global structure.

[0059] S140. Fuse the cross-modal features with the encoding features to obtain decoder input features, and generate a target image through the decoder according to the decoder input features;

[0060] Fuse the cross-modal features with the encoding features, and the decoder generates a target image according to the input fused features. The decoder is usually an upsampling network, such as a transposed convolution network, whose purpose is to generate a complete image from a low-dimensional feature vector. The generated image should be able to accurately reflect the details of the medical image and conform to the content mentioned in the text description. In actual implementation, the design of the decoder network may include multiple convolutional layers, transposed convolutional layers, and skip connections, etc., so that the generated image has high-quality details.

[0061] Further, refer toFigure 5 , step S140 includes:

[0062] S141. Project the cross-modal features into a spatial feature map through a fully connected layer;

[0063] S142. Concatenate the spatial feature map and the fused encoded features along the channel dimension to obtain the decoder input features;

[0064] S143. Upsample the decoder input features step by step through a transposed convolutional network to obtain the final upsampled image;

[0065] S144. Map the final upsampled image to the target medical image through an image mapping back to grayscale convolutional layer.

[0066] Through the mapping of the fully connected layer, the cross-modal features are projected into a high-dimensional feature space through a multi-layer perceptron (MLP) or transposed convolutional upsampling, and then converted into a spatial feature map. During the projection process, each element of the cross-modal features will undergo weighted summation and non-linear transformation through an activation function, and finally a spatial feature map suitable for subsequent processing is obtained. After projection, dimension matching is also required to ensure that the size of the projected feature map is consistent with the fused encoded feature space output by the dual-path encoder.

[0067] Concatenate the spatial feature map and the fused encoded features along the channel dimension, merge the spatial feature map and the encoded features along the channel dimension, and generate the decoder input features with a preset specified number of channels to prepare for subsequent image generation. Channel concatenation retains the dual-path encoded features and cross-modal features, which can avoid feature loss caused by single-modal dominance. Element-wise multiplication can also be used to merge the spatial feature map and the encoded features.

[0068] The decoder upsamples the obtained decoder input features step by step through a transposed convolutional network, so that the size of the feature map gradually returns to the size of the target image, and the final upsampled image is obtained. Through the transposed convolutional operation, the feature map can be gradually restored from a low resolution to a higher resolution, maintaining the structure and details of the image. This process can retain the main features of the image during the restoration process while reducing distortion.

[0069] After that, the decoder uses an image mapping back to grayscale convolutional layer to map the final upsampled image to the target medical image. This convolutional layer usually includes several convolutional kernels, and after the convolutional operation, the feature map is converted into the grayscale or pseudo-color map of the medical image. This is to convert the high-dimensional feature map generated by the decoder into a format that meets the requirements of the actual medical image and restore the visual details and semantic information of the original image. Through this mapping process, the decoder can finally generate a target image that meets the medical standards, ensuring its realism and details in terms of gray scale and texture, and being consistent with the type of the input medical image.

[0070] S150. Use the target image to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual-path encoder, and the decoder.

[0071] Use the target image to construct a joint loss function, which is used to guide the model to optimize during training. The joint loss function is usually composed of multiple single loss functions, and each single loss function focuses on different optimization objectives.

[0072] Further, referring to Figure 6 , step S150 includes:

[0073] S151. Construct a joint loss function through the target image, where the joint loss function includes a contrast loss and a structural similarity loss, the contrast loss includes an image mapping network loss and a text mapping network loss, and the structural similarity loss includes a luminance component loss, a contrast component loss, and a structural component loss;

[0074] S152. Based on the joint loss function, jointly optimize the network parameters of the image encoder, the text encoder, the dual-path encoder, and the decoder through the backpropagation algorithm.

[0075] The joint loss function includes a contrast loss and a structural similarity loss. The contrast loss measures the similarity between two modalities, ensuring that the image and the text are as consistent as possible in the feature space.

[0076] The image mapping network loss and the text mapping network loss are two parts of the contrast loss. Among them, the image mapping network loss is used to measure the difference between the features of the generated image and the features of the target image, aiming to minimize the deviation in the image feature space. The text mapping network loss: measures the difference between the features of the text description and the features of the target image, ensuring the consistency of the text information and the image content.

[0077] The structural similarity loss includes a luminance component loss for measuring the difference in image luminance, a contrast component loss for measuring the difference in image contrast, and a structural component loss for measuring the difference in image structural information. The combination of these loss functions enables the optimization process to not only consider the consistency between modalities but also ensure that the quality and structural features of the generated image are close to the target image.

[0078] By constructing such a joint loss function, it is possible to simultaneously optimize the matching degree between the image and the text and the quality of the image itself during the training process. Especially in cross-modal generation tasks, the joint loss can effectively balance the feature fusion of different modalities, thereby improving the quality and accuracy of image generation.

[0079] The joint loss function is used as the optimization objective, and gradient calculation and parameter update are performed through the backpropagation algorithm. The backpropagation algorithm calculates the gradient of the loss function from the output layer to the input layer in reverse, and updates the parameters in the network layer by layer. The process of backpropagation involves the influence of each parameter on the loss function, and can effectively guide the network to adjust in the direction of minimizing the loss. By calculating the gradient of the joint loss function, the backpropagation algorithm will update the parameters of the image encoder, text encoder, dual-path encoder and decoder simultaneously, so as to realize the synchronous optimization of cross-modal information. Through joint optimization, not only can the cross-modal matching ability of images and texts be improved, but also the details in the image generation process can be made more accurate. This joint optimization can efficiently integrate the features of multiple modules and improve the performance and generation effect of the overall network.

[0080] Furthermore, in addition to the contrast loss and structural similarity loss, perceptual loss or adversarial loss can also be considered. The perceptual loss measures the similarity between the generated image and the target image through the similarity of high-level features, which can enhance the perceptual quality of the image. The adversarial loss can enhance the authenticity of the image through a generative adversarial network (GANs), making the generated image more realistic.

[0081] Compared with the prior art, in the technical solution of the present invention, the visual features of medical images and the semantic features of text descriptions are respectively extracted through an image encoder and a text encoder, and contrastive learning is used to map the two types of features to a shared semantic space to achieve cross-modal alignment; a dual-path encoder is constructed to perform multi-scale feature extraction on the medical image: the first path captures hierarchical semantic information through a convolutional neural network, and the second path retains local significant features through max pooling, and the outputs of the two paths are fused layer by layer to construct spatial context features; the cross-modal features are projected into a spatial feature map through a multi-layer perceptron, and after being concatenated with the encoded features, they are input into a decoder for upsampling and reconstruction; the contrast loss and the structural similarity loss are jointly optimized to achieve end-to-end training. Finally, the SSIM index of the generated medical image is improved.

[0082] To more clearly illustrate the solution of the method of the present invention, the following will take the medical images of the rare disease of hypertrophic osteoarthropathy with a low incidence rate as an example for training a deep learning model to give a detailed description.

[0083] Taking the medical image of hypertrophic osteoarthropathy in medicine as an example, such as a diseased osteoarticular CT gray-scale image with a dimension of 224×244×1 channel, and the corresponding text is "periosteal hyperplasia of the middle phalanx". Preprocess the image and text, and standardize the image x img to ensure that the numerical ranges of the data are consistent.

[0084] Perform image-text embedding. Input the diseased osteoarticular CT gray-scale image and the corresponding text symptom description x text ∈R L , where L is the length of the input text.

[0085] For the image x img Extract features through the pre-trained ResNet-50 network to obtain the image embedding z img , that is, d i = 2048 is the dimension of the image embedding. For the text symptom description x text Process it through the pre-trained BERT model to generate the text embedding z text , that is, d t = 768 is the dimension after tokenization of the text, taking BERT-base as an example.

[0086] Perform cross-modal information fusion. After obtaining the embeddings of the image and the text, use z img and z text Train two mapping networks through contrastive learning to map the image and text information of different dimensions into the same shared space, whose dimension is d, such as 512 or 1024. Taking d = 1024 as an example.

[0087] The image mapping network receives Through a linear transformation (linear transformation matrix) to map it to the vector z in the shared space ’ img

[0088] z' img = Normalize(W img ·z img )

[0089] where Normalize is normalization, such as using L2 normalization to ensure that the output vector has unit length.

[0090] The text mapping network receives Through a linear transformation (linear transformation matrix) to map it to the vector z in the shared space ’ text

[0091] z' text = Normalize(W text ·z text )

[0092] where Normalize is also normalization, such as using L2 normalization.

[0093] After obtaining the image and text vectors of the same dimension, they are directly added together to fuse the bimodal information to obtain the final input.

[0094]

[0095] The original image is used as the input of the encoder in the dual-path codec through the dual-path encoder. One path extracts image information from the image through CNN; the other path directly downsamples the image using max pooling and gradually downsamples it to retain local significant features.

[0096] Initial: The input is

[0097]

[0098] It is the information of the 0th layer.

[0099] The CNN path uses a 3×3 convolutional kernel, a stride of 2, and a padding of 1. At the same time, as the network depth increases, the number of channels gradually increases to capture richer feature information. The dimensionality change of the intermediate result after each layer of CNN processing is 224×224×1 → 112×112×32 → 56×56×64 → 28×28×128 → 14×14×256 → 7×7×512.

[0100]

[0101] conv is the result after convolution. Each layer uses ReLU as the activation function (ReLU) and adds batch normalization (BN) to accelerate network training and improve stability.

[0102]

[0103] The direct downsampling path processes the image through a max pooling layer with a stride of 2 and a window size of 2.

[0104]

[0105] The downsampling process is 224×224×1 → 112×112×1 → 56×56×1 → 28×28×1 → 14×14×1 → 7×7×1. In each step, the information extracted by CNN and the downsampled information are added element-wise, and then the next round of downsampling is performed. The formula for each layer is as follows:

[0106] Overall calculation for each layer:

[0107]

[0108] The finally obtained output is Element-wise addition is meant. Padding is performed on the last dimension to make the last dimension the same as the other one.

[0109] The input features generate the target image through the decoder. The low-resolution result x of the dual-path encoder add and the result after fusing the image and text through contrastive learning are added element-wise for fusion to obtain the input x of the decoder decode . It is necessary to re-map it to the same dimension. A multi-layer perceptron (MLP) can be used to map it to 7×7×512.

[0110]

[0111] The is continuously upsampled to the target resolution to restore the details of the medical image. The low-resolution feature map is gradually upsampled to the target resolution through transposed convolution, i.e., 7×7×512→14×14×512→28×28×512→56×56×512→112×112×512→224×224×512. Each layer uses a transposed convolution with a kernel of 2×2 and a stride of 2 for upsampling, and batch normalization (BN) and ReLU are used as the activation function (ReLU).

[0112]

[0113] Obtain Finally, a convolutional layer is added to map the image back to a grayscale image

[0114] Finally, the generated grayscale CT images of hypertrophic osteoarthropathy joints can be used as expansion data for training other deep learning models.

[0115] The joint loss function is constructed using the said target image to jointly optimize the image encoder. During the training of the image and network mapping network, the contrastive loss function is used as the training objective. For each image i, its cosine similarity with the text j is calculated. The larger the value, the more relevant it is, and vice versa.

[0116]

[0117] The loss of the image mapping network is L img , and the loss of the text mapping network is L text . The overall loss is

[0118] Training of the image generated by the encoder-decoder. For the generated image and the target image y img Find the brightness difference where the image mean is used as the brightness H and W are the length and width of the image:

[0119]

[0120] Contrast difference where the image variance is used as the contrast

[0121]

[0122] Structural difference

[0123]

[0124] The differences above. Take these three comparison differences as losses,

[0125]

[0126] Train the image generated by the codec

[0127] Figure 7 is a schematic block diagram of a medical image generation device 600 based on dual-modal fusion provided by an embodiment of the present invention. As Figure 7 shown, corresponding to the above medical image generation method based on dual-modal fusion, the present invention also provides a medical image generation device 600 based on dual-modal fusion. The medical image generation device 600 based on dual-modal fusion includes units for executing the above medical image generation method based on dual-modal fusion, and the device can be configured in terminals such as desktop computers, tablet computers, smart phones, etc.

[0128] Specifically, please refer to Figure 7 The medical image generation device 600 based on dual-modal fusion includes:

[0129] An information preprocessing unit 601, configured to obtain a medical image and a corresponding text description, and perform standardized preprocessing to obtain a preprocessed image and a text sequence;

[0130] A cross-modal encoding and fusion unit 602, configured to extract image features from the preprocessed image through an image encoder, extract text features from the text sequence through a text encoder, and map the image features and the text features to a shared semantic space through contrastive learning and fuse them to obtain cross-modal features;

[0131] The dual-channel encoder working unit 603 is used to perform multi-scale feature extraction on the medical image through a dual-channel encoder to obtain fused encoded features;

[0132] The image decoding unit 604 is used to generate decoder input features by fusing the cross-modal features and the encoded features, and generate a target image through a decoder according to the decoder input features;

[0133] The optimization training unit 605 is used to construct a joint loss function using the target image to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder.

[0134] Furthermore, the information preprocessing unit 601 includes:

[0135] The information acquisition unit 6011 is used to obtain the medical image and the corresponding text description through an interactive platform;

[0136] The image preprocessing unit 6012 is used to perform format standardization, numerical standardization, and size unification on the medical image to obtain the preprocessed image;

[0137] The text preprocessing unit 6013 is used to extract key information, perform word segmentation, and regularize the length on the text description to obtain the text sequence.

[0138] Furthermore, the cross-modal encoding and fusion unit 602 includes:

[0139] The image encoding unit 6021 is used to input the preprocessed image into the image encoder for multi-level feature extraction to obtain an image embedding vector;

[0140] The text encoding unit 6022 is used to input the text sequence into the text encoder for context semantic encoding to obtain a text embedding vector

[0141] The embedding vector mapping unit 6023 is used to map the image embedding vector and the text embedding vector to a shared semantic space through a mapping network trained by contrastive learning;

[0142] The embedding vector normalization unit 6024 is used to perform normalization processing on the image embedding vector and the text embedding vector to obtain a normalized image feature and a normalized text feature;

[0143] The embedding vector fusion unit 6025 is used to perform element-wise addition fusion on the normalized image feature and the normalized text feature to obtain cross-modal features.

[0144] Furthermore, the dual-channel encoder working unit 603 includes:

[0145] A dual-channel encoder input unit 6031 for using the medical image as the input of the dual-channel encoder;

[0146] A first encoder unit 6032 for using a convolutional neural network to extract image semantic features layer by layer through the first encoding path of the dual-channel encoder;

[0147] A second encoder unit 6033 for obtaining hierarchical features by successively downsampling through a max-pooling layer through the second encoding path of the dual-channel encoder;

[0148] A fused encoding feature unit 6034 for element-wise adding the image semantic features of the current level and the hierarchical features at each encoding level to obtain fused encoding features.

[0149] Further, the image decoding unit 604 includes:

[0150] A fully connected layer projection unit 6041 for projecting the cross-modal features into a spatial feature map through a fully connected layer;

[0151] A channel splicing unit 6042 for splicing the spatial feature map and the fused encoding features in channels to obtain decoder input features;

[0152] A successive upsampling unit 6043 for successively upsampling the decoder input features through a transposed convolutional network to obtain a final upsampled image;

[0153] An image state restoration unit 6064 for mapping the final upsampled image to a target medical image through an image mapping back to a grayscale image convolutional layer.

[0154] Further, the optimization training unit 605 includes:

[0155] A joint loss function establishment unit 6051 for constructing a joint loss function through the target image, where the joint loss function includes a contrast loss and a structural similarity loss, the contrast loss includes an image mapping network loss and a text mapping network loss, and the structural similarity loss includes a luminance component loss, a contrast component loss, and a structural component loss;

[0156] A model optimization unit 6052 for jointly optimizing the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder based on the joint loss function through a backpropagation algorithm.

[0157] The above-mentioned medical image generation device 600 based on dual-modal fusion can be implemented in the form of a computer program, and the computer program can run on a computer device as shown in Figure 8 shown.

[0158] Please refer to Figure 8 , Figure 8 which is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a terminal or a server. Among them, the terminal can be an electronic device with communication functions such as a desktop computer, a tablet computer, or a smart phone. The server can be an independent server or a server cluster composed of multiple servers.

[0159] Refer to Figure 8 , the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501. Among them, the memory can include a non-volatile storage medium 503 and an internal memory 504.

[0160] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can be made to execute a medical image generation method based on dual-modal fusion.

[0161] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0162] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can be made to execute a medical image generation method based on dual-modal fusion.

[0163] The network interface 505 is used for network communication with other devices. Those skilled in the art can understand that Figure 8 the structure shown in

[0164] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0165] It should be understood that in the embodiments of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, and the storage medium is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0167] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the steps of the above method.

[0168] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disc, etc., which are all computer-readable storage media that can store program codes.

[0169] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0170] In several embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0171] The steps in the method embodiments of the present invention can be adjusted, combined, and deleted according to actual needs. The units in the device embodiments of the present invention can be combined, divided, and deleted according to actual needs. In addition, the functional units in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0173] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A medical image generation method based on dual-modal fusion, characterized in that: include: Obtain medical images and corresponding text descriptions, and perform standardized preprocessing to obtain preprocessed images and text sequences; Extracting image features from the preprocessed image through an image encoder, extracting text features from the text sequence through a text encoder, mapping the image features and the text features to a shared semantic space through contrastive learning, and fusing them to obtain cross-modal features; Performing multi-scale feature extraction on the medical image by a dual-path encoder to obtain fused coding features; fusing the cross-modal features with the encoding features to obtain decoder input features, and generating a target image through a decoder according to the decoder input features; The target image is used to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the two-way encoder and the decoder.

2. The medical image generation method based on dual-modal fusion according to claim 1, characterized in that: The steps of extracting image features from the preprocessed image by using an image encoder, and extracting text features from the text sequence by using a text encoder include: Inputting the preprocessed image into the image encoder for multi-level feature extraction to obtain an image embedding vector; The text sequence is input into the text encoder for contextual semantic encoding to obtain a text embedding vector.

3. The medical image generation method based on dual-modal fusion according to claim 2, characterized in that: The step of mapping the image features and the text features to a shared semantic space through contrastive learning and fusing them to obtain cross-modal features comprises: Mapping the image embedding vector and the text embedding vector to a shared semantic space through a mapping network trained by contrastive learning; Normalizing the image embedding vector and the text embedding vector to obtain normalized image features and normalized text features; The normalized image features and the normalized text features are added and fused element by element to obtain cross-modal features.

4. The medical image generation method based on dual-modal fusion according to claim 3 is characterized in that: The step of extracting multi-scale features from the medical image by a dual-path encoder to obtain fused coding features comprises: Using the medical image as input to the dual-path encoder; Extracting image semantic features layer by layer using a convolutional neural network through a first encoding path of the dual-path encoder; The second encoding path of the dual-path encoder is down-sampled step by step through the maximum pooling layer to obtain hierarchical features; At each coding level, the image semantic features of the current level are added element by element to the level features to obtain a fused coding feature.

5. The medical image generation method based on dual-modality fusion according to claim 4 is characterized in that: The step of fusing the cross-modal feature with the encoding feature to obtain a decoder input feature comprises: Projecting the cross-modal features into a spatial feature map through a fully connected layer; The spatial feature map and the fused coding feature are channel-concatenated to obtain a decoder input feature.

6. The medical image generation method based on dual-modal fusion according to claim 5, characterized in that: The step of generating a target image through a decoder according to the decoder input feature comprises: Upsampling the decoder input features step by step through a transposed convolutional network to obtain a final upsampled image; The final up-sampled image is mapped to a target medical image through an image mapping back to grayscale convolutional layer.

7. The medical image generation method based on dual-modal fusion according to claim 6, characterized in that: The step of using the target image to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the two-way encoder and the decoder comprises: Constructing a joint loss function through the target image, wherein the joint loss function includes contrast loss and structural similarity loss, the contrast loss includes image mapping network loss and text mapping network loss, and the structural similarity loss includes brightness component loss, contrast component loss and structural component loss; Based on the joint loss function, the network parameters of the image encoder, the text encoder, the two-way encoder and the decoder are jointly optimized through a back propagation algorithm.

8. A medical image generation device based on dual-modality fusion, characterized in that: Used to execute the medical image generation method based on dual-modal fusion as described in any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to execute the steps of the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Images