Medical image generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a medical image generation method and device, electronic equipment and storage medium, belonging to the technical field of artificial intelligence. The method comprises: generating Gaussian noise randomly through a preset modal decoupling network, and converting the Gaussian noise into a target modal feature vector; inputting a source modal medical image and the target modal feature vector into a preset cross-condition network, performing multi-scale feature extraction on the source modal medical image, obtaining a plurality of scale source modal feature vectors, then performing feature fusion on each scale source modal feature vector and the target modal feature vector respectively, obtaining a plurality of fusion modal feature vectors, finally performing image generation on the plurality of fusion modal feature vectors to obtain a target modal medical image, and the modal of the target modal medical image is different from the modal of the source modal medical image. The embodiments of the present application can reduce the difficulty of obtaining different modal medical images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a medical image generation method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In related technologies, various modalities of medical images are acquired by using magnetic resonance imaging (MRI) technologies such as T1 weighted imaging, T2 weighted imaging, contrast post-T1 weighted imaging, and T2 fluid attenuated inversion recovery imaging. However, the acquisition of some modalities such as contrast post-T1 weighted imaging and T2 fluid attenuated inversion recovery imaging is difficult. How to reduce the acquisition difficulty of different modalities of medical images has become a problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a medical image generation method and device, an electronic device, and a storage medium, which aims to reduce the acquisition difficulty of different modalities of medical images.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a medical image generation method, which comprises:

[0005] acquiring a source modality medical image;

[0006] generating Gaussian noise randomly by using a preset modality decoupling network, and converting the Gaussian noise into a target modality feature vector;

[0007] inputting the source modality medical image and the target modality feature vector into a preset cross-condition network, the cross-condition network comprising a feature extraction subnetwork, a conditional embedding subnetwork, and an image generation subnetwork;

[0008] performing multi-scale feature extraction on the source modality medical image by using the feature extraction subnetwork to obtain source modality feature vectors of multiple scales;

[0009] performing feature fusion on the source modality feature vectors of each scale and the target modality feature vector respectively by using the conditional embedding subnetwork to obtain multiple fusion modality feature vectors;

[0010] performing image generation on the multiple fusion modality feature vectors by using the image generation subnetwork to obtain a target modality medical image; the modality of the target modality medical image is different from the modality of the source modality medical image.

[0011] In some embodiments, the modality decoupling network is trained according to the following steps:

[0012] obtaining a first sample medical image;

[0013] extracting a modal feature of the first sample medical image by a preset modal representation model to obtain a sample modal feature vector;

[0014] performing noise addition processing on the sample modal feature vector to obtain a first modal feature vector;

[0015] performing noise reduction processing on the first modal feature vector by a preset denoising diffusion model to obtain a second modal feature vector;

[0016] calculating a noise loss according to the sample modal feature vector and the second modal feature vector to obtain noise loss data;

[0017] adjusting parameters of the denoising diffusion model according to the noise loss data to obtain the modal decoupling network.

[0018] In some embodiments, the noise addition processing on the sample modal feature vector to obtain the first modal feature vector comprises:

[0019] obtaining a noise addition strategy for each time step;

[0020] gradually adding noise to the sample modal feature vector according to the noise addition strategy for each time step to obtain the first modal feature vector.

[0021] In some embodiments, the modal representation model is trained according to the following steps:

[0022] obtaining a second sample medical image; the second sample medical image comprises an image sub-region;

[0023] randomly selecting an image sub-region from the second sample medical image to obtain a candidate sub-region;

[0024] obtaining a candidate medical image by occluding the candidate sub-region of the second sample medical image;

[0025] restoring an image by a preset modal model to obtain a restored medical image;

[0026] calculating a restoration loss according to the second sample medical image and the restored medical image to obtain image restoration loss data;

[0027] adjusting parameters of the preset modal model according to the image restoration loss data to obtain the modal representation model.

[0028] In some embodiments, the restoration loss calculation according to the second sample medical image and the restored medical image obtains image restoration loss data, including:

[0029] Obtaining the number of regions of the candidate sub-regions;

[0030] For each of the candidate sub-regions, difference calculation is performed on the candidate sub-region of the second sample medical image and the candidate sub-region of the restored medical image to obtain region difference data of the candidate sub-region;

[0031] The region difference data of all candidate sub-regions is integrated and calculated to obtain image difference data;

[0032] According to the image difference data and the number of regions, mean calculation is performed to obtain the image restoration loss data.

[0033] In some embodiments, the feature fusion of the source modality feature vector and the target modality feature vector of each scale is performed by the conditional embedding sub-network respectively to obtain a plurality of fusion modality feature vectors, including:

[0034] Obtaining a first feature dimension of the target modality feature vector and a second feature dimension of the source modality feature vector of each scale;

[0035] For each scale, feature conversion is performed on the target modality feature vector according to the first feature dimension and the second feature dimension to obtain a distribution representation feature vector;

[0036] For each scale, the distribution representation feature vector and the source modality feature vector are feature-aligned and added to obtain the fusion modality feature vector.

[0037] In some embodiments, the source modality medical image is obtained, including:

[0038] Obtaining an initial modality medical image;

[0039] Performing normalization processing on the initial modality medical image to obtain an intermediate modality medical image;

[0040] Performing region extraction on the intermediate modality medical image to obtain the source modality medical image.

[0041] To achieve the above-mentioned purposes, a second aspect of an embodiment of the present application proposes a medical image generation device, the device comprising:

[0042] An image acquisition module is configured to obtain a source modality medical image;

[0043] a modal feature vector generation module, configured to generate Gaussian noise randomly through a preset modal decoupling network, and convert the Gaussian noise into a target modal feature vector;

[0044] a modal feature vector input module, configured to input the source modal medical image and the target modal feature vector into a preset cross-condition network, the cross-condition network including a feature extraction subnetwork, a conditional embedding subnetwork, and an image generation subnetwork;

[0045] a modal feature vector extraction module, configured to perform multi-scale feature extraction on the source modal medical image through the feature extraction subnetwork to obtain source modal feature vectors at multiple scales;

[0046] a modal feature vector fusion module, configured to perform feature fusion on the source modal feature vector at each scale and the target modal feature vector respectively through the conditional embedding subnetwork to obtain multiple fusion modal feature vectors;

[0047] an image generation module, configured to perform image generation on the multiple fusion modal feature vectors through the image generation subnetwork to obtain a target modal medical image; the target modal medical image is of a different modality from the source modal medical image.

[0048] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0049] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0050] The medical image generation method and device, electronic device and storage medium provided by the present application generate Gaussian noise randomly through a preset modal decoupling network, convert the Gaussian noise into a target modal feature vector, perform feature fusion on a source modal medical image and the target modal feature vector through a cross-condition network, and finally generate a target modal medical image. This method effectively reduces the dependence on complex imaging conditions, enables efficient generation of medical images of high-difficulty modalities without multiple scans, and greatly reduces the difficulty of obtaining medical images of different modalities. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of the medical image generation method provided by the embodiments of the present application;

[0052] Figure 2 isFigure 1 the flowchart of step S101 in

[0053] Figure 3 is a training flowchart of the modal decoupling network provided by the embodiments of the present application;

[0054] Figure 4 is Figure 3 the flowchart of step S303 in

[0055] Figure 5 is a training flowchart of the modal representation model provided by the embodiments of the present application;

[0056] Figure 6 is Figure 5 the flowchart of step S505 in

[0057] Figure 7 is Figure 1 the flowchart of step S105 in

[0058] Figure 8 is a structural schematic diagram of the medical image generation device provided by the embodiments of the present application;

[0059] Figure 9 is a hardware structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0060] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0061] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0063] First, the terms involved in the present application are analyzed:

[0064] Artificial Intelligence (AI): A new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence; AI is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The field of research includes robots, language recognition, image recognition, natural language processing, and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also the theory, method, technology, and application system that uses digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve the best results.

[0065] Medical Images: Images obtained using various imaging techniques to obtain information about the internal structure and function of the human body, which play a crucial role in medical diagnosis, treatment planning, monitoring, and research. The types and generation methods of medical images include but are not limited to the following common imaging techniques: X-ray imaging, magnetic resonance imaging (MRI), computed tomography (CT), ultrasound imaging, and positron emission tomography (PET).

[0066] Magnetic Resonance Imaging (MRI): A technique that uses the nuclear magnetic resonance phenomenon of certain atomic nuclei in the human body to obtain radio frequency signals, which are processed by a computer to reconstruct images of a certain layer of the human body. MRI can directly produce cross-sectional, sagittal, coronal, and various oblique body layer images, without the artifacts produced in computed tomography detection, and without ionizing radiation, which has no adverse effects on the body. This technology can effectively assist doctors in diagnosing common brain diseases such as intracranial hematoma, extracranial hematoma, brain tumor, intracranial aneurysm, arteriovenous malformation, cerebral ischemia, intraspinal tumor, spinal cord cavity, and hydrocephalus.

[0067] In the related art, various modal medical images are acquired by using magnetic resonance imaging (MRI) technologies such as T1-weighted imaging (T1), T2-weighted imaging (T2), T1-weighted imaging with gadolinium enhancement (T1 Gd), and T2-Fluid-Attenuated Inversion Recovery (T2-FLAIR). Multi-modal magnetic resonance imaging is crucial for comprehensive analysis and diagnosis of diseases, and provides rich complementary information for analyzing brain tumors in a clinical environment. However, some modalities such as T1-weighted imaging with gadolinium enhancement and T2-Fluid-Attenuated Inversion Recovery require drug injection and multiple scans of the patient, and require high patient cooperation, so the acquisition of magnetic resonance images of these modalities is difficult. Due to factors such as imaging cost, imaging time limit, and safety impact, it is challenging to acquire medical images of multiple modalities. Therefore, how to reduce the difficulty of acquiring medical images of different modalities has become a problem to be solved.

[0068] Based on this, the embodiments of the present application provide a medical image generation method and device, an electronic device and a storage medium, aiming to reduce the difficulty of acquiring medical images of different modalities.

[0069] The medical image generation method and device, the electronic device and the storage medium provided by the embodiments of the present application are specifically explained by the following embodiments, and first, the medical image generation method in the embodiments of the present application is described.

[0070] The medical image generation method provided by the embodiments of the present application relates to the field of artificial intelligence. The medical image generation method provided by the embodiments of the present application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; and the software can be an application for implementing the medical image generation method, but is not limited to the above forms.

[0071] The application is operable in a variety of general purpose or special purpose computer systems environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0072] It should be noted that in each specific embodiment of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0073] Figure 1 is an optional flowchart of a medical image generation method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to including steps S101 to S106.

[0074] Step S101, acquiring a source modality medical image.

[0075] Step S102, randomly generating Gaussian noise through a preset modality decoupling network, and converting the Gaussian noise into a target modality feature vector.

[0076] Step S103, inputting the source modality medical image and the target modality feature vector into a preset cross-condition network.

[0077] Step S104, performing multi-scale feature extraction on the source modality medical image through a feature extraction sub-network to obtain a plurality of scales of source modality feature vectors.

[0078] Step S105, respectively, through conditional embedding sub-networks, the source modality feature vector and the target modality feature vector of each scale are fused to obtain a plurality of fusion modality feature vectors.

[0079] Step S106, through the image generation sub-network, the plurality of fusion modality feature vectors are image generated to obtain the target modality medical image.

[0080] The steps S101 to S106 shown in the embodiments of the present application, through the preset modality decoupling network, the Gaussian noise is randomly generated and converted into the target modality feature vector, and then the source modality medical image and the target modality feature vector are fused through the cross-condition network, and finally the target modality medical image is generated. This method effectively reduces the dependence on complex imaging conditions, so that the medical image of high difficulty modality can be efficiently generated without multiple scans or drug injections of the patient, thereby reducing the acquisition difficulty of different modality medical images.

[0081] In step S101 of some embodiments, in the embodiments of the present application, the source modality medical image can be T1-weighted imaging (T1-Weighted Imaging) and T2-weighted imaging (T2-Weighted Imaging) in nuclear resonance imaging, or other MRI modality images with low acquisition difficulty. It can also be other medical image types, such as CT images, X-ray images and PET images, and the modality of the image is determined to be low acquisition difficulty.

[0082] Specifically, please refer to Figure 2 In some embodiments, since the medical image acquisition instrument will introduce a pure black background area when shooting images, it is necessary to crop the background of the initial source modality medical image to extract effective image information, thereby improving the accuracy of subsequent processing. Step S101 can include but is not limited to steps S201 to S205:

[0083] Step S201, obtaining an initial modality medical image.

[0084] Step S202, normalizing the initial modality medical image to obtain an intermediate modality medical image.

[0085] Step S203, region extraction is performed on the intermediate modality medical image to obtain a source modality medical image.

[0086] In step S201 of some embodiments, the initial modality medical image is the medical image collected by the medical image acquisition instrument after shooting without processing, and the modality sequence can be T1-weighted imaging and T2-weighted imaging with low acquisition difficulty.

[0087] In step S202 of some embodiments, the pixel value of each pixel of the initial modality medical image is set to the interval [0, 1] or the interval [0, 255], and a normalized result, i.e., an intermediate modality medical image, is obtained.

[0088] In step S203 of some embodiments, the pixel value is determined as 1 (or 255) to represent that the current pixel color is black, a black background region is determined according to the pixel value, and then the intermediate modality medical image is cropped in the black background region to obtain a final source modality medical image.

[0089] The steps S201 to S203 shown in the embodiments of the present application can effectively eliminate background noise and improve the quality of the source modality image, thereby improving the accuracy of subsequent image processing and analysis.

[0090] Before step S102 of some embodiments, the medical image generation method further includes pre-training a modality decoupling network, which is used to generate. Specifically, please refer to Figure 3 In some embodiments, the modality decoupling network complies with the training process of Denoising Diffusion Implicit Models (DDIM), which includes a forward noise-increasing diffusion sub-model and a reverse noise-reducing diffusion sub-model. The training process of the modality decoupling network can include but is not limited to steps S301 to S306:

[0091] Step S301, obtaining a first sample medical image.

[0092] Step S302, performing modality feature extraction on the first sample medical image through a preset modality representation model to obtain a sample modality feature vector.

[0093] Step S303, performing noise-increasing processing on the sample modality feature vector to obtain a first modality feature vector.

[0094] Step S304, performing noise-reducing processing on the first modality feature vector through a preset denoising diffusion model to obtain a second modality feature vector.

[0095] Step S305, performing noise loss calculation according to the sample modality feature vector and the second modality feature vector to obtain noise loss data.

[0096] Step S306, adjusting the parameters of the denoising diffusion model according to the noise loss data to obtain a modality decoupling network.

[0097] In step S301 of some embodiments, the first sample medical image includes medical images of multiple modalities, in addition to the above-mentioned T1 weighted imaging and T2 weighted imaging, including post-contrast T1 weighted and T2 liquid attenuation inversion recovery imaging modalities, etc.

[0098] In step S302 of some embodiments, the modal representation model is composed of a feature encoder and a feature decoder, for learning the modal features of the first sample medical image. In this embodiment, the sample modal feature vector is the noise distribution feature of the first sample medical image. It should be noted that if the first sample medical image includes multiple modalities to be input, each modality is spliced in the channel dimension to achieve one-time input.

[0099] In step S303 of some embodiments, please refer to Figure 4 In some embodiments, step S303 can include but is not limited to steps S401 to S402:

[0100] Step S401, obtaining the noise adding strategy of each time step.

[0101] Step S402, step-by-step noise adding to the sample modal feature vector according to the noise adding strategy of each time step, to obtain the first modal feature vector.

[0102] In step S401 of some embodiments, the noise adding strategy refers to the specific way of step-by-step adding noise to the sample modal feature vector. Different noise adding strategies determine how to add noise to the input data at each time step, and the variation law of the intensity of the noise over time.

[0103] In this embodiment, the noise adding strategy is cosine strategy (Cosine), which can also be linear strategy (Linear), quadratic strategy (Quadratic) and logarithmic strategy (Logarithmic). The operator can make corresponding adjustments according to the actual model requirements, which are not limited to this.

[0104] In step S402 of some embodiments, the sample modal feature vector is input through the feature encoder, and the sample modal feature vector and the standard Gaussian noise are step-by-step mixed and added at multiple consecutive time steps, to finally obtain the noise distribution feature after noise adding, which is the first modal feature vector. This process can be represented by the standard noise function in the diffusion model, such as the analytical expression (1):

[0105]

[0106] Wherein, the current time step is represented as t, the first sample image is represented as y0, and the first sample image after adding t-step Gaussian noise, i.e. the first modal feature vector, is represented as y​​​​​​​​​​​t The noise addition strategy coefficient is represented as The standard Gaussian noise is represented as ∈. It can be understood that the first modal feature vector is subject to a normal distribution.

[0107] The steps S401 to S402 shown in the embodiments of the present application enhance the generalization ability of the model by means of step-by-step noise addition, and provide reliable data for subsequent training of the modal decoupling network.

[0108] In step S304 of some embodiments, the denoising diffusion model, i.e., the reverse denoising diffusion submodel part of the denoising diffusion implicit model, specifically, the denoising diffusion model can be a Mixture Density Network (MDN) including multiple residual blocks, and each residual block has an input layer, a time step embedding layer, and an output layer, each layer including layer normalization, an activation function, and a linear layer. The first modal feature vector is denoised by the multiple residual blocks, and the first modal feature vector is restored to obtain a restored noise distribution feature, and the noise distribution feature is the second modal feature vector. The above process can be represented by equation (2):

[0109]

[0110] where T represents the total number of time steps, and t represents the current time step. The second modal feature vector is represented as y r The first modal feature vector is represented as y T During the reverse generation process, the intermediate modal feature vector at the current time step t is represented as y t The preset MDN model parameter is represented as θ.

[0111] In step S305 of some embodiments, the noise loss data can be calculated according to equation (3):

[0112]

[0113] where y0 represents the sample modal feature vector, and y t It can be understood that the similarity between the restored noise distribution feature and the sample modal feature vector is calculated by the loss function.

[0114] In step S306 of some embodiments, the preset MDN model parameter θ of the denoising diffusion model or other related model parameters are adjusted, and the noise loss data is optimized. The model parameters are constantly adjusted until the similarity is greater than or equal to a similarity threshold, the optimization of the denoising diffusion model is stopped, and a modal decoupling network meeting the requirements is obtained.

[0115] The steps S301 to S306 shown in the embodiments of the present application optimize the parameters of the denoising diffusion model by calculating the noise loss data, to ensure that the generated modal decoupling network can effectively separate and reconstruct the features of different modalities.

[0116] Before step S302 of some embodiments, the medical image generation method further includes pre-training a modal representation model, please refer to Figure 5 In some embodiments, the training process of the modal representation model can further include but is not limited to steps S501 to S506:

[0117] Step S501, obtaining a second sample medical image, wherein the second sample medical image includes an image sub-region.

[0118] Step S502, randomly selecting an image sub-region from the second sample medical image to obtain a candidate sub-region.

[0119] Step S503, occluding the candidate sub-region of the second sample medical image to obtain a candidate medical image.

[0120] Step S504, performing image restoration on the candidate medical image by a preset modal model to obtain a restored medical image.

[0121] Step S505, performing restoration loss calculation according to the second sample medical image and the restored medical image to obtain image restoration loss data.

[0122] Step S506, adjusting the parameters of the preset modal model according to the image restoration loss data to obtain a modal representation model.

[0123] In step S501 of some embodiments, the medical image obtained by region division on the modal image of the first sample medical image is the second sample medical image. In this embodiment, the second sample medical image is evenly divided into a 6x6 rectangular grid, and each grid corresponds to an image sub-region.

[0124] In step S502 of some embodiments, in order to increase the diversity of input data to better train the model, some sub-regions need to be randomly selected from the divided image for further processing or analysis. These randomly selected sub-regions are called candidate sub-regions.

[0125] In step S503 of some embodiments, the pixel values corresponding to the candidate sub-region are set to 0, thereby realizing occlusion, and the obtained image is a candidate medical image.

[0126] In step S504 of some embodiments, the preset modal model is an initial modal representation model composed of a feature encoder and a feature decoder, and is used to learn the modal features of the second sample medical image. The preset modal model attempts to infer and reconstruct the image content of the occluded area according to the information of the remaining unoccluded area of the candidate medical image, so as to obtain a recovered version image as close as possible to the first sample image, that is, the recovered medical image.

[0127] In step S505 of some embodiments, referring to Figure 6 In some embodiments, step S505 includes but is not limited to steps S601 to S604:

[0128] In step S601, the number of regions of the candidate sub-regions is obtained.

[0129] In step S602, for each candidate sub-region, the difference between the candidate sub-region of the second sample medical image and the candidate sub-region of the recovered medical image is calculated to obtain the regional difference data of the candidate sub-region.

[0130] In step S603, the regional difference data of all candidate sub-regions is integrated and calculated to obtain the image difference data.

[0131] In step S604, the image difference data and the number of regions are averaged to obtain the image recovery loss data.

[0132] In step S601 of some embodiments, the number of regions is the number of occluded region blocks in the candidate medical image.

[0133] In step S602 of some embodiments, for each occluded candidate sub-region, the difference between the true value and the recovered value is calculated. Specifically, for each occluded sub-region, the actual pixel value in the second sample medical image is compared with the predicted pixel value in the recovered medical image, and the difference value between the two is calculated using L2 norm, which is the regional difference data.

[0134] In step S603 of some embodiments, all regional difference data is accumulated to obtain the image difference data.

[0135] In step S604 of some embodiments, the image difference data is divided by the number of regions to obtain the image recovery loss data. In summary, the image recovery loss data can be represented by the loss function formula (4):

[0136]

[0137] Wherein, R represents a set of candidate sub-regions, | R |wherein r denotes the number of regions, r denotes the current candidate sub-region, and pr denotes the real pixel value of the current candidate sub-region that is occluded in the candidate medical image, wherein pr denotes the predicted pixel value of the corresponding region in the recovered medical image. The objective of this loss function is to enable the modality representation model to predict the complete real image from the input image that is locally masked, so as to learn the contextual representation of the multi-modality medical image.

[0138] The steps S601 to S604 shown in the embodiments of the present application can accurately measure the difference between the recovered medical image and the original image by obtaining the number of candidate sub-regions and performing difference calculation on these regions, so as to ensure that the model can gradually optimize its prediction ability for the occluded region, thereby more effectively restoring the integrity of the medical image in actual application.

[0139] In step S506 of some embodiments, the related model parameters of the preset modality model are adjusted according to the image recovery loss data, and the image recovery loss data is optimized. The model parameters are constantly adjusted until the image recovery loss data is less than or equal to a certain preset threshold, or no longer changes, the optimization of the preset modality model is stopped, and the modality representation model that meets the requirements is obtained.

[0140] The steps S501 to S506 shown in the embodiments of the present application, from obtaining the sample medical image to optimizing the preset modality model, finally obtaining the modality representation model, realize efficient image recovery and multi-modality representation learning, effectively improve the performance of the modality representation model in processing complex multi-modality medical images, and enhance the practicality and reliability of the modality representation model in actual application.

[0141] In step S102 of some embodiments, the modality decoupling network pre-trained is used to randomly sample a Gaussian noise feature vector from a standard Gaussian distribution, and a noise distribution feature used for representing other modality medical images is generated by gradually reducing the noise, and the feature is the target modality feature vector. For example, in the embodiments of the present application, the modality medical images that are difficult to obtain are the scene of T1 weighted imaging and T2 weighted imaging that are relatively easy to obtain, and the target modality images are contrasted T1 weighted imaging and T2 liquid attenuation inversion recovery imaging that are difficult to obtain. Since the modality decoupling network has learned the features of these target modalities in the training process, the target modality feature vector at this time is the noise distribution feature of the contrasted T1 weighted imaging and T2 liquid attenuation inversion recovery imaging.

[0142] In step S103 of some embodiments, the cross-condition network comprises a feature extraction subnetwork, a conditional embedding subnetwork, and an image generation subnetwork. It should be noted that the cross-condition network can be a U-shaped convolutional network, in which the feature extraction subnetwork can be an encoder comprising convolutional blocks for extracting features of the source modality medical images. The conditional embedding subnetwork can be a multi-layer perceptron (MLP) layer consisting of two linear layers and an activation layer. The image generation subnetwork can be a decoder comprising convolutional blocks for generating or reconstructing images.

[0143] In step S104 of some embodiments, the feature extraction subnetwork down-samples the source modality medical images to obtain source modality feature vectors of different sizes, which can be (512, 32, 32), (256, 64, 64), and (128, 128, 128), etc.

[0144] Referring to Figure 7 In some embodiments, step S105 can include, but is not limited to, steps S701 to S703:

[0145] In step S701, a first feature dimension of the target modality feature vector and a second feature dimension of the source modality feature vector of each scale are obtained.

[0146] In step S702, for each scale, the target modality feature vector is converted according to the first feature dimension and the second feature dimension to obtain a distribution representation feature vector.

[0147] In step S703, for each scale, the distribution representation feature vector and the source modality feature vector are feature-aligned and added to obtain a fusion modality feature vector.

[0148] In step S701 of some embodiments, the first feature dimension is the number of channels of the target modality feature vector, and the second feature dimension is the number of channels of the source modality feature vector of each size. Following the example of step S104, the first feature dimension can be 768, and the second feature dimension can be 512, 256, and 128, respectively.

[0149] In step S702 of some embodiments, the target modality feature vector is convolved, fully connected, or other linearly transformed by the conditional embedding subnetwork, so that the number of channels of the target modality feature vector matches the number of channels of the source modality feature vector of the corresponding scale, that is, the first feature dimension is adjusted to be equal to the second feature dimension, and the converted feature vector is the distribution representation feature vector. Exemplarily, the first feature dimension of the target modality feature vector is converted from 768 to three distribution representation feature vectors with channel numbers of 512, 256, and 128, respectively.

[0150] In step S703 of some embodiments, the distribution representation feature vector is added to the source modality feature vector of the same channel number, and the calculation result is the fusion feature vector, so as to realize the merging of the target modality feature and the source modality feature vector of each scale.

[0151] The steps S701 to S703 shown in the embodiments of the present application adjust and fuse the feature dimensions of the target modality feature vector and the source modality feature vector, so as to effectively integrate different modality information, constrain the generation direction of the target modality, and further ensure the generation of a more controllable and reliable target modality.

[0152] In step S106 of some embodiments, the fusion modality feature vector is up-sampled to generate a final target modality medical image. It should be noted that the modality of the target modality medical image is different from that of the source modality medical image, and in the present embodiment, the target modality image is contrasted T1 weighted imaging and T2 liquid attenuation inversion recovery imaging. It can be understood that the target modality image can also be one modality image or multiple modality images, and the number and type of the generated modalities can be adjusted according to actual application requirements.

[0153] It can be understood that the cross-condition network needs to be trained in advance according to the mean square error loss function, the target modality feature vector and the source modality medical image are input into the cross-condition network, and finally a predicted target modality image is output. The above loss function can be represented by equation (5):

[0154]

[0155] Wherein, the loss value is represented as L Syn , the predicted target modality image is represented as The real target modality image is represented as Y.

[0156] In addition, the final generated target modality medical image is introduced into the above sample image set used for training, so as to iteratively update the modality representation model, the modality decoupling network and the cross-condition network, and further improve the generation quality of the medical image.

[0157] It should be noted that in the related art, there is a technical means of generating a magnetic resonance image by using a generative adversarial network or a denoising diffusion model. However, due to the difference between natural images and medical images, the generative adversarial network is difficult to be directly applied to the generation of medical images, and the training process is not stable, resulting in that the quality of the finally generated image cannot be guaranteed. The method using the denoising diffusion model needs to sample multiple times in the generation process of the image, thereby increasing the operation time. Compared with directly using the denoising diffusion model, the medical image generation method proposed in the embodiments of the present application only needs to randomly sample the noise feature once in the generation process of the target modality image each time, and the generation efficiency of the method is higher. In addition, the model is trained according to the existing medical image, so that the model mentioned in the method has higher adaptability and stability in the generation of medical images.

[0158] Please refer to Figure 8 The embodiments of the present application also provide a medical image generation device, which can implement the above medical image generation method, and the device comprises:

[0159] An image acquisition module is configured to acquire a source modality medical image.

[0160] A modality feature vector generation module is configured to randomly generate Gaussian noise through a preset modality decoupling network, and convert the Gaussian noise into a target modality feature vector.

[0161] A modality feature vector input module is configured to input the source modality medical image and the target modality feature vector into a preset cross-condition network, and the cross-condition network comprises a feature extraction subnetwork, a condition embedding subnetwork and an image generation subnetwork.

[0162] A modality feature vector extraction module is configured to perform multi-scale feature extraction on the source modality medical image through the feature extraction subnetwork, and obtain a plurality of scale source modality feature vectors.

[0163] A modality feature vector fusion module is configured to perform feature fusion on each scale source modality feature vector and the target modality feature vector through the condition embedding subnetwork, and obtain a plurality of fusion modality feature vectors.

[0164] An image generation module is configured to perform image generation on the plurality of fusion modality feature vectors through the image generation subnetwork, and obtain a target modality medical image. The modality of the target modality medical image is different from the modality of the source modality medical image.

[0165] The specific implementation of the medical image generation device is basically the same as that of the above medical image generation method, and will not be repeated here.

[0166] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the medical image generation method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0167] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0168] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0169] The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the medical image generation method of the embodiments of the present application.

[0170] The input / output interface 903 is used to realize information input and output.

[0171] The communication interface 904 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, a WI-FI, Bluetooth, etc.).

[0172] The bus 905 is used to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0173] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize communication connection between them in the device.

[0174] The embodiment of the present application further provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the medical image generation method.

[0175] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The medical image generation method, the medical image generation device, the electronic equipment and the storage medium provided by the embodiment of the present application can effectively reduce the dependence on complex imaging conditions, so that the medical image of a high-difficulty modality can be efficiently generated without multiple scans, thereby greatly reducing the difficulty of obtaining medical images of different modalities.

[0177] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0178] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0179] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0180] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0181] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but is used to connect like elements or to distinguish one claim from another. These terms can be used interchangeably when appropriate. Terms concerning the relative position of elements can be interpreted such that their use adheres to their normal meaning, but they can also be interpreted to mean the opposite according to specific claims.

[0182] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In several embodiments provided by the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0184] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the application.

[0185] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0186] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0187] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for generating medical images, characterized in that, include: Acquire source modal medical images; Gaussian noise is randomly generated through a preset modal decoupling network, and the Gaussian noise is converted into a target modal feature vector. The source modality medical image and the target modality feature vector are input into a preset cross-conditional network, which includes a feature extraction sub-network, a conditional embedding sub-network, and an image generation sub-network. The source modality medical image is subjected to multi-scale feature extraction through the feature extraction sub-network to obtain source modality feature vectors at multiple scales; The conditional embedding subnetwork is used to fuse the source modality feature vector and the target modality feature vector at each scale to obtain multiple fused modality feature vectors. Specifically, this includes: obtaining the first feature dimension of the target modality feature vector and the second feature dimension of the source modality feature vector at each scale; for each scale, performing feature transformation on the target modality feature vector based on the first feature dimension and the second feature dimension to obtain a distribution representation feature vector; and for each scale, adding the distribution representation feature vector and the source modality feature vector by feature alignment to obtain the fused modality feature vector. The image generation sub-network generates an image from multiple fused modal feature vectors to obtain a target modal medical image; the modality of the target modal medical image is different from the modality of the source modal medical image. The modal decoupling network is trained according to the following steps: A first sample medical image is acquired; modal features are extracted from the first sample medical image using a preset modal representation model to obtain a sample modal feature vector; the sample modal feature vector is denoised to obtain a first modal feature vector; the first modal feature vector is denoised using a preset denoising diffusion model to obtain a second modal feature vector; noise loss is calculated based on the sample modal feature vector and the second modal feature vector to obtain noise loss data; the parameters of the denoising diffusion model are adjusted based on the noise loss data to obtain the modal decoupling network.

2. The method according to claim 1, characterized in that, The step of performing noise enhancement on the sample modal feature vector to obtain the first modal feature vector includes: Obtain the noise addition strategy for each time step; The sample modal feature vector is gradually noise-added according to the noise-adding strategy at each time step to obtain the first modal feature vector.

3. The method according to claim 1, characterized in that, The modal representation model is trained according to the following steps: Acquire a second sample medical image; the second sample medical image includes an image sub-region; Candidate sub-regions are obtained by randomly selecting image sub-regions from the second sample medical images; The candidate sub-region of the second sample medical image is occluded to obtain a candidate medical image; The candidate medical images are restored using a preset modal model to obtain the restored medical images; Based on the second sample medical image and the restored medical image, the restoration loss is calculated to obtain image restoration loss data; The parameters of the preset modal model are adjusted based on the image restoration loss data to obtain the modal representation model.

4. The method according to claim 3, characterized in that, The step of calculating the restoration loss based on the second sample medical image and the restored medical image to obtain image restoration loss data includes: Obtain the number of regions in the candidate sub-regions; For each candidate sub-region, the difference between the candidate sub-region of the second sample medical image and the candidate sub-region of the restored medical image is calculated to obtain the regional difference data of the candidate sub-region; The regional difference data of all candidate sub-regions are integrated and calculated to obtain image difference data; The image restoration loss data is obtained by averaging the image difference data and the number of regions.

5. The medical image generation method according to any one of claims 1 to 4, characterized in that, The acquisition of source modal medical images includes: Acquire initial modal medical images; The initial modal medical image is normalized to obtain the intermediate modal medical image; Region extraction is performed on the intermediate modality medical image to obtain the source modality medical image.

6. A medical image generation device, characterized in that, The device includes: The image acquisition module is used to acquire source modal medical images; The modal feature vector generation module is used to randomly generate Gaussian noise through a preset modal decoupling network and convert the Gaussian noise into a target modal feature vector. The modal feature vector input module is used to input the source modal medical image and the target modal feature vector into a preset cross-conditional network, wherein the cross-conditional network includes a feature extraction sub-network, a conditional embedding sub-network, and an image generation sub-network; The modal feature vector extraction module is used to extract multi-scale features from the source modal medical image through the feature extraction sub-network to obtain source modal feature vectors at multiple scales. The modal feature vector fusion module is used to fuse the source modal feature vector and the target modal feature vector at each scale through the conditional embedding subnetwork to obtain multiple fused modal feature vectors. The image generation module generates an image from multiple fused modal feature vectors through the image generation sub-network to obtain a target modal medical image; the modality of the target modal medical image is different from the modality of the source modal medical image; The medical image generation device is further configured to: obtain a first feature dimension of the target modality feature vector and a second feature dimension of the source modality feature vector at each scale; for each scale, perform feature transformation on the target modality feature vector according to the first feature dimension and the second feature dimension to obtain a distribution representation feature vector; for each scale, perform feature alignment summation on the distribution representation feature vector and the source modality feature vector to obtain the fused modality feature vector; The medical image generation device is further configured to train the modal decoupling network according to the following steps: acquiring a first sample medical image; extracting modal features from the first sample medical image using a preset modal representation model to obtain a sample modal feature vector; performing noise enhancement on the sample modal feature vector to obtain a first modal feature vector; performing noise reduction on the first modal feature vector using a preset denoising diffusion model to obtain a second modal feature vector; calculating noise loss based on the sample modal feature vector and the second modal feature vector to obtain noise loss data; and adjusting the parameters of the denoising diffusion model based on the noise loss data to obtain the modal decoupling network.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the medical image generation method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the medical image generation method according to any one of claims 1 to 5.

Images