Medical image modal conversion method

Abstract

The embodiment of the invention discloses a medical image modal conversion method. The method comprises the following steps: acquiring a first target medical image collected for a target part of a target object in a first mode, a second mode to which the first target medical image is to be converted, and a pre-trained target generator; obtaining a predetermined second modal feature representing a second modal; and inputting the first target medical image and the second modal feature into a target generator, converting the first target medical image into a second modal by using the target generator and taking the second modal feature as priori knowledge, and obtaining and outputting a second target medical image. According to the technical scheme provided by the embodiment of the invention, the modal feature capable of indicating the medical image modal to which the target medical image needing to be generated belongs is input into the target generator, so that one target generator can be used for carrying out conversion of various modalities related to the medical image.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of medical image processing, and in particular to a medical image modality conversion method. Background Art

[0002] The conversion between different modalities of medical images is of great significance. Medical images of different modalities (such as magnetic resonance imaging (MRI), computed tomography (CT) and ultrasound) can provide different types of information, so integrating this information through modality conversion can assist doctors in making more comprehensive diagnoses.

[0003] At present, when it is impossible to collect medical images of the same object in different modalities, the task of converting from one modality to another is mainly completed based on the recurrent generative adversarial network. However, when using the recurrent generative adversarial network for modality conversion, there are at least two technical problems:

[0004] First, the cyclic generative adversarial network contains two generators, one to realize the conversion from modality 1 to modality 2, and the other to realize the conversion from modality 2 to modality 1. This makes the cyclic generative adversarial network only able to complete the conversion of two modalities, resulting in great limitations in modality conversion. Second, the cyclic generative adversarial network needs to construct two generators and two discriminators to separately complete the modality conversion task, which will increase the computational consumption exponentially. Summary of the invention

[0005] The embodiment of the present invention provides a medical image modality conversion method, which solves the problems of large limitations and high computational consumption of modality conversion.

[0006] According to one aspect of the present invention, a medical image modality conversion method is provided, which may include:

[0007] Acquire a first target medical image acquired in a first modality for a target part of a target object, a second modality to which the first target medical image is to be converted, and a pre-trained target generator;

[0008] Acquire a predetermined second modal feature representing a second modality;

[0009] The first target medical image and the second modality feature are input into the target generator, and the target generator is used to convert the first target medical image into the second modality using the second modality feature as prior knowledge to obtain and output the second target medical image.

[0010] The technical solution in the embodiment of the present invention is to obtain a first target medical image collected for a target part of a target object under a first modality, a second modality to which the first target medical image is to be converted, and a pre-trained target generator; obtain a predetermined second modality feature representing the second modality, the second modality feature can be used to indicate that a second target medical image under the second modality needs to be generated; input the first target medical image and the second modality feature into the target generator, so that the target generator can be used to convert the first target medical image to the second modality with the second modality feature as prior knowledge, and obtain and output the second target medical image. The above technical solution, by inputting the modality feature that can indicate the modality of the medical image to which the target medical image to be generated belongs into the target generator, enables one target generator to perform multiple modality conversions of related medical images, thereby solving the problem of large limitations and high computational consumption of modality conversion caused by using two target generators to perform two modality conversions.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 is a flow chart of a medical image modality conversion method provided according to an embodiment of the present invention;

[0014] Figure 2 is a flow chart of another medical image modality conversion method provided according to an embodiment of the present invention;

[0015] Figure 3 is a flowchart of another medical image modality conversion method provided according to an embodiment of the present invention;

[0016] Figure 4 is a flowchart of another medical image modality conversion method provided according to an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of a modality feature extraction process in another medical image modality conversion method provided according to an embodiment of the present invention;

[0018] Figure 6is a schematic diagram of a generator training process in yet another medical image modality conversion method provided according to an embodiment of the present invention;

[0019] Figure 7 is a schematic diagram of a network structure in another medical image modality conversion method provided according to an embodiment of the present invention;

[0020] Figure 8 is a structural block diagram of a medical image modality conversion device provided according to an embodiment of the present invention;

[0021] Fig. 9 It is a schematic diagram of the structure of an electronic device for implementing the medical image modality conversion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The situations of "target", "original", etc. are similar and will not be repeated here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] Figure 1 : is a flow chart of a medical image modality conversion method provided in an embodiment of the present invention. This embodiment can be applied to the case of performing modality conversion of medical images, and in particular, can be applied to the case of performing multiple modality conversions of medical images using one generator. The method can be executed by a medical image modality conversion device provided in an embodiment of the present invention, and the device can be implemented by software and / or hardware, and the device can be integrated in an electronic device, and the electronic device can be various user terminals or servers.

[0025] See also Figure 1The method of the embodiment of the present invention specifically includes the following steps:

[0026] S110, acquiring a first target medical image acquired in a first modality for a target part of a target object, a second modality to which the first target medical image is to be converted, and a pre-trained target generator.

[0027] The target object may be understood as an object for which medical image acquisition is to be performed in the first modality.

[0028] The target part can be understood as the part of the target object where medical image acquisition under the first modality is to be performed, for example, the head, chest or leg, etc. This depends on the actual situation and is not specifically limited here.

[0029] The first modality can be understood as a certain medical image modality, for example, an ultrasound image modality, an MRI image modality, or a CT image modality, etc. This depends on the actual situation and is not specifically limited here.

[0030] The first target medical image can be understood as a medical image acquired for the target part under the first modality. In combination with the above examples, it can be, for example, an ultrasonic image, an MRI image, or a CT image, etc. This is related to the actual situation and is not specifically limited here. Obtain the first target medical image. On this basis, in combination with the application scenarios that may be involved in the embodiments of the present invention, optionally, after acquiring the first target medical image, the first target medical image can be directly applied, or the first target medical image can be processed first, such as performing 2D image extraction processing and / or data enhancement processing, and then the processing result is used as the first target medical image for application; etc., which are not specifically limited here.

[0031] The second modality may be understood as a medical image modality to which the first target medical image is to be converted, and the medical image modality is different from the first modality. The second modality is acquired.

[0032] The target generator can be understood as a pre-trained generator that can be used to implement at least a modal conversion from the first modality to the second modality. On this basis, optionally, the target generator can also be used to implement at least one modal conversion from the second modality to the first modality, from the first modality to the third modality, from the third modality to the first modality, from the second modality to the third modality, and from the third modality to the second modality, etc. This is related to the actual situation and is not specifically limited here. Get the target generator.

[0033] S120: Obtain a predetermined second modal feature representing a second modality.

[0034] The second modality feature can be understood as a predetermined feature that can represent the second modality. In combination with the application scenarios that may be involved in the embodiments of the present invention, it is optional to predetermine the modality features corresponding to different medical image modalities, so that the modality features can be used as prior knowledge to enable the target generator to confirm the medical image modality to which the target medical image to be generated belongs.

[0035] Obtaining the second modal feature. In the embodiment of the present invention, optionally, the number of the second modal features can be one, two, or more, which depends on the actual setting and is not specifically limited here.

[0036] S130. Input the first target medical image and the second modality feature into a target generator, and utilize the target generator to convert the first target medical image into the second modality using the second modality feature as prior knowledge, to obtain and output the second target medical image.

[0037] Among them, the first target medical image and the second modal features are input into the target generator, so that the target generator can use the second modal features as prior knowledge to perform modal conversion on the first target medical image, obtain a second target medical image of the first target medical image under the second modality, and output the second target medical image, thereby completing the modal conversion of the first target medical image.

[0038] The technical solution in the embodiment of the present invention is to obtain a first target medical image collected for a target part of a target object under a first modality, a second modality to which the first target medical image is to be converted, and a pre-trained target generator; obtain a predetermined second modality feature representing the second modality, the second modality feature can be used to indicate that a second target medical image under the second modality needs to be generated; input the first target medical image and the second modality feature into the target generator, so that the target generator can be used to convert the first target medical image to the second modality with the second modality feature as prior knowledge, and obtain and output the second target medical image. The above technical solution, by inputting the modality feature that can indicate the modality of the medical image to which the target medical image to be generated belongs into the target generator, enables one target generator to perform multiple modality conversions of related medical images, thereby solving the problem of large limitations and high computational consumption of modality conversion caused by using two target generators to perform two modality conversions.

[0039] Figure 2It is a flowchart of another medical image modality conversion method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the target generator includes a target encoder and a target decoder; inputting the first target medical image and the second modality feature into the target generator, so as to use the target generator and the second modality feature as prior knowledge to convert the first target medical image to the second modality, and obtain and output the second target medical image, which may include: inputting the first target medical image and the second modality feature into the target encoder, so as to use the target encoder and the second modality feature as prior knowledge to perform feature encoding on the first target medical image, and obtain the medical image feature; inputting the medical image feature into the target decoder, so as to use the target decoder to perform feature decoding on the medical image feature, and obtain and output the second target medical image of the first target medical image in the second modality. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0040] See also Figure 2 The method of this embodiment may specifically include the following steps:

[0041] S210, acquiring a first target medical image acquired in a first modality for a target part of a target object, a second modality to which the first target medical image is to be converted, and a pre-trained target generator;

[0042] Among them, the target generator includes a target encoder and a target decoder.

[0043] Among them, the target encoder can be understood as a pre-trained encoder, and similarly, the target decoder can be understood as a pre-trained decoder. In an embodiment of the present invention, optionally, the target encoder and the target decoder can be an encoder and a decoder in the network architecture of U-Net.

[0044] S220: Obtain a predetermined second modal feature representing a second modality.

[0045] S230. Input the first target medical image and the second modality feature into a target encoder, and utilize the target encoder to perform feature encoding on the first target medical image using the second modality feature as prior knowledge to obtain a medical image feature.

[0046] Among them, the first target medical image and the second modality features are input into the target encoder, so that the target encoder can be used to perform feature encoding of the first target medical image with the second modality features as prior knowledge to obtain medical image features, especially medical image features that can represent the first target medical image and the second modality.

[0047] S240: Input the medical image features into a target decoder, and use the target decoder to perform feature decoding on the medical image features, thereby obtaining and outputting a second target medical image of the first target medical image in a second modality.

[0048] The medical image features are input into the target decoder, so that the target decoder can be used to perform feature decoding on the medical image features to obtain and output a second target medical image.

[0049] The technical solution of the embodiment of the present invention is to feature encode the first target medical image by taking the second modality features as prior knowledge, and then feature decode the medical image features obtained by the feature encoding, thereby achieving effective conversion of the first target medical image in the first modality to the second modality.

[0050] An optional technical solution, optionally, the target encoder may include multiple target encoding layers and multiple target downsampling layers, and the second modal feature is applied in the following manner:

[0051] During the feature encoding process, the first intermediate feature output by the target downsampling layer corresponding to the second modal feature in the multiple target downsampling layers is spliced ​​with the second modal feature in the channel dimension, and the spliced ​​result is input into the target encoding layer connected to the corresponding target downsampling layer in the multiple target encoding layers.

[0052] Among them, the target encoder may include multiple target coding layers and multiple target downsampling layers. In the present technical solution, optionally, the layers in the target encoder may be connected in the following manner: some target coding layers connected in sequence-target downsampling layers-some target encoders connected in sequence-target downsampling layers-some target encoders connected in sequence, and so on, which are not specifically limited here.

[0053] The first intermediate feature can be understood as a feature output by a target downsampling layer X corresponding to the second modal feature in a plurality of target downsampling layers during the feature encoding process, and the target downsampling layer X can be obtained by presetting. The first intermediate feature is concatenated with the second modal feature in the channel dimension, and then the concatenated result is input into a target coding layer connected to the target downsampling layer X in a plurality of target coding layers, so as to apply the prior knowledge represented by the second modal feature in the feature encoding process.

[0054] On this basis, it should be noted that when the number of second modal features is two or more, each second modal feature can be processed separately based on the above steps, so as to fully apply the prior knowledge represented by different second modal features.

[0055] The above technical solution can fully apply the prior knowledge represented by the second modal features in the feature encoding process, thereby improving the accuracy of modal conversion.

[0056] Figure 3 It is a flowchart of another medical image modality conversion method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the target generator is pre-trained in the following manner: a first sample medical image and a second modality feature collected from a sample part of a sample object under a first modality are obtained, wherein the sample part and the target part are the same part; the first sample medical image and the second modality feature are input into the generator to be trained, so as to use the generator to be trained and use the second modality feature as prior knowledge to perform modality conversion of the first sample medical image to obtain a second converted medical image; the second converted medical image and the second modality feature are input into the discriminator to be trained, and the first predicted label output by the discriminator to be trained and the preset first true label are calculated for loss, and the parameters in the generator to be trained are updated according to the loss calculation result to obtain the target generator; wherein the first true label represents that the second converted medical image is a medical image under the second modality. The explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0057] See also Figure 3 The method of this embodiment may specifically include the following steps:

[0058] S310: Acquire a first sample medical image acquired from a sample part of a sample object under a first modality, and acquire a predetermined second modality feature representing a second modality.

[0059] Among them, the samples in the embodiments of the present invention and the targets described above are relative concepts. Specifically, the samples are used in the model training process, and the targets are used in the model reasoning process. This description is also applicable to the various technical solutions described later and will not be repeated later.

[0060] S320, inputting the first sample medical image and the second modal feature into the generator to be trained, so as to utilize the generator to be trained and take the second modal feature as prior knowledge to perform modal conversion of the first sample medical image to obtain a second converted medical image.

[0061] Among them, the generator to be trained can be understood as a generator that has training needs. In the embodiment of the present invention, optionally, the generator can be an original generator that has not been trained, or an intermediate generator that has undergone some training but not completed the training, etc., which is not specifically limited here.

[0062] The process of obtaining the second converted medical image is similar to that of the second target medical image, and will not be repeated here. It should be noted that since the generator to be trained has not been trained yet, it is unknown whether the second converted medical image generated by the generator to be trained is a medical image under the second modality.

[0063] S330, inputting the second converted medical image and the second modality feature into the discriminator to be trained, performing loss calculation on the first predicted label output by the discriminator to be trained and the preset first true label, and updating the parameters in the generator to be trained according to the loss calculation result to obtain a target generator;

[0064] The first true label represents that the second converted medical image is a medical image under the second modality.

[0065] Among them, the meaning of the discriminator to be trained is similar to that of the generator to be trained, which will not be repeated here.

[0066] The first predicted label can be understood as the prediction result of the discriminator to be trained on whether the second converted medical image is a medical image under the second modality represented by the second modality feature, while the first true label indicates that the second converted medical image is a medical image under the second modality. Therefore, the loss can be calculated through these two labels, and the parameters in the generator to be trained are updated according to the loss calculation result, so as to obtain the target generator.

[0067] S340, acquiring a first target medical image acquired for a target part of a target object in a first modality and a second modality to which the first target medical image is to be converted;

[0068] The target site and the sample site are the same site.

[0069] S350, inputting the first target medical image and the second modality feature into a target generator, using the target generator to convert the first target medical image into the second modality with the second modality feature as prior knowledge, to obtain and output the second target medical image.

[0070] The technical solution of the embodiment of the present invention realizes effective training of the generator through the discriminator.

[0071] In an optional technical solution, the discriminator to be trained includes multiple network layers to be trained and multiple downsampling layers to be trained, and the second modality feature is applied in the following manner:

[0072] During the label prediction process, the second intermediate feature output by the downsampling layer to be trained corresponding to the second modal feature in the multiple downsampling layers to be trained is concatenated with the second modal feature in the channel dimension, and the concatenated result is input into the network layer to be trained connected to the corresponding downsampling layer to be trained in the multiple network layers to be trained.

[0073] Among them, similar to the target encoder described above, the discriminator to be trained also uses the second modal feature as prior knowledge to output the first predicted label, and the role of this prior knowledge in the label prediction process is similar to its role in the feature encoding process, which will not be repeated here.

[0074] As another optional technical solution, the target generator is also pre-trained in the following way:

[0075] Acquire a predetermined first modal feature representing a first modality;

[0076] Inputting the second converted medical image and the first modality feature into the generator to be trained, so as to use the generator to be trained and take the first modality feature as prior knowledge to perform modality conversion on the second converted medical image to obtain a second restored medical image;

[0077] Loss calculation is performed on the second restored medical image and the first sample medical image, and the parameters in the generator to be trained are updated according to the loss calculation result to obtain a target generator.

[0078] Among them, the meaning of the first modal feature is similar to that of the second modal feature, which will not be repeated here.

[0079] The process of obtaining the second restored medical image is similar to that of the second converted medical image, and will not be repeated here. Since the second restored medical image is a medical image obtained by converting the first sample medical image to the second modality and then converting it back to the first modality, the loss calculation can be performed on the second restored medical image and the first sample medical image, and the parameters in the generator to be trained can be updated according to the loss calculation result to obtain the target generator.

[0080] The above technical solution, by calculating the cyclic conversion loss, enables the generator to successfully convert the modality of the input medical image while retaining the content and details of the input medical image.

[0081] On this basis, optionally, the target generator is also pre-trained in the following way:

[0082] Inputting the second restored medical image and the first modality feature into the discriminator to be trained, performing loss calculation on the second predicted label output by the discriminator to be trained and the preset second true label, and updating the parameters in the generator to be trained according to the loss calculation result to obtain a target generator;

[0083] The second true label represents that the second restored medical image is a medical image under the first modality.

[0084] The above technical solution adds the loss of related labels on the basis of the cyclic conversion loss, further improving the training effect of the generator.

[0085] As another optional technical solution, the target generator is also pre-trained in the following way:

[0086] Acquire a predetermined first modal feature representing a first modality;

[0087] Inputting the first sample medical image and the first modal feature into the generator to be trained, using the generator to be trained and taking the first modal feature as prior knowledge, performing modal conversion on the first sample medical image to obtain a first restored medical image;

[0088] Loss calculation is performed on the first restored medical image and the first sample medical image, and the parameters in the generator to be trained are updated according to the loss calculation result to obtain a target generator.

[0089] The above technical solution imposes restrictions on the generator by calculating the autoencoder loss. The restriction is achieved by the generator generating the same mode from one mode to complete self-reconstruction, thereby avoiding the generation of unrealistic details and improving the stability of the generator.

[0090] On this basis, optionally, the target generator is also pre-trained in the following way:

[0091] Inputting the first restored medical image and the first modality feature into the discriminator to be trained, performing loss calculation on the third predicted label output by the discriminator to be trained and the preset third true label, and updating the parameters in the generator to be trained according to the loss calculation result to obtain a target generator;

[0092] The third true label represents that the first restored medical image is a medical image under the first modality.

[0093] The above technical solution adds the loss of related labels on the basis of the cyclic conversion loss, further improving the training effect of the generator.

[0094] Figure 4It is a flowchart of another medical image modality conversion method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the second modality feature is predetermined in the following manner: obtaining a pre-trained modality feature extractor, and a plurality of second sample medical images acquired from a sample part of a sample object under a second modality, wherein the sample part and the target part are the same part; for each second sample medical image in the plurality of second sample medical images, inputting the second sample medical image into the modality feature extractor to obtain the medical image modality feature output by the target feature extraction layer of the modality feature extractor; and obtaining the second modality feature according to the medical image modality features corresponding to the plurality of second sample medical images. Among them, the explanations of the terms that are the same as or corresponding to the above-mentioned embodiments are not repeated here.

[0095] See also Figure 4 The method of this embodiment may specifically include the following steps:

[0096] S410: Obtain a pre-trained modality feature extractor, and obtain a plurality of second sample medical images acquired from sample parts of the sample object under a second modality.

[0097] The modality feature extractor can be understood as a pre-trained network for extracting modality features of medical images. The modality feature extractor is obtained.

[0098] S420. For each second sample medical image in the plurality of second sample medical images, input the second sample medical image into a modality feature extractor to obtain a medical image modality feature output by a target feature extraction layer of the modality feature extractor.

[0099] Among them, the modal feature extractor includes multiple modal feature extraction layers, and the target feature extraction layer can be understood as the modal feature extraction layer in the multiple modal feature extraction layers that corresponds to the second modal feature to be obtained. On this basis, combined with the application scenarios that may be involved in the embodiments of the present invention, optionally, the above correspondence can be expressed in the following way: Assuming that the number of second modal features is N, then for the modal feature extraction layer corresponding to a second modal feature, the order of the second modal feature in the N second modal features is the same as the order of the modal feature extraction layer in the multiple modal feature extraction layers. Of course, the above correspondence can also be expressed in other ways, which are not specifically limited here.

[0100] Here, any second sample medical image among multiple second sample medical images is taken as an example, and the second sample medical image is input into the modal feature extractor to obtain the medical image modal feature output by the target feature extraction layer. The medical image modal feature can characterize the medical image modality of the second sample medical image.

[0101] S430: Obtain a second modality feature representing a second modality according to the medical image modality features respectively corresponding to the plurality of second sample medical images.

[0102] Among them, the second modal feature is obtained according to the medical image modal features corresponding to multiple second sample medical images. For example, the mean, median or mode of these medical image modal features can be used as the second modal feature. This can be set according to actual needs and is not specifically limited here.

[0103] S440, acquiring a first target medical image acquired in a first modality for a target part of a target object, a second modality to which the first target medical image is to be converted, and a pre-trained target generator;

[0104] The target site and the sample site are the same site.

[0105] S450. Input the first target medical image and the second modality feature into a target generator, and utilize the target generator to convert the first target medical image into the second modality using the second modality feature as prior knowledge, to obtain and output the second target medical image.

[0106] The technical solution of the embodiment of the present invention obtains the modal features corresponding to the different medical image modalities by utilizing training sets of different medical image modalities and combining them with a modal feature extractor.

[0107] As an optional technical solution, the modal feature extractor is pre-trained in the following way:

[0108] For each second sample medical image in the plurality of second sample medical images, taking the second sample medical image and the second modality as a group of training samples;

[0109] Based on at least multiple groups of training samples, a modality classification network is trained to obtain a modality feature extractor.

[0110] The above technical solution uses multiple groups of training samples under different medical image modalities to train the modality classification network, thereby effectively obtaining the modality feature extractor.

[0111] In order to better understand the above-mentioned technical solutions as a whole, Figure 5-Figure 7 The specific examples given are for illustrative purposes. In this example, the medical image modality may be referred to as modality for short.

[0112] For example, the general implementation process of this example is as follows:

[0113] 1. Collect medical data of multiple modalities;

[0114] 2. Construct a modal feature extractor and use it to extract modal features of different modes;

[0115] 3. Use the modality features as prior knowledge to inform the generator of the modality of the medical image to be generated, which includes modality conversion loss, autoencoder loss and cyclic conversion loss;

[0116] 4. Use the discriminator to improve the generation ability of the generator.

[0117] Next, we will elaborate on the above example from two aspects: data and model.

[0118] Data: The data in this example contains K modalities of the same part, such as CT enhancement, CT plain scan, and MRI. Each modality contains N sample data, a total of N*K samples, where each sample is represented by a 3D medical image. The data of each modality is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The training set is used to train the model, the validation set is used to select the best performing model from the trained models, and the test set is used to evaluate the performance of the model.

[0119] In this example, we need to construct the modality classification labels of medical images in advance to train the modality classification network and obtain the modality feature extractor. The medical images and their modality classification labels are constructed as follows: (x (i) ,y (i) ), where i represents the i-th medical image, if x (i) is a medical image under modality 1, then its label y (i) =1; if x (i) is a medical image under modality 2, then its label y (i) =2, and so on.

[0120] Model: The complete implementation of this example is as follows:

[0121] S1 uses the modality feature extractor to classify medical images under different modalities and obtain modality features. For details, see Figure 5 and Figure 7 ,in, Figure 7 The first column shows the network structure of the discriminator, the second column shows the network structure of the modality feature extractor, the third column shows the network structure of the encoder, and the fourth column shows the network structure of the decoder.

[0122] S11 Input medical images and their modality classification labels: Input training set sample i (x (i) ,y (i) ).

[0123] S12 2D image extraction: based on 3D medical images x (i), randomly select a position from its z-axis, and extract c 2D medical images that are continuous with the position, then resize these 2D medical images so that their sizes are consistent with c×w×d. The c×w×d 3D medical image is subsequently processed.

[0124] S13 image enhancement: performing image enhancement on the 3D medical image output by S12, for example, image enhancement may be performed based on at least one of rotation, translation, elastic transformation, and noise addition.

[0125] S14 modality classification: Input the 3D medical image output by S13 into the modality classification network, obtain the modality classification result predicted by the modality classification network, and compare the modality classification result with y (i) For comparison, the cross entropy function is used to calculate the loss, and the parameters in the modality classification network are updated according to the loss.

[0126] After the training is completed, S15 uses the validation set to select the classification performance from the trained modal classification networks, that is, the one with the best classification accuracy, and applies it as a modal feature extractor.

[0127] S16 obtains modal features: input all the data of the training set of K modalities into the modal feature extractor output by S15, and calculate the medical image modal features of each sample output by the first, second and third modal feature extraction layers of the modal feature extractor, respectively. The three modal feature extraction layers here are respectively equivalent to the target feature extraction layers described above. Subsequently, the medical image modal features output by the first modal feature extraction layer of all data belonging to the same modality are averaged, the medical image modal features output by the second modal feature extraction layer are averaged, and the medical image modal features output by the third modal feature extraction layer are averaged, and these average results are respectively used as the modal features of the modality. Exemplarily, here taking the jth modality as an example, the modal feature of this modality is z j,1 ,z j,2 ,z j,3 .

[0128] The above S1 uses training sets of different modalities and obtains three modal features of different modalities through a modal feature extractor. These modal features can be used as prior knowledge in the subsequent medical image generation process.

[0129] The following S2-S4 describe the training process of the generator. For details, see Figure 6 ,in, Figure 6 The first row corresponds to S2, the second row corresponds to S4, and the third row corresponds to S3.

[0130] S2 uses a generator to complete the conversion of any modality. In order to achieve the conversion of any modality with only one generator, the generator needs to be given prior knowledge, that is, to tell the generator that the input medical image should be generated into the required modality at this time. The generator in this example uses the U-Net network architecture.

[0131] S21 Input medical image: Input training set sample i x (i) .

[0132] S22 2D image extraction: based on 3D medical images x (i) , randomly select a position from its z-axis, and extract c 2D medical images that are continuous with the position, then resize these 2D medical images so that their sizes are consistent with c×w×d. The c×w×d 3D medical image is subsequently processed.

[0133] S23 mode p generates mode q: convert x (i) Input to the generator for feature encoding and feature decoding. Among them, the mode p represents x (i) The mode q represents the generator needs to convert x (i) Generate mode q from mode p.

[0134] S24 feature encoding: input x (i) To the encoder of the generator, the first intermediate feature obtained by the first downsampling layer of the encoder is combined with z q,1 The concatenation is performed in the channel dimension, and the concatenation result is then input into the subsequent encoding layer. The first intermediate feature obtained by the second downsampling layer of the encoder is combined with z q,2 The concatenation is performed in the channel dimension, and the concatenation result is then input into the subsequent encoding layer. The first intermediate feature obtained by the third downsampling layer of the encoder is combined with z q,3 The splicing is performed in the channel dimension, and the splicing result is input into the subsequent encoding layer to obtain the medical image features output by the last encoding layer.

[0135] It should be noted that as the network depth increases, the global information gradually weakens, while the local information gradually increases. However, when the generator generates medical images, it needs not only local information but also global information. Therefore, the modal features of the first few layers are concatenated to enhance the global information.

[0136] S25 feature decoding: Upsample the medical image features output by the encoder, and concatenate the upsampled results with the coded features of the same size obtained in the encoder in the feature dimension, and input the concatenated results into the decoding layer to obtain the decoded features. Then, upsample the decoded features, concatenate the upsampled results with the coded features of the same size obtained in the encoder in the feature dimension, and input the concatenated results into the next decoding layer to obtain the decoded features. And so on, until the final generated medical image is obtained.

[0137] Combined with the above explanation, mode p represents the first mode, x (i) represents the first sample medical image; modality q represents the second modality, represents a second transformed medical image.

[0138] S26 discriminator distinguishes true from false: Input to the discriminator, and compare the second intermediate feature obtained by the first downsampling layer of the discriminator with z q,1 Concatenate in the channel dimension and then input the concatenated result into the subsequent network layer. The second intermediate feature obtained by the second downsampling layer of the discriminator is combined with z q,2 The concatenation is performed in the channel dimension, and the concatenation result is then input into the subsequent network layer. The second intermediate feature obtained by the third downsampling layer of the discriminator is combined with z q,3 The concatenation is performed in the channel dimension and the concatenation result is input into the subsequent network layer. Finally, the remaining network layers of the discriminator are used to calculate the input features and output the first predicted label. Let the first true label be 1, that is, let the discriminator think is under mode q, and the cross entropy function is used to calculate the loss between the first predicted label and the first true label, recorded as use Update the parameters in the generator.

[0139] S3 calculates the autoencoder loss. This is to impose a restriction on the generator to avoid generating unrealistic details. This restriction is achieved by the generator generating the same modality from one modality, that is, completing self-reconstruction.

[0140] S31 Input medical image: Input training set sample i x (i) .

[0141] S32 2D image extraction: Similar to S22, the specific implementation process will not be repeated here.

[0142] S33 modal p generates modal p: convert x (i) Input to the generator for feature encoding and feature decoding. Among them, the mode p represents x (i)modality, can also be regarded as a medical image restoration task.

[0143] S34 feature encoding: Similar to S24, except that z corresponding to mode p is used here p,1 、z p,2 and z p,3 The specific implementation process will not be described in detail.

[0144] S35 feature decoding: Similar to S25, the specific implementation process is not repeated here. This step generates a medical image That is the first restored medical image described above.

[0145] S36 discriminator distinguishes true from false: Input to the discriminator, the subsequent processing is similar to S26, except that z corresponding to mode p is applied here p,1 、z p,2 and z p,3 , which will not be described here. Get the third predicted label output by the discriminator. Let the third true label be 1, that is, let the discriminator think Under mode p, the cross entropy function is used to calculate the loss between the two labels, which is recorded as

[0146] S37 reconstruction loss: and x (i) The loss is calculated based on the mean square error, and the loss is calculated based on the loss and S36. Accumulate, and then update the parameters in the generator based on the accumulated results.

[0147] S4 Cyclic Conversion Loss: In order to further ensure that the generator can successfully convert the modality of the input medical image and retain the content and details of the medical image, this example adds a cyclic conversion loss.

[0148] S41 image loop recovery: the output of S25 Input to the encoder, the subsequent processing is similar to S24, except that the z corresponding to the mode p is applied here p,1 、z p,2 and z p,3 , I will not go into details here.

[0149] S42 feature decoding: Similar to S25, the specific implementation process is not repeated here. This step generates a medical image That is the second restored medical image described above.

[0150] S43 discriminator distinguishes true from false: Input to the discriminator, the subsequent processing is similar to S26, except that the z corresponding to the mode p is applied here p,1 、zp,2 and z p,3 , which will not be described here. Get the second predicted label output by the discriminator. Let the second true label be 1, that is, let the discriminator think Under mode p, the cross entropy function is used to calculate the loss between the two labels, which is recorded as

[0151] S44 reconstruction loss: and x (i) The loss is calculated based on the mean square error, and the loss is calculated based on the loss and S43. Accumulate, and then update the parameters in the generator based on the obtained cyclic adversarial generation loss.

[0152] The above S4 is the input and the modal characteristics z of mode p p,1 、z p,2 and z p,3 ,Will Restore to x (i) . This ensures that x (i) and There is only a difference in modality, but no difference in the content and details of the medical images.

[0153] S5 updates the discriminator. In practical applications, the discriminator and the generator can be trained alternately.

[0154] S51 inputs a real medical image, and inputs all real medical images and their corresponding modal features into the discriminator, and lets the discriminator judge as true to calculate the loss.

[0155] S52 inputs the generated medical image, and inputs all the generated medical images and their corresponding modal features into the discriminator, and lets the discriminator determine that they are false to calculate the loss.

[0156] S53 updates the parameters in the discriminator based on the losses of S51 and S52.

[0157] S6 selects the best performing generator from the trained generators based on the validation set.

[0158] S7 model prediction.

[0159] S71 inputs any medical image x and the modality k to be converted, and inputs x into the generator output by S6 for feature encoding and feature decoding.

[0160] S72 feature encoding: Similar to S24, except that z corresponding to mode k is used here k,1 、z k,2 and z k,3 The specific implementation process will not be described in detail.

[0161] S73 feature decoding: Similar to S25, the specific implementation process is not repeated here. This step generates a medical image That is, convert x into a medical image under modality k.

[0162] The above example only requires one generator and one discriminator, but it can support conversion of any number of modalities.

[0163] Figure 8 This is a structural block diagram of a medical image modality conversion device provided in an embodiment of the present invention, and the device is used to execute the medical image modality conversion method provided in any of the above embodiments. The device and the medical image modality conversion method in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the medical image modality conversion device, reference can be made to the embodiment of the above medical image modality conversion method. Figure 8 The device may specifically include: a target generator acquisition module 510, a second modality feature acquisition module 520 and a medical image modality conversion module 530.

[0164] A target generator acquisition module 510 is used to acquire a first target medical image acquired in a first modality for a target part of a target object, a second modality to which the first target medical image is to be converted, and a pre-trained target generator;

[0165] A second modal feature acquisition module 520, configured to acquire a predetermined second modal feature representing a second modality;

[0166] The medical image modality conversion module 530 is used to input the first target medical image and the second modality features into the target generator, so as to utilize the target generator and the second modality features as prior knowledge to convert the first target medical image to the second modality, and obtain and output the second target medical image.

[0167] Optionally, the target generator includes a target encoder and a target decoder;

[0168] The medical image modality conversion module 530 is specifically used for:

[0169] Inputting the first target medical image and the second modality feature into a target encoder, and using the target encoder to perform feature encoding on the first target medical image using the second modality feature as prior knowledge to obtain a medical image feature;

[0170] The medical image features are input into a target decoder, and the target decoder is used to perform feature decoding on the medical image features, thereby obtaining a second target medical image that is output as the first target medical image in a second modality.

[0171] On this basis, optionally, the target encoder includes multiple target encoding layers and multiple target downsampling layers, and the medical image modality conversion module 530 applies the second modality feature in the following manner:

[0172] During the feature encoding process, the first intermediate feature output by the target downsampling layer corresponding to the second modal feature in the multiple target downsampling layers is spliced ​​with the second modal feature in the channel dimension, and the spliced ​​result is input into the target encoding layer connected to the corresponding target downsampling layer in the multiple target encoding layers.

[0173] Optionally, the target generator is pre-trained using the following modules:

[0174] A first sample medical image acquisition module, used to acquire a first sample medical image acquired from a sample part of a sample object in a first modality and a second modality feature, wherein the sample part and the target part are the same part;

[0175] A second converted medical image obtaining module may be used to input the first sample medical image and the second modality feature into the generator to be trained, so as to utilize the generator to be trained and take the second modality feature as prior knowledge to perform modality conversion of the first sample medical image to obtain a second converted medical image;

[0176] The first obtaining module of the target generator is used to input the second converted medical image and the second modality feature into the discriminator to be trained, and perform loss calculation on the first predicted label output by the discriminator to be trained and the preset first true label, and update the parameters of the generator to be trained according to the loss calculation result to obtain the target generator; wherein the first true label represents that the second converted medical image is a medical image under the second modality.

[0177] On this basis, optionally, the discriminator to be trained may include multiple network layers to be trained and multiple downsampling layers to be trained, and the second converted medical image obtaining module applies the second modality feature in the following manner:

[0178] During the label prediction process, the second intermediate feature output by the downsampling layer to be trained corresponding to the second modal feature in the multiple downsampling layers to be trained is concatenated with the second modal feature in the channel dimension, and the concatenated result is input into the network layer to be trained connected to the corresponding downsampling layer to be trained in the multiple network layers to be trained.

[0179] Alternatively, the target generator is also pre-trained using the following modules:

[0180] A first modal feature first acquisition module, used to acquire a predetermined first modal feature representing a first modality;

[0181] A second restored medical image obtaining module may be used to input the second converted medical image and the first modality feature into the generator to be trained, so as to utilize the generator to be trained and take the first modality feature as prior knowledge to perform modality conversion on the second converted medical image to obtain a second restored medical image;

[0182] The second module of the target generator is used to calculate the loss of the second restored medical image and the first sample medical image, and update the parameters in the generator to be trained according to the loss calculation result to obtain the target generator.

[0183] On this basis, optionally, the target generator is also pre-trained through the following modules:

[0184] The third obtaining module of the target generator is used to input the second restored medical image and the first modality feature into the discriminator to be trained, perform loss calculation on the second predicted label output by the discriminator to be trained and the preset second true label, and update the parameters of the generator to be trained according to the loss calculation result to obtain the target generator; wherein the second true label represents that the second restored medical image is a medical image under the first modality.

[0185] Optionally, the target generator is also pre-trained through the following modules:

[0186] A first modal feature second acquisition module, used to acquire a predetermined first modal feature representing a first modality;

[0187] A first restored medical image obtaining module may be used to input the first sample medical image and the first modal feature into the generator to be trained, so as to utilize the generator to be trained and take the first modal feature as prior knowledge to perform modal conversion of the first sample medical image to obtain a first restored medical image;

[0188] The fourth module of the target generator is used to perform loss calculation on the first restored medical image and the first sample medical image, and update the parameters in the generator to be trained according to the loss calculation result to obtain the target generator.

[0189] Optionally, the second modal feature is predetermined by the following module:

[0190] A second sample medical image acquisition module, used to acquire a pre-trained modality feature extractor and a plurality of second sample medical images acquired from a sample part of a sample object under a second modality, wherein the sample part and the target part are the same part;

[0191] A medical image modality feature obtaining module may be used to input each second sample medical image in a plurality of second sample medical images into a modality feature extractor to obtain a medical image modality feature output by a target feature extraction layer of the modality feature extractor;

[0192] The second modality feature obtaining module is used to obtain the second modality feature according to the medical image modality features respectively corresponding to the plurality of second sample medical images.

[0193] On this basis, optionally, the modal feature extractor is pre-trained through the following modules:

[0194] A training sample obtaining module may be used to use the second sample medical image and the second modality as a group of training samples for each second sample medical image in the plurality of second sample medical images;

[0195] The modal feature extractor obtaining module is used to train the modal classification network based on at least multiple groups of training samples to obtain a modal feature extractor.

[0196] The medical image modality conversion device provided by the embodiment of the present invention obtains the first target medical image collected for the target part of the target object under the first modality, the second modality to which the first target medical image is to be converted, and the pre-trained target generator through the target generator acquisition module; obtains the second modality feature representing the second modality determined in advance through the second modality feature acquisition module, and the second modality feature can be used to indicate that the second target medical image under the second modality needs to be generated; and inputs the first target medical image and the second modality feature into the target generator through the medical image modality conversion module, so that the target generator can be used to convert the first target medical image to the second modality with the second modality feature as prior knowledge, and obtain and output the second target medical image. The above device, by inputting the modality feature that can indicate the medical image modality to which the target medical image to be generated belongs into the target generator, enables one target generator to perform multiple modality conversions of the medical image, thereby solving the problem of large limitations and large computational consumption of modality conversion caused by using two target generators to perform two modality conversions.

[0197] The medical image modality conversion device provided in the embodiment of the present invention can execute the medical image modality conversion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0198] It is worth noting that in the embodiment of the above-mentioned medical image modality conversion device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0199] Fig. 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0200] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0201] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0202] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a medical image modality conversion method.

[0203] In some embodiments, the medical image modality conversion method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the medical image modality conversion method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the medical image modality conversion method in any other appropriate manner (e.g., by means of firmware).

[0204] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0205] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0206] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0207] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0208] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0209] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0210] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0211] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A medical image modality conversion method, characterized in that: include: Acquire a first target medical image acquired in a first modality for a target part of a target object, a second modality to which the first target medical image is to be converted, and a pre-trained target generator; Acquire a predetermined second modal feature representing the second modality; The first target medical image and the second modality feature are input into the target generator, so as to utilize the target generator and the second modality feature as prior knowledge to convert the first target medical image into the second modality, thereby obtaining and outputting the second target medical image.

2. The method according to claim 1, characterized in that The target generator includes a target encoder and a target decoder; The step of inputting the first target medical image and the second modality feature into the target generator, using the target generator and taking the second modality feature as prior knowledge to convert the first target medical image into the second modality, and obtaining and outputting the second target medical image, comprises: Inputting the first target medical image and the second modality feature into the target encoder, so as to utilize the target encoder to perform feature encoding on the first target medical image using the second modality feature as prior knowledge to obtain a medical image feature; The medical image features are input into the target decoder, so as to use the target decoder to perform feature decoding on the medical image features, thereby obtaining and outputting a second target medical image of the first target medical image in the second modality.

3. The method according to claim 2, characterized in that The target encoder includes a plurality of target encoding layers and a plurality of target downsampling layers, and the second modality feature is applied in the following manner: During the feature encoding process, the first intermediate feature output by the target downsampling layer corresponding to the second modal feature in the multiple target downsampling layers is spliced ​​with the second modal feature in the channel dimension, and the spliced ​​result is input into the target encoding layer connected to the corresponding target downsampling layer in the multiple target encoding layers.

4. The method according to claim 1, characterized in that: The target generator is pre-trained in the following way: Acquire a first sample medical image acquired from a sample part of a sample object under the first modality and a feature of the second modality, wherein the sample part and the target part are the same part; Inputting the first sample medical image and the second modality feature into a generator to be trained, so as to use the generator to be trained and take the second modality feature as prior knowledge to perform modality conversion of the first sample medical image to obtain a second converted medical image; Inputting the second converted medical image and the second modality feature into the discriminator to be trained, performing loss calculation on the first predicted label output by the discriminator to be trained and the preset first true label, and updating the parameters in the generator to be trained according to the loss calculation result to obtain the target generator; The first true label represents that the second converted medical image is a medical image under the second modality.

5. The method according to claim 4, characterized in that The discriminator to be trained includes a plurality of network layers to be trained and a plurality of downsampling layers to be trained, and the second modality feature is applied in the following manner: During the label prediction process, the second intermediate feature output by the downsampling layer to be trained corresponding to the second modal feature in the multiple downsampling layers to be trained is spliced ​​with the second modal feature in the channel dimension, and the spliced ​​result is input into the network layer to be trained connected to the corresponding downsampling layer to be trained in the multiple network layers to be trained.

6. The method according to claim 4, characterized in that The target generator is also pre-trained in the following way: Acquire a predetermined first modal feature representing the first modality; Inputting the second converted medical image and the first modality feature into the generator to be trained, so as to use the generator to be trained and take the first modality feature as prior knowledge to perform modality conversion on the second converted medical image to obtain a second restored medical image; Loss calculation is performed on the second restored medical image and the first sample medical image, and the parameters in the generator to be trained are updated according to the loss calculation result to obtain the target generator.

7. The method according to claim 6, characterized in that The target generator is also pre-trained in the following way: Inputting the second restored medical image and the first modality feature into the discriminator to be trained, performing loss calculation on the second predicted label output by the discriminator to be trained and the preset second true label, and updating the parameters in the generator to be trained according to the loss calculation result to obtain the target generator; The second true label represents that the second restored medical image is a medical image under the first modality.

8. The method according to claim 4, characterized in that The target generator is also pre-trained in the following way: Acquire a predetermined first modal feature representing the first modality; Inputting the first sample medical image and the first modal feature into the generator to be trained, so as to utilize the generator to be trained and use the first modal feature as prior knowledge to perform modal conversion on the first sample medical image to obtain a first restored medical image; Loss calculation is performed on the first restored medical image and the first sample medical image, and the parameters in the generator to be trained are updated according to the loss calculation result to obtain the target generator.

9. The method according to claim 1, characterized in that: The second modal feature is predetermined by: Acquire a pre-trained modality feature extractor and a plurality of second sample medical images acquired from a sample part of a sample object under the second modality, wherein the sample part and the target part are the same part; For each of the plurality of second sample medical images, inputting the second sample medical image into the modality feature extractor to obtain a medical image modality feature output by a target feature extraction layer of the modality feature extractor; The second modality feature is obtained according to the medical image modality features respectively corresponding to the plurality of second sample medical images.

10. The method according to claim 9, characterized in that The modality feature extractor is pre-trained in the following way: For each of the second sample medical images in the plurality of second sample medical images, taking the second sample medical image and the second modality as a group of training samples; The modality classification network is trained based on at least a plurality of groups of the training samples to obtain the modality feature extractor.

Images