Image mode conversion method and device, electronic equipment and storage medium
By using the target forward and backward generation models in CT and MRI image conversion, the model parameters are optimized to take into account both global and local information, the problem of inaccurate image conversion in the prior art is solved, and higher image modal conversion accuracy is achieved.
Patent Information
- Application Number
- CN202411736414.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-07
Smart Images
Figure CN119579399A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to an image modality conversion method, device, electronic device, and storage medium. Background Art
[0002] With the development of technology, more and more models are being applied to the medical field. For example, a computed tomography (CT) image of a user's examination site is converted into a magnetic resonance imaging (MRI) image.
[0003] Currently, in deep learning-based methods, the conversion between CT and MRI images is usually implemented using a generative model. For example, the CT image of the same user is used as input, and the MRI image of the same user is used as a supervisory label for model generation training.
[0004] However, the generative model obtained in this way often fails to take into account both global and local information when converting images, and often generates unrealistic detail information, resulting in inaccurate converted images. Summary of the invention
[0005] The embodiments of the present invention provide an image modality conversion method, device, electronic device and storage medium to accurately and conveniently determine a target forward generation model and a target backward generation model, so that accurate image modality conversion can be achieved through the target forward generation model or the target backward generation model, thereby improving the accuracy of image modality conversion.
[0006] In a first aspect, an embodiment of the present invention provides an image modality conversion method, comprising:
[0007] Obtaining a first modality image and a second modality image corresponding to the training sample;
[0008] Inputting the first modality image into a forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and inputting the second modality image into a backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample;
[0009] Determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image, optimize the model parameters of the forward discriminator based on the forward discrimination loss value to obtain a target forward generation model; the target forward generation model is used to convert the first modality image into the second modality image;
[0010] Based on the second modality image and the fourth modality image, a backward generation loss value corresponding to the backward generation model is determined, and model parameters of the backward generation model are optimized based on the backward generation loss value. Based on the first modality image and the fourth modality image, a backward discrimination loss value corresponding to the backward discriminator is determined, and model parameters of the backward discriminator are optimized based on the backward discrimination loss value to obtain a target backward generation model; the target backward generation model is used to convert the second modality image into the first modality image.
[0011] In a second aspect, an embodiment of the present invention further provides an image modality conversion device, the device comprising:
[0012] A modality image acquisition module, used to acquire a first modality image and a second modality image corresponding to a training sample;
[0013] A modality image conversion module, configured to input the first modality image into a forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and input the second modality image into a backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample;
[0014] A forward generation model training module, used to determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image, optimize the model parameters of the forward discriminator based on the forward discrimination loss value, to obtain a target forward generation model; the target forward generation model is used to convert the first modality image into the second modality image;
[0015] A backward generation model training module is used to determine a backward generation loss value corresponding to the backward generation model based on the second modality image and the fourth modality image, optimize the model parameters of the backward generation model based on the backward generation loss value, and determine a backward discrimination loss value corresponding to a backward discriminator based on the first modality image and the fourth modality image, optimize the model parameters of the backward discriminator based on the backward discrimination loss value to obtain a target backward generation model; the target backward generation model is used to convert the second modality image into the first modality image.
[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0017] one or more processors;
[0018] A memory for storing one or more programs;
[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the image modality conversion method provided by any embodiment of the present invention.
[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image modality conversion method as provided in any embodiment of the present invention.
[0021] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the image modality conversion method provided by any embodiment of the present invention.
[0022] The technical solution of the embodiment of the present invention is to obtain a first modality image and a second modality image corresponding to a training sample; input the first modality image into a forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and input the second modality image into a backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample; determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image; and The forward discriminant loss value is used to optimize the model parameters of the forward discriminator to obtain a target forward generation model; the target forward generation model is used to convert the first modality image into the second modality image; the backward generation loss value corresponding to the backward generation model is determined based on the second modality image and the fourth modality image, the model parameters of the backward generation model are optimized based on the backward generation loss value, and the backward discriminant loss value corresponding to the backward discriminator is determined based on the first modality image and the fourth modality image, the model parameters of the backward discriminator are optimized based on the backward discriminant loss value to obtain a target backward generation model; the target backward generation model is used to convert the second modality image into the first modality image. In this way, the target forward generation model and the target backward generation model can be accurately and conveniently determined, and accurate image modality conversion can be achieved through the target forward generation model or the target backward generation model, thereby improving the accuracy of image modality conversion.
[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1 is a flow chart of an image modality conversion method provided by Embodiment 1 of the present invention;
[0026] Figure 2 is a flow chart of an image modality conversion method provided by Embodiment 2 of the present invention;
[0027] Figure 3 is an example diagram of a cyclic adversarial generative network system involved in Embodiment 2 of the present invention;
[0028] Figure 4 is a structural schematic diagram of an image modality conversion device provided in Embodiment 3 of the present invention;
[0029] Figure 5 It is a schematic diagram of the structure of an electronic device for implementing the image modality conversion method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0032] Embodiment 1
[0033] Figure 1 A flowchart of an image modality conversion method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a target forward generation model and a target backward generation model are determined and accurate image modality conversion can be achieved through the target forward generation model or the target backward generation model. The method can be performed by an image modality conversion device, which can be implemented in the form of hardware and / or software, and the image modality conversion device can be configured in an electronic device. Figure 1 As shown, the method includes:
[0034] S110: Acquire a first modality image and a second modality image corresponding to the training sample.
[0035] Wherein, each training sample may be composed of a first modality image and a second modality image corresponding to the same target. This embodiment may include N training samples. The first modality image may refer to an image whose image content representation modality is the first modality. The second modality image may refer to an image whose image content representation modality is the second modality. For example, the first modality may be a CT modality, hereinafter represented as A modality or modality A. The second modality may be an MRI modality, hereinafter represented as B modality or modality B. The first modality image may refer to a captured CT image, and the second modality image may refer to a captured MRI image.
[0036] Specifically, obtain the first modality image of the abdominal modality A of the training sample i and the second modality image of the abdominal modality B
[0037] Exemplarily, after acquiring the first modality image and the second modality image corresponding to the training sample, size transformation and data enhancement may be performed on the acquired images.
[0038] Specifically, for the first modality image and the second modality image Resize the images so that the sizes of the two images are the same, both h×w×d. and the second modality image Data enhancement is performed separately, such as rotation, translation, elastic transformation, and noise addition. This keeps the image size consistent, takes into account both global and local information in the image, and avoids the trained generative model from generating unrealistic detail information.
[0039] S120, input the first modality image into the forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and input the second modality image into the backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample.
[0040] Among them, the forward generative model can be used to convert an image of the first modality into an image of the second modality. The third modality image can refer to an image obtained by converting the first modality image into the second modality. The backward generative model can be used to convert the second modality image into the first modality image. The fourth modality image can refer to an image obtained by converting the second modality image into the first modality.
[0041] Specifically, the first modality image Input into the forward generation model F of the recurrent adversarial generation network to generate the third modality image The second modality image Input into the backward generation model B of the cyclic adversarial generation network and output the fourth modality image
[0042] S130. Determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to the forward discriminator based on the second modality image and the third modality image, optimize the model parameters of the forward discriminator based on the forward discrimination loss value to obtain a target forward generation model.
[0043] Among them, the target forward generation model is used to convert the first modality image into the second modality image. The forward generation loss value may refer to the generation deviation value between the first modality image and the third modality image. The forward discriminator may be used to discriminate the deviation value between the image generated by the forward generation model and the accurate second modality image. The forward discriminant loss value may refer to the deviation value between the image generated by the forward generation model and the accurate second modality image. The target forward generation model may refer to a generation model obtained after iterative training that can accurately realize image modality conversion.
[0044] Specifically, the image feature deviation between the first modality image and the third modality image is compared and the forward generation loss value corresponding to the forward generation model is calculated. The forward generation loss values corresponding to each training sample are accumulated and averaged, and the model parameters of the forward generation model are updated according to the average loss value. The image feature deviation between the second modality image and the third modality image is discriminated by a forward discriminator and the forward discriminant loss value corresponding to the forward discriminator is calculated. The forward discriminant loss values corresponding to each training sample are accumulated and averaged, and the model parameters of the forward discriminator are updated according to the average loss value. When the preset convergence condition is reached, the iterative training is stopped to obtain the trained target forward generation model.
[0045] S140. Determine a backward generation loss value corresponding to the backward generation model based on the second modality image and the fourth modality image, optimize the model parameters of the backward generation model based on the backward generation loss value, and determine a backward discrimination loss value corresponding to the backward discriminator based on the first modality image and the fourth modality image, optimize the model parameters of the backward discriminator based on the backward discrimination loss value to obtain a target backward generation model.
[0046] Among them, the target backward generation model is used to convert the second modality image into the first modality image. The backward generation loss value may refer to the generation deviation value between the second modality image and the fourth modality image. The backward discriminator may be used to discriminate the deviation value between the image generated by the backward generation model and the accurate first modality image. The backward discriminant loss value may refer to the deviation value between the image generated by the backward generation model and the accurate first modality image. The target backward generation model may refer to a generation model obtained by iterative training that can accurately realize image modality conversion.
[0047] Specifically, the image feature deviation between the second modality image and the fourth modality image is compared and the backward generation loss value corresponding to the backward generation model is calculated. The backward generation loss values corresponding to each training sample are accumulated and averaged, and the model parameters of the backward generation model are updated according to the average loss value. The image feature deviation between the first modality image and the fourth modality image is discriminated by a backward discriminator and the backward discriminant loss value corresponding to the backward discriminator is calculated. The backward discriminant loss values corresponding to each training sample are accumulated and averaged, and the model parameters of the backward discriminator are updated according to the average loss value. When the preset convergence condition is reached, the iterative training is stopped to obtain the trained target backward generation model.
[0048] The technical solution of the embodiment of the present invention is to obtain a first modality image and a second modality image corresponding to a training sample; input the first modality image into a forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and input the second modality image into a backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample; determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image, and optimize the model parameters of the forward generation model based on the forward generation loss value; and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image. The model parameters of the forward discriminator are optimized based on the forward discriminant loss value to obtain the target forward generative model; the target forward generative model is used to convert the first modality image into the second modality image; the backward generative loss value corresponding to the backward generative model is determined based on the second modality image and the fourth modality image, the model parameters of the backward generative model are optimized based on the backward generative loss value, and the backward discriminant loss value corresponding to the backward discriminator is determined based on the first modality image and the fourth modality image, the model parameters of the backward discriminator are optimized based on the backward discriminant loss value to obtain the target backward generative model; the target backward generative model is used to convert the second modality image into the first modality image. In this way, the target forward generative model and the target backward generative model can be accurately and conveniently determined, and accurate image modality conversion can be achieved through the target forward generative model or the target backward generative model, thereby improving the accuracy of image modality conversion.
[0049] On the basis of the above technical solution, the method also includes: obtaining a first modal image corresponding to the target user of the image modality to be converted; determining a second modal image corresponding to the target user based on the first modal image and a target forward generation model; or, obtaining a second modal image corresponding to the target user of the image modality to be converted; determining the first modal image corresponding to the target user based on the second modal image and a target backward generation model.
[0050] The target user may refer to a user having an image of one modality.
[0051] Specifically, the first modality image corresponding to the target user of the image modality to be converted is obtained; the first modality image is input into the target forward generation model for image modality conversion to obtain the second modality image corresponding to the target user, so that the target user does not need to shoot the image in the second modality, and the target forward generation model can be used to achieve accurate image modality conversion to obtain an accurate second modality image. Alternatively, the second modality image corresponding to the target user of the image modality to be converted is obtained; the second modality image is input into the target backward generation model for image modality conversion to obtain the first modality image corresponding to the target user, so that the target user does not need to shoot the image in the first modality, and the target backward generation model can be used to achieve accurate image modality conversion to obtain an accurate first modality image.
[0052] Embodiment 2
[0053] Figure 2 This is a flowchart of an image modality conversion method provided by Embodiment 2 of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining each loss value. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:
[0054] S210: Acquire a first modality image and a second modality image corresponding to the training sample.
[0055] S220, input the first modality image into the forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and input the second modality image into the backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample.
[0056] S230. Based on the trained classification model, perform block feature comparison on the first modality image and the third modality image to determine a first contrast loss value corresponding to the forward generation model; based on the forward discriminator, perform true prediction on the second modality image and the third modality image to determine a first adversarial loss value corresponding to the forward generation model; optimize model parameters of the forward generation model based on the first contrast loss value and the first adversarial loss value.
[0057] Among them, the classification model can be a ResNet classification model. The first contrast loss value can refer to the feature contrast deviation value between each block in the first modality image and each block in the third modality image. The first contrast loss value can be used to force the model to pay attention to the detail information between the image before and after the modality conversion. The first adversarial loss value can refer to the image contrast deviation value between the first modality image as a whole and the third modality image as a whole. The first adversarial loss value can be used to force the model to pay attention to the overall information between the images before and after the modality conversion.
[0058] Specifically, Figure 3 An example diagram of a recurrent adversarial generative network system is given. Figure 3 , the third modality image and the first modality image Divide it into m grid blocks in the same way, each grid is an image of a small area. Import the pre-trained ResNet classification model, select the publicly trained ResNet classification network model, download the model, and import the trained model into the ResNet model of this solution. and A total of m+1 small blocks of images are spliced in the channel dimension and input into the ResNet classification network for feature extraction. and A total of m+1 small blocks of images are input into the ResNet classification network for feature extraction, and all extracted features are concatenated in the channel dimension and then input into the fully connected layer for classification. The classification label is 1, which means that the ResNet model receives After that, you need to Find and The small image with the most similar content, that is, the first small image The advantage of this setting is that after the mode conversion and before mode conversion Image content details to ensure and Except for the different modalities, the content details of the images are the same. and A total of m+1 small blocks of images are input into the ResNet classification network for feature extraction, and all extracted features are concatenated in the channel dimension and then input into the fully connected layer for classification. The classification label is 2, which means that the ResNet model receives After that, you need to Find and The small image with the closest content, that is, the second small image And so on, until the comparison is complete. and Calculate the contrastive learning loss and use the cross entropy loss to calculate the classification loss of contrastive learning between the above blocks That is, the first contrast loss value. Calculate the adversarial loss and convert the third modality image Input to the forward discriminator In the above example, we calculate the loss of the prediction as true, which is recorded as That is, the first adversarial loss value. The first adversarial loss value is used to convert the forward generative model from one mode to another. The first contrast loss value and the first adversarial loss value corresponding to each training sample are accumulated and averaged to obtain the average loss value and the model parameters of the forward generative model are updated based on the average loss value.
[0059] S240, based on the forward discriminator, predict the second modal image and the third modal image as false, and determine the third adversarial loss value corresponding to the forward discriminator; based on the forward discriminator, predict the second modal image itself as true, and determine the fourth adversarial loss value corresponding to the forward discriminator; based on the third adversarial loss value and the fourth adversarial loss value, optimize the model parameters of the forward discriminator to obtain the target forward generation model.
[0060] The target forward generative model is used to convert the first modality image into the second modality image. The forward discriminator can use the second modality image for discrimination.
[0061] Specifically, the adversarial loss is calculated, and the third modality image Input to the forward discriminator In the above example, we calculate the loss of false predictions, which is recorded as That is the third adversarial loss value. Calculate the adversarial loss and transform the second modality image Input to the discriminator In the above example, we calculate the loss of the prediction as true, which is recorded as That is, the fourth adversarial loss value. The third adversarial loss value and the fourth adversarial loss value corresponding to each training sample are accumulated and averaged to obtain an average loss value, and the model parameters of the forward discriminator are updated based on the average loss value.
[0062] S250, based on the trained classification model, perform block feature comparison on the second modality image and the fourth modality image to determine the second contrast loss value corresponding to the backward generation model; based on the backward discriminator, perform true discrimination on the first modality image and the fourth modality image to determine the second adversarial loss value corresponding to the backward generation model; optimize the model parameters of the backward generation model based on the second contrast loss value and the second adversarial loss value.
[0063] Among them, the classification model can be a ResNet classification model. The second contrast loss value can refer to the feature contrast deviation value between each block in the second modality image and each block in the fourth modality image. The second contrast loss value can be used to force the model to pay attention to the detail information between the image before and after the modality conversion. The second adversarial loss value can refer to the image contrast deviation value between the second modality image as a whole and the fourth modality image as a whole. The second adversarial loss value can be used to force the model to pay attention to the overall information between the images before and after the modality conversion.
[0064] Specifically, in addition to the forward generation process, this embodiment also adds the same contrast learning to the backward generation model to ensure that the generated image has better details. and the second modality image Divide it into m grid blocks in the same way, each grid is an image of a small area. Import the pre-trained ResNet classification model, select the publicly trained ResNet classification network model, download the model, and import the trained model into the ResNet model of this solution. and the second modality image The m blocks of images are input into the classification convolutional model to compare the similarities between different small block models. Calculate the contrastive learning loss and use the cross entropy loss to calculate the classification loss of the contrastive learning between the above blocks. That is, the second contrast loss value. Calculate the adversarial loss and convert the fourth modality image Input to the backward discriminator In the above example, we calculate the loss of the prediction as true, which is recorded as That is, the second adversarial loss value. The second adversarial loss value is used to convert the backward generation model from one mode to another. The second contrast loss value and the second adversarial loss value corresponding to each training sample are accumulated and averaged to obtain the average loss value and the model parameters of the backward generation model are updated based on the average loss value.
[0065] S260. Based on the prediction of the first modal image and the fourth modal image as false by the backward discriminator, a fifth adversarial loss value corresponding to the backward discriminator is determined; based on the prediction of the first modal image itself as true by the backward discriminator, a sixth adversarial loss value corresponding to the backward discriminator is determined; based on the fifth adversarial loss value and the sixth adversarial loss value, model parameters of the backward discriminator are optimized to obtain a target backward generation model.
[0066] The target backward generative model is used to convert the second modality image into the first modality image. The backward discriminator can use the first modality image for discrimination.
[0067] Specifically, the adversarial loss is calculated, and the fourth modality image Input to the backward discriminator In the above example, we calculate the loss of false predictions, which is recorded as That is the fifth adversarial loss value. Calculate the adversarial loss and transform the first modality image Input to the discriminator In the above example, we calculate the loss of the prediction as true, which is recorded as That is, the sixth adversarial loss value. The fifth adversarial loss value and the sixth adversarial loss value corresponding to each training sample are accumulated and averaged to obtain the average loss value, and the model parameters of the backward discriminator are updated based on the average loss value.
[0068] It should be noted that in this embodiment, contrastive learning is combined with the recurrent adversarial generative network, thereby increasing the constraints on the recurrent adversarial generative network to avoid generating unrealistic objects, and increasing more local detail information, so that the recurrent generative network can maintain the content of the input image and only change the style, i.e., the modality, of the input image.
[0069] The technical solution of the embodiment of the present invention determines the first contrast loss value corresponding to the forward generation model by performing block feature comparison on the first modality image and the third modality image based on the trained classification model, so as to focus on the local details of the generated image; determines the first adversarial loss value corresponding to the forward generation model by discriminating the second modality image and the third modality image as true based on the forward discriminator; optimizes the model parameters of the forward generation model based on the first contrast loss value and the first adversarial loss value, so as to jointly optimize the forward generation model based on the first contrast loss value focusing on the local details of the generated image and the first adversarial loss value focusing on the global details of the generated image, so as to avoid generating unrealistic details, and thereby obtain a generation model that can perform modal conversion more accurately. Based on the forward discriminator, the second modality image and the third modality image are predicted to be false, and the third adversarial loss value corresponding to the forward discriminator is determined; based on the forward discriminator, the second modality image itself is predicted to be true, and the fourth adversarial loss value corresponding to the forward discriminator is determined; based on the third adversarial loss value and the fourth adversarial loss value, the model parameters of the forward discriminator are optimized to obtain a target forward generation model; the target forward generation model is used to convert the first modality image into the second modality image. Based on the trained classification model, the second modality image and the fourth modality image are subjected to block feature comparison, and the second contrast loss value corresponding to the backward generative model is determined, so that the local details of the generated image can be focused on; based on the backward discriminator, the first modality image and the fourth modality image are predicted to be true, and the second adversarial loss value corresponding to the backward generative model is determined; based on the second contrast loss value and the second adversarial loss value, the model parameters of the backward generative model are optimized, so that the backward generative model is jointly optimized based on the second contrast loss value focusing on the local details of the generated image and the second adversarial loss value focusing on the global details of the generated image, so as to avoid the generation of unrealistic details, and thereby obtain a generative model that can perform modal conversion more accurately. Based on the backward discriminator, the first modality image and the fourth modality image are predicted to be false, and the fifth adversarial loss value corresponding to the backward discriminator is determined; based on the backward discriminator, the first modality image itself is predicted to be true, and the sixth adversarial loss value corresponding to the backward discriminator is determined; based on the fifth adversarial loss value and the sixth adversarial loss value, the model parameters of the backward discriminator are optimized to obtain a target backward generation model; the target backward generation model is used to convert the second modality image into the first modality image, and further determine the target forward generation model and the target backward generation model that can more accurately realize the modality conversion, and accurate image modality conversion can be achieved through the target forward generation model or the target backward generation model, thereby further improving the accuracy of image modality conversion.
[0070] On the basis of the above technical solution, the method also includes: inputting the third modality image into the backward generation model for modality conversion processing to obtain a fifth modality image corresponding to the training sample, and inputting the first modality image into the backward generation model for modality conversion processing to obtain a seventh modality image corresponding to the training sample; inputting the fourth modality image into the forward generation model for modality conversion processing to obtain a sixth modality image corresponding to the training sample, and inputting the second modality image into the forward generation model for modality conversion processing to obtain an eighth modality image corresponding to the training sample; determining the first modality image based on the L1 norm loss function The first reconstruction loss value between the second modality image and the fifth modality image, and the second reconstruction loss value between the first modality image and the seventh modality image are determined, and the model parameters of the forward generation model are optimized based on the first reconstruction loss value and the second reconstruction loss value to obtain the optimized target forward generation model; the third reconstruction loss value between the second modality image and the sixth modality image, and the fourth reconstruction loss value between the second modality image and the eighth modality image are determined based on the L1 norm loss function, and the model parameters of the backward generation model are optimized based on the third reconstruction loss value and the fourth reconstruction loss value to obtain the optimized target backward generation model.
[0071] The fifth modality image may refer to an image of the first modality converted into an image of the second modality and then converted back into an image of the first modality. The fifth modality image may be represented as The sixth modality image may refer to an image of the second modality converted into an image of the first modality and then converted back into an image of the second modality. The sixth modality image may be represented as The seventh modality image may refer to an image of the first modality converted into an image of the first modality. The seventh modality image may be represented as The eighth modality image may refer to an image of the second modality converted into an image of the second modality. The eighth modality image may be represented as
[0072] Specifically, the second modality image Input into the forward generation model F of the recurrent adversarial generation network and output the eighth modality image The purpose of this process is to ensure that no matter what is input into the forward generation model F, the final output of the forward generation model F must be an image of modality B. Input into the backward generation model B of the recurrent adversarial generation network and output the seventh modality image The purpose of this process is to ensure that no matter what is input into the backward generation model B, the final output of the backward generation model B must be an image of modality A. Input into the backward generation model B of the recurrent adversarial generation network and output the fifth modality image This process ensures that after modality A generates modality B, the original modality A can still be regenerated through the inverse process. Input into the forward generation model F of the recurrent adversarial generation network and output the sixth modality image This process ensures that after modality B generates modality A, the original modality B can still be regenerated through the inverse process. Calculate the reconstruction loss and calculate the fifth modality image based on L1 and the first modality image The loss and calculate the seventh modality image and the first modality image The loss is recorded as Calculate the reconstruction loss and calculate the sixth modality image based on L1 and the second modality image The loss is calculated for the eighth modality image and the second modality image The loss is recorded as The first reconstruction loss value and the second reconstruction loss value corresponding to each training sample are accumulated and averaged to obtain an average loss value, and the model parameters of the forward generation model are updated based on the average loss value. The third reconstruction loss value and the fourth reconstruction loss value corresponding to each training sample are accumulated and averaged to obtain an average loss value, and the model parameters of the backward generation model are updated based on the average loss value.
[0073] The following is an embodiment of an image modality conversion device provided by an embodiment of the present invention. The device and the image modality conversion methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the image modality conversion device, reference can be made to the embodiment of the above-mentioned image modality conversion method.
[0074] Embodiment 3
[0075] Figure 4 This is a schematic diagram of the structure of an image mode conversion device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a modality image acquisition module 410, a modality image conversion module 420, a forward generation model training module 430 and a backward generation model training module 440.
[0076] Among them, the modality image acquisition module 410 is used to acquire the first modality image and the second modality image corresponding to the training sample; the modality image conversion module 420 is used to input the first modality image into the forward generation model for modality conversion processing to obtain the third modality image corresponding to the training sample, and input the second modality image into the backward generation model for modality conversion processing to obtain the fourth modality image corresponding to the training sample; the forward generation model training module 430 is used to determine the forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine the forward discriminator corresponding to the second modality image and the third modality image. A forward discriminant loss value, based on which the model parameters of the forward discriminant are optimized to obtain a target forward generation model; the target forward generation model is used to convert the first modality image into the second modality image; a backward generation model training module 440 is used to determine a backward generation loss value corresponding to the backward generation model based on the second modality image and the fourth modality image, optimize the model parameters of the backward generation model based on the backward generation loss value, and determine a backward discriminant loss value corresponding to the backward discriminator based on the first modality image and the fourth modality image, optimize the model parameters of the backward discriminator based on the backward discriminant loss value to obtain a target backward generation model; the target backward generation model is used to convert the second modality image into the first modality image.
[0077] The technical solution of the embodiment of the present invention is to obtain a first modality image and a second modality image corresponding to a training sample; input the first modality image into a forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and input the second modality image into a backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample; determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image, and optimize the model parameters of the forward generation model based on the forward generation loss value; and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image. The model parameters of the forward discriminator are optimized based on the forward discriminant loss value to obtain the target forward generative model; the target forward generative model is used to convert the first modality image into the second modality image; the backward generative loss value corresponding to the backward generative model is determined based on the second modality image and the fourth modality image, the model parameters of the backward generative model are optimized based on the backward generative loss value, and the backward discriminant loss value corresponding to the backward discriminator is determined based on the first modality image and the fourth modality image, the model parameters of the backward discriminator are optimized based on the backward discriminant loss value to obtain the target backward generative model; the target backward generative model is used to convert the second modality image into the first modality image. In this way, the target forward generative model and the target backward generative model can be accurately and conveniently determined, and accurate image modality conversion can be achieved through the target forward generative model or the target backward generative model, thereby improving the accuracy of image modality conversion.
[0078] On the basis of the above technical solution, the "determining the forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image" in the forward generation model training module 430 is specifically used for: performing block feature comparison on the first modality image and the third modality image based on the trained classification model, and determining the first comparison loss value corresponding to the forward generation model; and determining the first adversarial loss value corresponding to the forward generation model based on the forward discriminator to discriminate whether the second modality image and the third modality image are predicted to be true.
[0079] On the basis of the above technical solution, the "determining the backward generation loss value corresponding to the backward generation model based on the second modality image and the fourth modality image" in the backward generation model training module 440 is specifically used for: performing block feature comparison on the second modality image and the fourth modality image based on the trained classification model, and determining the second comparison loss value corresponding to the backward generation model; and determining the second adversarial loss value corresponding to the backward generation model based on the prediction of the first modality image and the fourth modality image as true by the backward discriminator.
[0080] On the basis of the above technical solution, the "determining the forward discriminant loss value corresponding to the forward discriminator based on the second modality image and the third modality image" in the forward generation model training module 430 is specifically used for: determining the third adversarial loss value corresponding to the forward discriminator based on the forward discriminator predicting the second modality image and the third modality image as false; determining the fourth adversarial loss value corresponding to the forward discriminator based on the forward discriminator predicting the second modality image itself as true.
[0081] On the basis of the above technical solution, the "determining the backward discriminant loss value corresponding to the backward discriminator based on the first modal image and the fourth modal image" in the backward generation model training module 440 is specifically used for: determining the fifth adversarial loss value corresponding to the backward discriminator based on the prediction of the first modal image and the fourth modal image as false by the backward discriminator; determining the sixth adversarial loss value corresponding to the backward discriminator based on the prediction of the first modal image itself as true by the backward discriminator.
[0082] On the basis of the above technical solution, the device also includes:
[0083] A first modality image acquisition module, used to acquire a first modality image corresponding to a target user of an image modality to be converted;
[0084] A second modality image generation module, used to determine a second modality image corresponding to a target user based on the first modality image and a target forward generation model;
[0085] Or, a second modality image acquisition module, used to acquire a second modality image corresponding to a target user of the image modality to be converted;
[0086] The first modality image generation module is used to determine the first modality image corresponding to the target user based on the second modality image and the target backward generation model.
[0087] On the basis of the above technical solution, the device also includes:
[0088] A first image modality conversion module is used to input the third modality image into the backward generation model for modality conversion processing to obtain a fifth modality image corresponding to the training sample, and input the first modality image into the backward generation model for modality conversion processing to obtain a seventh modality image corresponding to the training sample;
[0089] A second image modality conversion module is used to input the fourth modality image into the forward generation model for modality conversion processing to obtain a sixth modality image corresponding to the training sample, and input the second modality image into the forward generation model for modality conversion processing to obtain an eighth modality image corresponding to the training sample;
[0090] a forward generation model optimization module, used to determine a first reconstruction loss value between the first modality image and the fifth modality image, and a second reconstruction loss value between the first modality image and the seventh modality image based on an L1 norm loss function, and optimize model parameters of the forward generation model based on the first reconstruction loss value and the second reconstruction loss value to obtain an optimized target forward generation model;
[0091] The backward generation model optimization module is used to determine a third reconstruction loss value between the second modality image and the sixth modality image, and a fourth reconstruction loss value between the second modality image and the eighth modality image based on the L1 norm loss function, and optimize the model parameters of the backward generation model based on the third reconstruction loss value and the fourth reconstruction loss value to obtain an optimized target backward generation model.
[0092] The image modality conversion device provided in the embodiment of the present invention can execute the image modality conversion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the image modality conversion method.
[0093] It is worth noting that in the above-mentioned embodiment of image modality conversion, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0094] Embodiment 4
[0095] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0096] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0097] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0098] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image modality conversion method.
[0099] In some embodiments, the image modality conversion method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image modality conversion method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image modality conversion method in any other appropriate manner (e.g., by means of firmware).
[0100] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0101] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0102] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0103] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0104] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0105] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0106] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the image modality conversion method provided in any embodiment of the present application.
[0107] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and also conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The program product and the image modality conversion method disclosed in each embodiment of the present application belong to the same inventive concept, so it will not be repeated here.
[0108] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0109] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. An image modality conversion method, characterized in that: include: Obtaining a first modality image and a second modality image corresponding to the training sample; Inputting the first modality image into a forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and inputting the second modality image into a backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample; Determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image, optimize the model parameters of the forward discriminator based on the forward discrimination loss value to obtain a target forward generation model; the target forward generation model is used to convert the first modality image into the second modality image; Based on the second modality image and the fourth modality image, a backward generation loss value corresponding to the backward generation model is determined, and model parameters of the backward generation model are optimized based on the backward generation loss value. Based on the first modality image and the fourth modality image, a backward discrimination loss value corresponding to the backward discriminator is determined, and model parameters of the backward discriminator are optimized based on the backward discrimination loss value to obtain a target backward generation model; the target backward generation model is used to convert the second modality image into the first modality image.
2. The method according to claim 1, characterized in that The determining, based on the first modality image and the third modality image, a forward generation loss value corresponding to the forward generation model includes: Performing block feature comparison on the first modality image and the third modality image based on the trained classification model to determine a first contrast loss value corresponding to the forward generation model; Based on the forward discriminator, the second modality image and the third modality image are predicted to be true, and a first adversarial loss value corresponding to the forward generation model is determined.
3. The method according to claim 1, characterized in that The determining, based on the second modality image and the fourth modality image, a backward generation loss value corresponding to the backward generation model includes: Performing block feature comparison on the second modality image and the fourth modality image based on the trained classification model to determine a second contrast loss value corresponding to the backward generation model; Based on the backward discriminator, the first modality image and the fourth modality image are predicted to be true, and a second adversarial loss value corresponding to the backward generation model is determined.
4. The method according to claim 1, characterized in that The determining, based on the second modality image and the third modality image, a forward discriminant loss value corresponding to the forward discriminator includes: Determine, based on the forward discriminator, whether the second modality image and the third modality image are predicted to be false, and determine a third adversarial loss value corresponding to the forward discriminator; Based on the forward discriminator's judgment that the second modal image itself is predicted to be true, a fourth adversarial loss value corresponding to the forward discriminator is determined.
5. The method according to claim 1, characterized in that: The determining, based on the first modality image and the fourth modality image, a backward discriminant loss value corresponding to the backward discriminator includes: Determining, based on the backward discriminator, that the first modality image and the fourth modality image are predicted to be false, a fifth adversarial loss value corresponding to the backward discriminator; Based on the judgment that the backward discriminator predicts the first modal image itself as true, a sixth adversarial loss value corresponding to the backward discriminator is determined.
6. The method according to claim 1, characterized in that The method further comprises: Acquire a first modality image corresponding to a target user of an image modality to be converted; Determining a second modality image corresponding to the target user based on the first modality image and a target forward generation model; Or, obtaining a second modality image corresponding to a target user of the image modality to be converted; A first modality image corresponding to the target user is determined based on the second modality image and a target backward generation model.
7. The method according to claim 1, characterized in that The method further comprises: Inputting the third modality image into the backward generation model for modality conversion processing to obtain a fifth modality image corresponding to the training sample, and inputting the first modality image into the backward generation model for modality conversion processing to obtain a seventh modality image corresponding to the training sample; Inputting the fourth modality image into the forward generation model for modality conversion processing to obtain a sixth modality image corresponding to the training sample, and inputting the second modality image into the forward generation model for modality conversion processing to obtain an eighth modality image corresponding to the training sample; Determine a first reconstruction loss value between the first modality image and the fifth modality image, and a second reconstruction loss value between the first modality image and the seventh modality image based on an L1 norm loss function, and optimize model parameters of the forward generation model based on the first reconstruction loss value and the second reconstruction loss value to obtain an optimized target forward generation model; Based on the L1 norm loss function, a third reconstruction loss value between the second modality image and the sixth modality image, and a fourth reconstruction loss value between the second modality image and the eighth modality image are determined, and model parameters of the backward generation model are optimized based on the third reconstruction loss value and the fourth reconstruction loss value to obtain an optimized target backward generation model.
8. An image mode conversion device, characterized in that: The device comprises: A modality image acquisition module, used to acquire a first modality image and a second modality image corresponding to a training sample; A modality image conversion module, configured to input the first modality image into a forward generation model for modality conversion processing to obtain a third modality image corresponding to the training sample, and input the second modality image into a backward generation model for modality conversion processing to obtain a fourth modality image corresponding to the training sample; A forward generation model training module, used to determine a forward generation loss value corresponding to the forward generation model based on the first modality image and the third modality image, optimize the model parameters of the forward generation model based on the forward generation loss value, and determine a forward discrimination loss value corresponding to a forward discriminator based on the second modality image and the third modality image, optimize the model parameters of the forward discriminator based on the forward discrimination loss value, to obtain a target forward generation model; the target forward generation model is used to convert the first modality image into the second modality image; A backward generation model training module is used to determine a backward generation loss value corresponding to the backward generation model based on the second modality image and the fourth modality image, optimize the model parameters of the backward generation model based on the backward generation loss value, and determine a backward discrimination loss value corresponding to a backward discriminator based on the first modality image and the fourth modality image, optimize the model parameters of the backward discriminator based on the backward discrimination loss value to obtain a target backward generation model; the target backward generation model is used to convert the second modality image into the first modality image.
9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image modality conversion method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image modality conversion method as described in any one of claims 1 to 7 is implemented.