Image generation method, apparatus, device, storage medium, and computer program product

Abstract

The application discloses an image generation method and device, equipment, storage medium and computer program product, and relates to the field of machine learning. The method comprises the following steps: acquiring a first modality image; performing modality conversion on the first modality image through a first candidate network to obtain a first generated image; performing modality restoration on the first generated image through a second candidate network to obtain a first restored image; acquiring a constraint loss value based on the first generated image and the first restored image; and training the first candidate network based on the constraint loss value to obtain an image conversion network. That is, the constraint loss value corresponding to the first candidate network is determined by performing modality conversion on the first modality image and then performing modality restoration to obtain the first generated image and the first restored image, respectively, so that the training mode of the first candidate network can improve the accuracy of modality conversion in a three-dimensional image space.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and in particular to an image generation method, device, equipment, storage medium and computer program product. BACKGROUND

[0002] Image synthesis is used for image conversion between different modal images, where different modal images refer to images obtained in different ways, such as X-ray images obtained by X-ray (X-RAY) and magnetic resonance imaging (MRI) images obtained by nuclear magnetic resonance, which correspond to two different modal images, or different modal images refer to different styles of images.

[0003] In related technologies, a generative adversarial network (GAN) is usually trained to obtain an image generation model for generating images of different modalities. In the training process, the GAN network is trained by sample images and reference images, where the sample images and the reference images are a sample image group with a matching relationship determined in advance.

[0004] However, in the above method, the GAN network is trained by obtaining a sample image group with a matching relationship. Since the samples used for training need to have a matching relationship, the sample image group is difficult to obtain. In addition, in actual application, different modal images have large differences, which leads to low accuracy of the output results of the GAN network and poor performance of the model. SUMMARY

[0005] The embodiments of the present application provide an image generation method, device, equipment, storage medium and computer program product, which can improve the accuracy of the generated target three-dimensional image. The technical solution is as follows:

[0006] In one aspect, an image generation method is provided, which includes:

[0007] obtaining a first modal image, the first modal image corresponding to a first modality;

[0008] performing modality conversion on the first modal image by a first candidate network to obtain a first generated image, the first generated image corresponding to a second modality, and the first generated image being a three-dimensional image, the first modality and the second modality being different modalities;

[0009] performing modality restoration on the first generated image by a second candidate network to obtain a first restored image, the first restored image corresponding to the first modality;

[0010] obtain a constraint loss value based on the first generated image and the first restored image, the constraint loss value being used to indicate a mapping loss when the first candidate network maps the first modality image to a three-dimensional image space;

[0011] train the first candidate network based on the constraint loss value to obtain an image conversion network, the image conversion network being used to perform modality conversion on an image belonging to the first modality to obtain a three-dimensional image belonging to the second modality.

[0012] In another aspect, an image generation apparatus is provided, and the apparatus comprises:

[0013] an obtaining module configured to obtain a first modality image, the first modality image corresponding to a first modality;

[0014] a conversion module configured to perform modality conversion on the first modality image by using a first candidate network to obtain a first generated image, the first generated image corresponding to a second modality, and the first generated image being a three-dimensional image, the first modality and the second modality being different modalities;

[0015] a restoring module configured to perform modality restoration on the first generated image by using a second candidate network to obtain a first restored image, the first restored image corresponding to the first modality;

[0016] the obtaining module is further configured to obtain a constraint loss value based on the first generated image and the first restored image, the constraint loss value being used to indicate a mapping loss when the first candidate network maps the first modality image to a three-dimensional image space;

[0017] a training module configured to train the first candidate network based on the constraint loss value to obtain an image conversion network, the image conversion network being used to perform modality conversion on an image belonging to the first modality to obtain a three-dimensional image belonging to the second modality.

[0018] In another aspect, a computer device is provided, and the computer device comprises a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement an image generation method as described in any of the embodiments of the present application.

[0019] In another aspect, a computer-readable storage medium is provided, and the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by a processor to implement an image generation method as described in any of the embodiments of the present application.

[0020] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image generation method of any of the above embodiments.

[0021] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0022] The first modality image is respectively modality converted and modality restored by the first candidate network and the second candidate network, and the first generated image and the first restored image are respectively obtained, wherein the first candidate network is used to convert the first modality image into a three-dimensional image of the second modality, and the constraint loss value corresponding to the first candidate network in the three-dimensional image space is determined according to the first generated image and the first restored image, so that the training manner of the first candidate network can make the effect of the target three-dimensional image generated by the finally trained image conversion network better, that is, the training effect of the image conversion network can be improved by introducing the three-dimensional image space in the first candidate network, so that the accuracy of the output target three-dimensional image is higher.

[0023] In the medical field, due to the more diversified modalities of medical images, the information focus of different modalities is different, so the medical image is converted into a three-dimensional medical image by the image conversion network, which can better perform image segmentation on the three-dimensional medical image for medical diagnosis. In the process of training the first candidate network, the first generated image and the first restored image corresponding to the first modality image are obtained, so as to determine the constraint loss value corresponding to the first candidate network, which is used to determine the mapping loss of different features corresponding to the first candidate network in the three-dimensional image space in different scenes, so as to train the first candidate network in a targeted manner, and obtain the image conversion network finally meeting the current scene, which is suitable for improving the accuracy of modality conversion of medical images, so as to better perform subsequent medical diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a technical schematic diagram of the image generation method provided by an exemplary embodiment of the present application;

[0026] Figure 2 is a schematic diagram of an implementation environment provided by an example embodiment of the present application;

[0027] Figure 3 is a flowchart of an image generation method provided by an example embodiment of the present application;

[0028] Figure 4 is a flowchart of an image generation method provided by another example embodiment of the present application;

[0029] Figure 5 is a flowchart of an image generation method provided by another example embodiment of the present application;

[0030] Figure 6 is a schematic diagram of an image generation method provided by an example embodiment of the present application;

[0031] Figure 7 is a schematic diagram of a brain image generation process provided by an example embodiment of the present application;

[0032] Figure 8 is a structural block diagram of an image generation apparatus provided by an example embodiment of the present application;

[0033] Figure 9 is a structural block diagram of an image generation apparatus provided by another example embodiment of the present application;

[0034] Figure 10 is a schematic diagram of a server structure provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0036] First, the terms involved in the embodiments of the present application will be briefly introduced.

[0037] Modality: every source or form of information can be a modality, such as: human has tactile sense and auditory sense; information media includes voice, video, text, etc.; various sensors, including: radar, infrared, accelerometer, etc., each of the above can be called a modality.

[0038] In the embodiments of the present application, different forms of images are taken as different modal images, such as X-ray images, CT images, and MRI images. The image information reflected by different modal images has different focuses. X-ray images can clearly observe bones, CT images can reflect tissue and bleeding conditions, and MRI images are suitable for observing soft tissues.

[0039] Generative Adversarial Networks (GAN): a kind of unsupervised deep learning model. The GAN network includes at least two modules: a generative model and a discriminative model. In the training process of the model, the two models learn from each other through mutual game to train the GAN network and improve the accuracy of the model output results.

[0040] First, in the related art, please refer to Figure 1 , which shows the image generation related technology diagram provided by an example embodiment of the present application, as Figure 1 shown, for an image generation training task, a candidate generation network 100 is set to train it. First, a sample image 101 and a reference image 102 are obtained, wherein the sample image 101 and the reference image 102 are a set of sample images with matching relationship. By inputting the sample image 101 into the generator 103 and adding random noise 104, the generated image 105 is output. The reference image 102 and the generated image 105 are input into the discriminator 106, and the generated image 105 and the reference image 102 corresponding to the discrimination result 107 are output. If the generated image 105 is consistent with the reference image 102, the discrimination result 107 is "1", if the generated image 105 is inconsistent with the reference image 102, the discrimination result 107 is "0", and the candidate generation network 100 is trained according to the discrimination result 107. Finally, the target generation network 108 is obtained.

[0041] In the above technology, the candidate generation network is trained by using a preconfigured sample image set. Because the difference between images is large, it is difficult to obtain a sample image set with matching relationship. At the same time, the generalization performance of the target generation network obtained by the training process is poor in actual application, resulting in low accuracy of the output results and unable to meet the task requirements.

[0042] The image generation method provided in the application can perform modality conversion and modality restoration on the first modality image by the first candidate network and the second candidate network, respectively obtain the first generated image belonging to the second modality and the first restored image belonging to the first modality, determine the constraint loss value corresponding to the first candidate network in the three-dimensional image space according to the first generated image and the first restored image, and thus train the first candidate network in the manner, so that the effect of the target three-dimensional image generated by the finally trained image conversion network is better, that is, the training effect of the image conversion network can be improved by introducing the three-dimensional image space in the first candidate network, so that the accuracy of the output target three-dimensional image is higher.

[0043] Secondly, the implementation environment involved in the embodiment of the application is described, and schematically, please refer to Figure 2 The implementation environment involves a terminal 210 and a server 220, and the terminal 210 and the server 220 are connected through a communication network 230.

[0044] In some embodiments, the terminal 210 sends an image generation request to the server 220, wherein the image generation request contains an original image used for modality conversion, the server 220 receives the image generation request sent from the terminal 210, performs modality conversion on the original image, generates a target three-dimensional image corresponding to the original image, and feeds back the target three-dimensional image to the terminal 210.

[0045] The image conversion network 221 is obtained by inputting the first modality image 222 into the first candidate network 223 to generate the first generated image 224, inputting the first generated image 224 into the second candidate network 225 to obtain the first restored image 226, determining the constraint loss value according to the first generated image 224 and the first restored image 226 to train the first candidate network 223, and finally obtaining the image conversion network 221.

[0046] The terminal 210 described above can be a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, a smart television, a smart vehicle, and various forms of terminal devices, and the embodiment of the application does not limit this.

[0047] It is worth noting that the server 220 described above can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, and basic cloud computing services such as platform.

[0048] Among them, cloud technology refers to the unified hardware, software, network and other series of resources in the wide area network or local area network to realize the calculation, storage, processing and sharing of data. Cloud technology is based on the network technology, information technology, integration technology, management platform technology, application technology and other technologies applied in the cloud computing business model, which can form a resource pool, and can be used as needed, flexible and convenient. Cloud computing technology will become an important support. The background service of the technical network system needs a lot of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, every item may have its own identification mark in the future, and it needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support, which can only be realized through cloud computing.

[0049] In some embodiments, the server 220 described above can also be implemented as a node in a blockchain system.

[0050] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the brain images used for training in the present application are obtained under sufficient authorization.

[0051] In combination with the above-mentioned term introduction and implementation environment, the application scenarios of the present application are exemplified.

[0052] 1. Applied in medical scenarios. Taking the current task of generating a three-dimensional brain image as an example, a sample brain image contained in a public data set is obtained, the sample brain image is modality converted by a first candidate network to obtain a brain generated image, the brain generated image is a three-dimensional image corresponding to the sample brain image, the brain generated image is modality restored by a second candidate network to obtain a brain restored image, a constraint loss value is determined according to the brain generated image and the brain restored image, the first candidate network is trained according to the constraint loss value to obtain an image conversion network, which is used to convert a brain image in a first modality into a three-dimensional brain image in a second modality, which is used for subsequent image segmentation of the brain image, auxiliary diagnosis and treatment, etc. For example, converting a T1 weighted image into a T2 weighted MRI image.

[0053] 2. Applied in advertising scenarios.In order to obtain more advertising materials, the sample advertising materials of the first modality are input into the first candidate network for modality conversion to generate sample generated images of the second modality, the sample generated images are corresponding three-dimensional images under the second modality, the sample generated images are input into the second candidate network for modality restoration to obtain sample restored images of the first modality, the constraint loss value is determined according to the sample restored images and the sample generated images, the first candidate network is trained to obtain the material conversion network, and in the application process, the target material is input into the material conversion network, and three-dimensional material images under different modalities corresponding to the target material are output, so that the generation efficiency of the material is improved.

[0054] It is worth noting that the above application scenarios are only illustrative examples, and the image generation method in the embodiment of the present application is not limited to the application scenarios. In addition, the classification prediction method can also be used in medical image alignment, image style transfer and other application scenarios.

[0055] Illustratively, the image generation method provided by the present application is described. Figure 3 which shows the flowchart of the image generation method provided by an example embodiment of the present application. The method can be executed by a terminal, or by a server, or by a terminal and a server together. In this embodiment, the method is executed by the server, as shown in Figure 3 The method comprises the following steps.

[0056] Step 301: obtaining a first modality image.

[0057] The first modality image corresponds to the first modality.

[0058] Illustratively, the first modality image is an image corresponding to the first modality.

[0059] Optionally, the first modality is determined according to the style type of the image, such as: the image corresponding to the first modality is a cartoon image, and the images under other modalities are realistic portrait images; or the first modality is determined according to the color value of the image, such as: the image corresponding to the first modality is a grayscale image; or the first modality is determined according to the specific content of the image, such as: the first modality image content includes a cat, which is not limited.

[0060] Illustratively, the first modality image is a sample training image in a public training data set, which is not limited.

[0061] Optionally, the way of obtaining the first modality image comprises obtaining from a locally stored image data set; or downloading from a public training data set, which is not limited.

[0062] Step 302: performing modality conversion on the first modality image through a first candidate network to obtain a first generated image.

[0063] The first generated image corresponds to the second modality, and the first generated image is a three-dimensional image. The first modality and the second modality are different modalities.

[0064] In some embodiments, the first candidate network is a network for modal conversion of the input first modality image.

[0065] Optionally, the first modality and the second modality differ in that the images corresponding to each of them are from different data sources, such as: the images of the first modality are downloaded from a website, and the images of the second modality are obtained by shooting; or, the first modality and the second modality differ in that the images corresponding to each of them are of different data content types, such as: the images of the first modality are cartoon images, and the images of the second modality are real images, without limitation.

[0066] In some embodiments, the first generated image is an image generated by the first candidate network after modal conversion of the first modality image corresponding to the first modality, and the first generated image corresponds to the second modality, indicating that the first generated image and the first modality image correspond to different modalities.

[0067] Optionally, the first generated image and the first modality image do not have a content correspondence relationship, such as: the first modality image contains a dog, but the first generated image contains a zebra; or, the first generated image and the first modality image have an image content correspondence relationship, that is, the image contents corresponding to the first generated image and the first modality image are consistent, such as: the image contents contained in the first modality image and the first generated image are both cats, without limitation.

[0068] Illustratively, the modal conversion mode includes the following conversion modes:

[0069] 1. The modal conversion includes pixel point sequence reorganization, that is, the distribution sequence of each pixel point in the first modality image is obtained, and the modal conversion is realized by adjusting the distribution sequence of each pixel point;

[0070] 2. The modal conversion includes dimension conversion, such as: the first modality image corresponds to a first type of dimension image (such as: a two-dimensional image), and the first generated image obtained by mapping it to a second type of dimension space is a second type of dimension image (such as: a three-dimensional image);

[0071] 3. The modal conversion includes feature fusion, that is, the element features corresponding to each pixel point of the first modality image are obtained, and the modal conversion is realized by fusing each element feature;

[0072] 4. The modal conversion includes threshold conversion, that is, by obtaining the threshold information of each region corresponding to the first modal image, the modal conversion is realized by adjusting the threshold information of each region, such as: the first modal image contains region a and region b, and the color of region a is more prominent. After threshold conversion, the first generated image region b is generated.

[0073] It is worth noting that the above modal conversion mode is only an illustrative example, and the embodiments of the present application are not limited thereto.

[0074] Optionally, the modal conversion is random conversion, or conversion according to a specified rule, such as: converting a real image into a cartoon image, which is not limited herein.

[0075] Optionally, the first modal and the second modal have an association relationship, such as: the first modal and the second modal are both brain images, but correspond to different types (MRI image and CT image), or the first modal and the second modal do not have an association relationship, which is not limited herein.

[0076] Step 303, modal restoration is performed on the first generated image by the second candidate network to obtain a first restored image.

[0077] The first restored image corresponds to the first modal.

[0078] Illustratively, the first restored image is an image corresponding to the first modal after modal restoration.

[0079] Optionally, the first restored image is the same as or different from the first modal image, which is not limited herein.

[0080] Illustratively, the modal restoration mode includes at least one of the following modes:

[0081] 1. The modal restoration includes pixel point distribution restoration, that is, the distribution of each pixel point in the first generated image is obtained, and the modal restoration is realized by arranging and recombining each pixel point distribution corresponding to the first modal;

[0082] 2. The modal restoration includes dimension reduction, that is, the first restored image obtained by mapping the first generated image to a first type dimension space is a first type dimension image (such as: a two-dimensional image);

[0083] 3. The modal restoration includes feature decomposition, that is, the image features corresponding to the first generated image are obtained, and the modal restoration is realized by structurally decomposing the image features;

[0084] 4. The modal restoration includes threshold restoration, that is, by obtaining the threshold information of each region corresponding to the first modal image, the modal restoration is realized by restoring the threshold information of each region.

[0085] It should be noted that the above modal reduction mode is only an illustrative example, and the embodiments of the present application are not limited thereto.

[0086] Optionally, the first candidate network and the second candidate network are networks of the same architecture type, or the first candidate network and the second candidate network are networks of different architecture types, and the embodiments of the present application are not limited thereto.

[0087] In step 304, a constraint loss value is obtained based on the first generated image and the first restored image.

[0088] The constraint loss value is used to indicate a mapping loss of the first candidate network mapping the first modality image into the corresponding three-dimensional image space.

[0089] Illustratively, the mapping loss refers to a loss value of different types of features generated in the process of outputting the first generated image by the first candidate network mapping the image feature representation corresponding to the first modality image to the three-dimensional image space.

[0090] Optionally, the constraint loss value includes at least one of the following types:

[0091] 1. Dimension conversion loss, i.e., a loss value corresponding to the dimension conversion of the first modality image in the three-dimensional image space;

[0092] 2. Domain constraint loss, i.e., a feature loss corresponding to the first modality image existing in the first generated image obtained by the first candidate network in the process of modality conversion of the first modality image;

[0093] 3. Texture constraint loss, i.e., a region segmentation loss corresponding to the process of generating the first generated image by modality conversion of the first modality image;

[0094] 4. Contour constraint loss, i.e., an image contour boundary loss corresponding to the process of generating the first generated image by modality conversion of the first modality image.

[0095] It should be noted that the above constraint loss value is only an illustrative example, and the embodiments of the present application are not limited thereto.

[0096] Optionally, the constraint loss value is obtained in at least one of the following ways:

[0097] 1. The constraint loss value is determined by obtaining the difference between the first generated image and the first modality image;

[0098] 2. The constraint loss value is determined by obtaining the respective image features of the first generated image and the first modality image, and fusing the respective image features.

[0099] 3. Obtain image features corresponding to the first generated image and the first modality image respectively, determine the constraint loss value by determining the distance between the image features of the two;

[0100] 4. Obtain image features corresponding to the first generated image and the first modality image respectively, splice the image features corresponding to the two respectively, and determine the constraint loss value;

[0101] 5. Construct a constraint loss model, obtain image features corresponding to the first generated image and the first modality image respectively, input the image features corresponding to the two respectively into the constraint loss model, and output the obtained result as the constraint loss value.

[0102] It is worth noting that the above method for obtaining the constraint loss value is only an illustrative example, and the embodiments of the present application are not limited thereto.

[0103] Step 305, training the first candidate network based on the constraint loss value to obtain an image conversion network.

[0104] The image conversion network is used to perform modality conversion on the image belonging to the first modality to obtain a three-dimensional image belonging to the second modality.

[0105] Optionally, the first candidate network is gradient trained by the constraint loss value, or the first candidate network is cyclically iteratively trained by the constraint loss value, and the present application is not limited thereto.

[0106] Illustratively, the image conversion network is used to perform modality conversion on the image belonging to the first modality in a three-dimensional image space to obtain a three-dimensional image belonging to the second modality.

[0107] Optionally, the image belonging to the first modality is a two-dimensional image, or the image belonging to the first modality is a three-dimensional image, and the present application is not limited thereto.

[0108] Optionally, the image conversion network performs modality conversion in the same way as the first candidate network performs modality conversion, or differently, and the present application is not limited thereto.

[0109] In summary, the image generation method provided by the embodiment of the present application converts and restores the first modality image by the first candidate network and the second candidate network respectively, to obtain the first generated image and the first restored image, wherein the first candidate network is used to convert the first modality image into a three-dimensional image of the second modality, and the constraint loss value corresponding to the first candidate network in the three-dimensional image space is determined according to the first generated image and the first restored image, so that the first candidate network is trained in the manner, which can make the effect of the target three-dimensional image generated by the finally trained image conversion network better, that is, the training effect of the image conversion network can be improved by introducing the three-dimensional image space in the first candidate network, so that the accuracy of the output target three-dimensional image is higher.

[0110] In an optional embodiment, the constraint loss value includes a dimension conversion loss. For illustration, refer to Figure 4 FIG. 2 shows a flowchart of an image generation method provided by an example embodiment of the present application. The method can be executed by a terminal, a server, or both the terminal and the server. In this embodiment, the method is executed by a server, as shown in FIG. 2, that is, step 302 includes 302a, and step 304 further includes steps 3041a and 3041b. The method includes the following steps: Figure 3

[0111] Step 302a inputs the first modality image into a generator to output a first generated image.

[0112] For illustration, the first candidate network adopts a generator in a generative adversarial network architecture, and the first modality image is input into the generator to output the first generated image.

[0113] In the GAN network, the generator is a corresponding generative model. The first modality image is input into the generator, the generator synthesizes the first modality image by adding a random noise, and outputs the first generated image belonging to the second modality.

[0114] In some embodiments, the generator maps the image feature representation corresponding to the first modality image to the three-dimensional image space to obtain the generated feature representation corresponding to the image feature representation of the first modality image in the three-dimensional image space, and determines the first generated image according to the generated feature representation.

[0115] ​In this embodiment, the generator included in the first candidate network is composed of three convolutional layers with strides of 1, 2, and 2 as the front end, six residual blocks, two fractional convolution kernels with a stride of one-half, and a back-end convolutional layer with a stride of 1. Convolution-Batch Norm-ReLU is applied to each place except the output layer, and Tanh activation is used at the end. Each residual block includes two convolutional layers, each with 128 filters. A 7x7 volume convolution kernel is used for the first and last layers, and a 3x3x3 is used on other layers.

[0116] In this embodiment, the generator included in the first candidate network is denoted as generator G.

[0117] Step 303: modal restoration is performed on the first generated image by the second candidate network to obtain a first restored image.

[0118] The first restored image corresponds to the first modality.

[0119] In this embodiment, the second candidate network corresponds to the same architecture as the first candidate network, that is, the generator in the second candidate network also uses the corresponding generation model in the GAN network. It is worth noting that the generator included in the first candidate network and the generator included in the second candidate network are the respective generators of the two candidate networks. In this embodiment, the generator included in the second candidate network is denoted as generator F.

[0120] Illustratively, the structure of the generator F in the second candidate network is the same as that of the generator G, which will not be described here.

[0121] Step 3041a: obtain a second modality image.

[0122] The second modality image is a pre-provided image of the second modality.

[0123] Illustratively, the second modality image is an image corresponding to the second modality.

[0124] Illustratively, the second modality image is a sample training image in a public training data set, which is not limited here.

[0125] Optionally, the way to obtain the second modality image includes obtaining from a locally stored image data set or downloading from an existing public training data set, which is not limited here.

[0126] Optionally, the second modality image has a corresponding relationship with the first modality image, that is, the first modality image and the second modality image are a pre-matched image sample set; or the second modality image does not have a corresponding relationship with the first modality image, that is, the first modality image and the second modality image are respectively randomly acquired as sample images for training, which is not limited.

[0127] In the embodiment, the first modality image is first input into the generator G to generate the first generated image, and then the first generated image is input into the generator F to generate the first restored image, so as to realize the training process from the generator G to the generator F, that is, the process from the first candidate network to the second candidate network. However, in another feasible embodiment, in order to improve the generalization performance of the model, the second modality image is first input into the generator F to generate the second generated image, and then the second generated image is input into the generator G to generate the second restored image, and the corresponding constraint loss value is determined based on the second generated image and the second restored image to train the second candidate network, that is, to realize the training process from the second candidate network to the first candidate network.

[0128] The image generation method provided in the application includes the process from the first candidate network to the second candidate network and the training process from the second candidate network to the first candidate network, that is, the generator G and the generator F are constructed in a three-dimensional image space, and are bidirectional mapping functions for modality conversion between the first modality image and the second modality image. Since the training processes on both sides are consistent and only the training directions are different, the training process from the first candidate network to the second candidate network is taken as an example for specific description in the embodiment.

[0129] In step 3041b, the dimension conversion loss value is obtained based on the image feature distribution difference between the first generated image and the second modality image, and based on the image feature distribution difference between the first modality image and the first restored image.

[0130] The dimension conversion loss value is used to indicate the feature loss of the dimension conversion process of the first candidate network in the three-dimensional image space.

[0131] Illustratively, the dimension conversion loss value is used to determine the corresponding feature loss in the dimension conversion process in the three-dimensional image space in the process of modality conversion from the first modality image to generate the first generated image, wherein the dimension conversion includes: converting a two-dimensional image into a three-dimensional image; or converting a three-dimensional image into a three-dimensional image, but corresponding to different image feature distributions, which is not limited.

[0132] In some embodiments, the discriminant loss is determined based on a difference in image feature distribution between the second modality image and the first generated image; the generation loss is determined based on a difference in image feature distribution between the first modality image and the first restored image; and the generation loss and the discriminant loss are taken as the dimension conversion loss value, which is used to indicate a loss generated when the first candidate network performs image dimension conversion through a three-dimensional image space.

[0133] Illustratively, the dimension conversion loss value includes two parts: the discriminant loss and the generation loss, wherein the discriminant loss is determined according to the second modality image and the first generated image; and the generation loss is determined according to the first modality image and the first restored image.

[0134] In some embodiments, the first candidate network further includes a discriminator in a generative adversarial network architecture, which is used to determine a difference in image feature distribution between the second modality image and the first generated image, and determine the discriminant loss according to the difference between the second modality image and the first generated image, that is, input the second modality image into the discriminator to obtain a reference prediction result, which is used to indicate a probability that the second modality image is a reference image; input the first generated image into the discriminator to obtain a matching prediction result, which is used to indicate a matching relationship between the first generated image and the second modality image; and determine the discriminant loss based on the reference prediction result and the matching prediction result.

[0135] The discriminator is a corresponding discriminant model in the GAN network, and the first generated image and the second modality image are input into the discriminant model, and the output of the discriminant model is a probability value, which is distributed between 0 and 1, 1 represents a real sample, that is, the second modality image, and 0 represents a fictitious sample, that is, the first generated image, which is used to determine the similarity between the first generated image and the second modality image, and when the first generated image is closer to the second modality image, the output probability value is larger, indicating that the accuracy of the generated image generated by the generator is higher.

[0136] In this embodiment, the discriminator is used to determine the difference between the first generated image of the second modality and the second modality image, that is, the second modality image is taken as a reference image and input into the discriminator to obtain a reference prediction probability corresponding to the second modality image, at this time, the current discriminator takes the second modality image as a reference image, inputs the first generated image into the discriminator, and outputs a matching prediction probability as a matching result corresponding to the current first generated image and the second modality image, for example, displays the image feature distribution probability corresponding to the first generated image on the second modality image, which is used to indicate the matching result of the image feature corresponding to the first generated image on the second modality image.

[0137] It is worth noting that the above method of calculating the discrimination loss is the discrimination loss corresponding to the training process from the first candidate network to the second candidate network. However, in the training process from the second candidate network to the first candidate network, the second candidate network also includes a discriminator. Therefore, there is also a discrimination loss in this training process.

[0138] The discriminant loss is determined based on the reference prediction probability and the matching prediction result. For illustration, please refer to Formula 1:

[0139] Formula 1: L b (D G D F (G, F) = L(G, D) G )+L(F,D F )

[0140] Among them, L b For the total discriminant loss, L(G, D) G L(F, D) represents the discriminant loss corresponding to the discriminator in the first candidate network. F ) represents the discriminant loss corresponding to the discriminator in the second candidate network. In practical applications, the total discriminant loss is used. The higher the probability of the image feature distribution of the first generated image corresponding to the second modality image, the higher the actual performance of the first generated image in fitting the second modality image, that is, the smaller the discriminant loss value.

[0141] In this embodiment, the discriminator in the first candidate network is denoted as D. G In constructing the discriminator D G During the process, the block size is fixed to a 70×70×70 volume pixel by overlapping, and a stack of Convolution-BatchNorm-Leaky ReLU layers is used to train the discriminator D. G Discriminator D G Convolution operations are performed on the entire voxel, and the corresponding prediction result is determined by averaging all the results.

[0142] In some embodiments, a first feature representation corresponding to a first modality image is determined; a second feature representation corresponding to a first reconstructed image is determined; and a generation loss is determined based on the feature representation distance between the first feature representation and the second feature representation.

[0143] In this embodiment, when the first modal image corresponds to the first modality, its corresponding image feature representation is used as the first feature representation, and when the first restored image corresponds to the first modality, its corresponding image generation representation is used as the second feature representation.

[0144] In this embodiment, the generation loss is determined according to the first feature representation and the feature representation distance corresponding to the first feature representation. For illustration, please refer to Equation 2:

[0145] Equation 2:

[0146]

[0147] wherein X is the first modality image, L(G(X)) is the first restored image, Y is the second modality image, G(F(Y)) is the second restored image obtained in the training process from the second candidate network to the first candidate network (here, the process of obtaining the second restored image is the same as that of obtaining the first restored image, and thus is not specifically described here), is the total generation loss, represents the feature representation distance corresponding to the first modality image and the first restored image, and the generation loss corresponding to the generator G is obtained, that is, the generation loss is the generation loss in the training process from the first candidate network to the second candidate network, is the generation loss corresponding to the generator F, that is, the generation loss is the generation loss generated in the training process from the second candidate network to the first candidate network. In actual application process, the total generation loss is used for training. In the training process from the first candidate network to the second candidate network, the smaller X-F(G(X)) is, the smaller the distance between the feature distribution of the current first modality image and the first restored image is, that is, the smaller the generation loss is. The training process from the second candidate network to the first candidate network is consistent, and thus is not described here.

[0148] In this embodiment, the total generation loss and the total discrimination loss are used as two parts of the loss corresponding to the dimension conversion loss value, for determining the loss corresponding to the dimension conversion of the bidirectional mapping function corresponding to the first candidate network and the second candidate network in the three-dimensional image space.

[0149] In summary, the image generation method provided in the embodiments of the present application converts the first modality image through the first candidate network and the second candidate network to obtain the first generated image and the first restored image, respectively, wherein the first candidate network is used to convert the first modality image into a three-dimensional image of the second modality, and the constraint loss value corresponding to the first candidate network in the three-dimensional image space is determined according to the first generated image and the first restored image, so that the first candidate network is trained in this way. The effect of the target three-dimensional image generated by the finally trained image conversion network is better, that is, the introduction of the three-dimensional image space in the first candidate network can improve the training effect of the image conversion network, so that the accuracy of the output target three-dimensional image is higher.

[0150] In this embodiment of the application, by determining the generation loss corresponding to the generator and the discrimination loss corresponding to the discriminator as the dimension transformation loss, the accuracy of the first candidate network in performing feature mapping on the first modality image in the three-dimensional image space can be improved, thereby achieving the transformation accuracy in the dimension transformation process. This can effectively reduce the loss corresponding to dimension transformation and improve the model accuracy.

[0151] In an optional embodiment, the constraint loss value further includes a domain constraint loss value, a texture constraint loss value, and a contour constraint loss value. For illustrative examples, please refer to [reference needed]. Figure 5 This document illustrates a flowchart of an image generation method provided in an exemplary embodiment of this application. This method can be executed by a terminal, a server, or both. In this embodiment, the method is described as being executed by a server. Figure 5 As shown, when the constraint loss value includes the constraint loss value, step 304 further includes steps 3042a, 3042b, and 3042c; when the constraint loss value includes the texture constraint loss value, step 304 further includes steps 3043a, 3043b, and 3043c; and when the constraint loss value includes the contour constraint loss value, step 304 further includes steps 3044a, 3044b, and 3044c. The method includes the following steps:

[0152] 1. When the constraint loss value includes the domain constraint loss value.

[0153] Step 3042a: Obtain the first feature distribution of the first modality image in the three-dimensional image space.

[0154] In this embodiment, during the process of inputting the first modality image into the generator to obtain the first generated image of the second modality, it is usually assumed that the feature distributions of the two image modalities are unchanged in their corresponding image domains. However, in reality, when the image is converted across modalities, its feature distribution changes in the image domain. This is especially true during the acquisition of multiple sequence medical images, where medical image sequences are used to indicate different patterns describing diseases or symptoms. In order to improve the generalization performance of the model and reduce the impact of the differences in the corresponding image domains between modalities during cross-modal conversion, domain constraint loss is introduced.

[0155] In this embodiment, during the process of inputting the first modality image into the first candidate network, the first modality image is mapped into a three-dimensional image space to extract the image feature distribution corresponding to the first modality image as the first feature distribution, which is used to represent the feature distribution result corresponding to the first modality.

[0156] Step 3042b: Obtain the second feature distribution of the first generated image in the three-dimensional image space.

[0157] Illustratively, the first generated image is mapped into the three-dimensional image space for determining an image feature distribution corresponding to the first generated image.

[0158] In step 3042c, a domain constraint loss value is obtained based on a distance between the first feature distribution and the second feature distribution.

[0159] The domain constraint loss value is used to indicate a conversion loss from the first feature distribution to the second feature distribution in the three-dimensional image space.

[0160] Illustratively, the domain constraint loss value is used to determine a feature similarity of the first feature distribution and the second feature distribution corresponding to each other in the three-dimensional image space, i.e., a similar feature corresponding to each other in the current first modality and the second modality.

[0161] In this embodiment, the domain constraint loss value is determined by Maximum Mean Discrepancy (MMD), and the specific determination manner is illustratively shown in Formula Three:

[0162] Formula Three:

[0163] E X [Φ(X)] corresponds to the first feature distribution of the first modality, E Y [Φ(Y)] represents the second distribution feature corresponding to the second modality, and the smaller the difference between the first distribution feature and the second distribution feature, the shorter the feature distance between the first distribution feature and the second distribution feature, and the higher the similarity of the current first distribution feature and the second distribution feature.

[0164] 2. When the constraint loss value includes a texture constraint loss value.

[0165] In step 3043a, a first segmentation result corresponding to the first generated image is obtained.

[0166] The first segmentation result is used to indicate a reference probability distribution of the first modality image corresponding to the first generated image.

[0167] In some embodiments, the first generated image is input into a segmenter, and the first segmentation result is output. The segmenter is used for region segmentation of the input image.

[0168] Illustratively, the first candidate network further includes a segmenter, which is used for region segmentation of the input image, and is used for determining a texture representation corresponding to each region of the input image.

[0169] In this embodiment, the first generated image is input into the segmenter, and the first segmentation result is output, which is used to determine the texture feature corresponding to the first generated image.

[0170] Illustratively, a fully convolutional network (FCN) is used as the segmenter in the embodiment.

[0171] At step 3043b, a second segmentation result corresponding to the second modality image is obtained.

[0172] Illustratively, the second modality image is input into the segmenter, and a second segmentation result is output, which is used to represent the texture feature corresponding to the second modality image.

[0173] At step 3043c, a texture constraint loss value is determined based on a segmentation difference between the first segmentation result and the second segmentation result.

[0174] The texture constraint loss value is used to indicate a texture feature loss of the first candidate network when mapping the first modality image to the three-dimensional image space.

[0175] Illustratively, in order to determine the context information corresponding to the first generated image, it is necessary to determine that the texture representation contained in the second modality image can be correctly reflected in the first generated image. For example, when the second modality image is a brain image, the brain texture information corresponding to the brain image is an important factor for generating a corresponding three-dimensional brain image. Because the brain texture information and the disease progression, functional change, and other information are key factors for brain pathology analysis, therefore, the texture constraint loss value is introduced to determine the loss value of the texture feature in the three-dimensional image space in the process of inputting the first modality image into the first candidate network to generate the first generated image.

[0176] In the embodiment, a two-stage texture loss function is preset to preserve the texture feature corresponding to the process of generating the first generated image by the first candidate network from the first modality image. Illustratively, please refer to Equation Four:

[0177] Equation Four:

[0178]

[0179] represents a total texture constraint loss value, represents a texture constraint loss value corresponding to the training process from the first candidate network to the second candidate network, that is, the segmentation difference between the first segmentation result and the second segmentation result, represents a texture constraint loss value corresponding to the training process from the second candidate network to the first candidate network. Taking the training process from the first candidate network to the second candidate network as an example, represents the first segmentation result corresponding to the first generated image, ​The second segmentation result corresponding to the second modality image indicates that the smaller the difference between the first segmentation result and the second segmentation result is, the higher the texture feature similarity corresponding to the first generated image and the second modality image is, and the smaller the texture constraint loss value is.

[0180] 3. When the constraint loss value includes the contour constraint loss value.

[0181] Step 3044a, input the first generated image into the discriminator, and output a matching prediction result.

[0182] The matching prediction result is used to indicate the matching relationship corresponding to the first generated image and the second modality image.

[0183] The content of the matching prediction result in step 3044a has been described in detail in the above step 3041b, and will not be repeated here.

[0184] Step 3044b, input the first generated image into the segmenter, and output a first segmentation result.

[0185] The segmenter is used for region segmentation of the input image.

[0186] The content of the first segmentation result in step 3044b has been described in detail in the above step 3043a, and will not be repeated here.

[0187] Step 3044c, based on the matching prediction result and the first segmentation result, determine a contour constraint loss value.

[0188] The contour constraint loss value is used to indicate the boundary feature loss of the first generated image in the three-dimensional image space.

[0189] In this embodiment, the contour information of the image can be used for image analysis and semantic segmentation of the image. The contour information is used to provide semantic information and context relationship corresponding to the image. For example, when the first modality image is a brain image, the contour information corresponding to the brain image facilitates better understanding of the anatomical structure of the brain and the progression of the disease. In the process of generating the brain MRI image of the second modality corresponding to the brain image of the first modality, it is necessary to ensure that the contour of the brain image of the first modality maintains clear boundaries in the cross-modality process, which is the key to realizing the brain MRI image with good contour boundary display. Therefore, the contour constraint loss value is introduced to determine the boundary feature loss of the first generated image in the three-dimensional image space corresponding to the first modality image.

[0190] In the training process from the first candidate network to the second candidate network, the first generated image is input into the discriminator and the segmenter corresponding to the first candidate network respectively, and in the training process from the second candidate network to the first candidate network, the second generated image is input into the discriminator and the segmenter corresponding to the second candidate network respectively, to determine the contour constraint loss value, wherein the segmenter in the first candidate network and the segmenter in the second candidate network are implemented as deconvolution operations, and follow the FCN network.

[0191] In step 305, the first candidate network is trained based on the constraint loss value, to obtain the image conversion network.

[0192] Illustratively, the four constraint loss values provided by the embodiments of the present application can be combined differently according to different task scenarios, to adjust the model parameters of the first candidate network.

[0193] In summary, the image generation method provided by the embodiments of the present application converts and restores the first modality image by the first candidate network and the second candidate network respectively, to obtain the first generated image and the first restored image, wherein the first candidate network is used to convert the first modality image into a three-dimensional image of the second modality, and the constraint loss value corresponding to the first candidate network in the three-dimensional image space is determined according to the first generated image and the first restored image, so that the first candidate network is trained in this way, which can make the target three-dimensional image generated by the finally trained image conversion network better, that is, by introducing the three-dimensional image space into the first candidate network, the training effect of the image conversion network can be improved, so that the accuracy of the output target three-dimensional image is higher.

[0194] In the embodiments, by setting the domain constraint loss value, the texture constraint loss value and the contour constraint loss value, the image conversion network can meet the image generation task in different task scenarios, to improve the model accuracy and generalization performance of the image conversion network.

[0195] In an optional embodiment, illustratively, please refer to Figure 6 which shows the training process of the image generation method provided by an exemplary embodiment of the present application, as shown in Figure 6 the method comprises:

[0196] A first modality image 610 is obtained, wherein the first modality image 610 is a brain image of a first modality obtained from a public data set, the first modality image is input into a generator (G) 620 in a first candidate network, a first generated image 630 is generated, wherein the first generated image 630 is a three-dimensional brain image of a second modality, the first generated image 630 is input into a generator (F) 640 in a second candidate network, and a first restored image 650 is output, wherein the first restored image 650 corresponds to a brain image of the first modality, and the first modality and the second modality correspond to different modalities.

[0197] In the process of model training, a second modality image 660 is also included, wherein the second modality image 660 is a brain image of a second modality obtained from a public data set. According to the difference between the image feature distributions of the first generated image 630 and the second modality image 660, and according to the difference between the image feature distributions of the first modality image 610 and the first restored image 650, four different constraint loss values are determined, including a dimension conversion loss value, a domain constraint loss value, a texture constraint loss value, and a contour constraint loss value.

[0198] In addition, the first candidate network also includes a segmenter 670 and a discriminator 680, the first generated image 630 is input into the segmenter 670, and a first segmentation result 671 is output, the second modality image 660 is input into the segmenter 670, and a second segmentation result 672 is output, and the first generated image 630 and the second modality image 660 are input into the discriminator 680, and a matching prediction result 690 is output, wherein the output value of the matching prediction result 690 is “1” indicating that the current first generated image 630 and the second modality image 660 match, and the output value of the matching prediction result 690 is “0” indicating that the current first generated image 630 and the second modality image 660 do not match.

[0199] In the training process, a generation loss is determined according to the first modality image 610 and the first restored image 650, and an adversarial loss is determined according to the first generated image 630 and the second modality image 660, and the generation loss and the adversarial loss are taken as the dimension conversion loss.

[0200] The domain constraint loss value is determined according to the first feature distribution corresponding to the first modality image 610 and the second feature distribution corresponding to the first generated image 630.

[0201] The texture constraint loss value is determined according to the first segmentation result and the second segmentation result.

[0202] The contour constraint loss value is determined according to the matching prediction result and the first segmentation result.

[0203] According to four different constraint loss values, at least one of the first candidate networks is selected according to a current task scene to be trained, and finally an image conversion network is obtained, which is used to generate a target three-dimensional brain image.

[0204] The image generation method in this embodiment is evaluated on three data sets: a first public data set, a second public data set, and a third public data set. The three data sets contain brain images of four sequences, namely T1 weighted images, T2 weighted images, Proton Density (PD) weighted images, and Fluid Attenuated Inversion Recovery (FLAIR) images. The four sequences correspond to different modalities, and the four sequence images show different brain features. The image generation method is evaluated in three scenarios, which are selected according to the complexity of matching the image domain corresponding to the first modality and the image domain corresponding to the second modality: (1) The task corresponding to the first public data set is to convert the PD weighted image to the T2 weighted image; (2) The task corresponding to the second public data set is to convert the T1 weighted image to the T2 weighted image; (3) The task corresponding to the third public data set is to convert the FLAIR image to the T1 weighted image. Under scenario (1), for example, refer to Figure 7 , which shows a brain image generation process diagram provided by an example embodiment of the present application, as shown in Figure 7 , the PD weighted image 701 is input into the first candidate network as the first modality image, and the output T2 weighted synthetic image 702 is obtained as the target three-dimensional image, Figure 7 , which also includes the T2 weighted image 703 as the second modality image (the image is obtained from the public data set).

[0205] In each dataset, there are well-aligned paired images with significant appearance variations obtained by different imaging modalities. All paired data are used as ground truth images to validate the quality of the synthesized results. Quantitatively, 239 unpaired PD-weighted images and T2-weighted images are manually selected from the first public dataset, 8 unpaired T1-weighted images and T2-weighted images, and 90 unpaired T1-weighted images and FLAIR images are selected from the second public dataset for training. The rest of the data: 100 images in the first public dataset, 4 images in the second public dataset, and 40 images in the third public dataset are used for testing. For the FCN, both real scans and synthesized results are provided to produce three main brain tissue classes: Cerebral Spinal Fluid (CSF), Gray Matter (GM), and White Matter (WM), giving average quantification of brain volumes. The tissue prior probability templates are pre-set good brain image segmentation templates for validating the segmentation results corresponding to their models. For the evaluation criteria, the Peak Signal-To-Noise Ratio (PSNR), the Structural Similarity Index (SSIM), and the Dice Score (a measure of segmentation overlap, the higher the score, the more accurate the result) are used to compare the results.

[0206] For illustration, refer to Table 1, which shows the constraint loss values corresponding to the model training effects under different collocations provided by the embodiments of the present application, as shown in Table 1:

[0207] Table 1

[0208]

[0209]

[0210] To evaluate the performance of the image generation method, an ablation study is first conducted to examine each constraint loss value to evaluate the importance of each component in the image conversion network. Specifically, for converting PD-weighted images to T2-weighted images on the first public dataset, the dimension conversion loss value, the domain constraint loss value, the texture constraint loss value, and the contour constraint loss value are used respectively, and they are freely combined with the GAN network to study the image aspect effect quality and its segmentation performance, and the detailed results are shown in the first part of Table 1.

[0211] As can be seen from Table 1, with the help of the dimension conversion loss value, the domain constraint loss value, the texture constraint loss value and the contour constraint loss value, the visual and segmentation results are greatly improved. The appearance score is sensitive to the dimension conversion loss value, the domain constraint loss value and the texture constraint loss value, and the segmentation result is more sensitive to the domain constraint loss value and the contour constraint loss value.

[0212] The analysis shows that the domain constraint loss value and the contour constraint loss value are important to the visual effect and the segmentation result. The second part of results in Table 1 shows that the dimension conversion loss value and the domain constraint loss value are the most important pair combination of the four constraints. The third part of Table 1 shows the performance of different combinations of three constraints. The results show that based on the GAN network, the combination of the dimension conversion loss value, the domain constraint loss value and the texture constraint loss value realizes 3.23 dB, 0.0414 and 13.39% improvement in PSNR, SSIM and Dice score respectively, and further combining the contour constraint loss value realizes the best segmentation performance, with 14.07% improvement in Dice score.

[0213] To sum up, the image generation method provided by the embodiment of the application, by the first candidate network and the second candidate network, the first modality image is respectively converted and restored in the modality, and the first generated image and the first restored image are respectively obtained, wherein the first candidate network is used to convert the first modality image into a three-dimensional image of the second modality, and the constraint loss corresponding to the first candidate network in the three-dimensional image space is determined according to the first generated image and the first restored image, so that the first candidate network is trained in this way. The way of training the image conversion network can make the effect of the target three-dimensional image generated by the finally trained image conversion network better, that is, by introducing the three-dimensional image space in the first candidate network, the training effect of the image conversion network can be improved, so that the accuracy of the output target three-dimensional image is higher.

[0214] In the present scheme, the proposed image generation method can generate a transferable modality representation with rich semantic features, texture details and anatomical structure preservation. By introducing four constraint loss values, the GAN framework is effectively customized to realize the synthesis of brain images of different sequences. The present method is based on medical image synthesis as the technical background, but in practice, this technology can be applied to other unsupervised synthesis tasks, such as natural image style transfer, without limitation.

[0215] Figure 8 is a structural block diagram of an image generation device provided by an exemplary embodiment of the application, as shown in Figure 8 The device comprises the following parts:

[0216] The acquisition module 810 is configured to acquire a first modality image, wherein the first modality image corresponds to a first modality.

[0217] The conversion module 820 is configured to perform modality conversion on the first modality image by using a first candidate network to obtain a first generated image, the first generated image corresponds to a second modality, and the first generated image is a three-dimensional image, and the first modality and the second modality are different modalities.

[0218] The restoration module 830 is configured to perform modality restoration on the first generated image by using a second candidate network to obtain a first restored image, the first restored image corresponds to the first modality.

[0219] The acquisition module 810 is further configured to acquire a constraint loss value based on the first generated image and the first restored image, and the constraint loss value is used to indicate a mapping loss of the first candidate network in mapping the first modality image to a three-dimensional image space.

[0220] The training module 840 is configured to train the first candidate network based on the constraint loss value to obtain an image conversion network, and the image conversion network is used to perform modality conversion on an image belonging to the first modality to obtain a three-dimensional image belonging to the second modality.

[0221] In an optional embodiment, the constraint loss value includes a dimension conversion loss value.

[0222] The acquisition module 810 includes:

[0223] The acquisition unit 811 is configured to acquire a second modality image, and the second modality image is a pre-provided image of the second modality.

[0224] The acquisition unit 811 is further configured to acquire the dimension conversion loss value based on an image feature distribution difference between the first generated image and the second modality image, and based on an image feature distribution difference between the first modality image and the first restored image, and the dimension conversion loss value is used to indicate a feature loss generated when the first candidate network performs image dimension conversion by using the three-dimensional image space.

[0225] In an optional embodiment, the acquisition unit 811 is further configured to determine a discrimination loss based on the image feature distribution difference between the second modality image and the first generated image, determine a generation loss based on the image feature distribution difference between the first modality image and the first restored image, and take the generation loss and the discrimination loss as the dimension conversion loss value.

[0226] In an optional embodiment, the first candidate network includes a discriminator in a discriminative generation network architecture.

[0227] The acquisition unit 811 is further configured to input the second modality image into the discriminator to output a reference prediction result, the reference prediction result being used to indicate a probability that the second modality image is a reference image; input the first generated image into the discriminator to output a matching prediction result, the matching prediction result being used to indicate a matching relationship between the first generated image and the second modality image; and determine the discrimination loss based on the reference prediction result and the matching prediction result.

[0228] In an optional embodiment, the acquisition unit 811 is further configured to determine a first feature representation corresponding to the first modality image; determine a second feature representation corresponding to the first restored image; and determine the generation loss based on a feature representation distance between the first feature representation and the second feature representation.

[0229] In an optional embodiment, the first candidate network includes a generator in a generative adversarial network architecture.

[0230] The acquisition unit 811 is further configured to input the first modality image into the generator to output the first generated image.

[0231] In an optional embodiment, the constraint loss value includes a domain constraint loss value.

[0232] The acquisition module 810 is further configured to acquire a first feature distribution of the first modality image in the three-dimensional image space; acquire a second feature distribution of the first generated image in the three-dimensional image space; and determine the domain constraint loss value based on a distance between the first feature distribution and the second feature distribution, the domain constraint loss value being used to indicate a conversion loss from the first feature distribution to the second feature distribution.

[0233] In an optional embodiment, the constraint loss value includes a texture constraint loss value.

[0234] The acquisition module 810 is further configured to acquire a first segmentation result corresponding to the first generated image, the first segmentation result being used to indicate a reference probability distribution of the first modality image corresponding to the first generated image; acquire a second segmentation result corresponding to the first modality image; and determine a texture constraint loss value based on a segmentation difference between the first segmentation result and the second segmentation result, the texture constraint loss value being used to indicate a texture feature loss of the first candidate network in mapping the first modality image to the three-dimensional image space.

[0235] In an optional embodiment, the acquisition module 810 is further configured to input the first generated image into a segmenter to output the first segmentation result, the segmenter being used to perform region segmentation on an input image.

[0236] In an optional embodiment, the constraint loss value comprises a contour constraint loss value.

[0237] The acquisition module 810 is further configured to input the first generated image into a discriminator to obtain a matching prediction result, the matching prediction result being used to indicate a matching relationship between the first generated image and the second modality image; input the first generated image into a segmenter to obtain the first segmentation result, the segmenter being used to perform region segmentation on an input image; and determine the contour constraint loss value based on the matching prediction result and the first segmentation result, the contour constraint loss value being used to indicate a boundary feature loss of the first generated image in the three-dimensional image space.

[0238] To sum up, the image generation apparatus provided in the embodiments of the present application can convert and restore the first modality image by using the first candidate network and the second candidate network, respectively, to obtain the first generated image and the first restored image, wherein the first candidate network is used to convert the first modality image into a three-dimensional image of the second modality, and the constraint loss corresponding to the first candidate network in the three-dimensional image space is determined based on the first generated image and the first restored image, so that the first candidate network is trained in this way, which can make the target three-dimensional image generated by the finally trained image conversion network have a better effect, that is, the training effect of the image conversion network can be improved by introducing the three-dimensional image space into the first candidate network, so that the accuracy of the output target three-dimensional image is higher.

[0239] It should be noted that the image generation apparatus provided in the above embodiments is only exemplified by the division of the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the image generation apparatus and the image generation method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0240] Figure 10 The structure of the server provided in an example embodiment of the present application is shown. Specifically:

[0241] The server 1000 includes a central processing unit (CPU) 1001, a system memory 1004, including a random access memory (RAM) 1002 and a read-only memory (ROM) 1003, and a system bus 1005 that couples the system memory 1004 to the central processing unit 1001. The server 1000 also includes a mass storage device 1006 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0242] The mass storage device 1006 connects to the central processing unit 1001 through a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1006 and its associated computer-readable media provide non-volatile storage for the server 1000. That is, the mass storage device 1006 can include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0243] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Computer storage media would not, however, include communication media including wired or wireless signaling media that communicate program code in a modulated data signal. The system memory 1004 and mass storage device 1006 described above can be embodied as a memory component.

[0244] According to various embodiments of the present application, the server 1000 can further operate connected to a network with a remote computer via a network, such as the Internet. That is, the server 1000 can be connected to a network 1012 via a network interface unit 1011 connected to the system bus 1005, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 1011.

[0245] The above-described memory further includes one or more programs stored in the memory configured to be executed by the CPU.

[0246] Embodiments of the present application also provide a computer device including a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, at least one program, code set or instruction set being loaded and executed by the processor to implement the image generation method provided by the above-described various method embodiments.

[0247] Embodiments of the present application also provide a computer readable storage medium, the computer readable storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, at least one program, code set or instruction set being loaded and executed by the processor to implement the image generation method provided by the above-described various method embodiments.

[0248] Embodiments of the present application also provide a computer program product or computer program, the computer program product or computer program including computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the image generation method described in any of the above embodiments.

[0249] Optionally, the computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a solid state disk (SSD), an optical disk, etc. Among them, the random access memory can include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The above-mentioned application embodiment serial number is only for description, not representing the pros and cons of the embodiments.

[0250] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by program. The program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0251] The above only describes optional embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image generation method characterized by, The method comprises: acquiring a first modality image corresponding to a first modality; performing modality conversion on the first modality image by a first candidate network to obtain a first generated image corresponding to a second modality, wherein the first generated image is a three-dimensional image, and the first modality and the second modality are different modalities; the modality conversion comprises pixel point sequence reorganization, dimension conversion, feature fusion, and threshold conversion; performing modality restoration on the first generated image by a second candidate network to obtain a first restored image corresponding to the first modality; the modality restoration comprises pixel point distribution restoration, dimension restoration, feature decomposition, and threshold restoration; based on the first generated image and the first restored image, acquiring a constraint loss value for indicating a mapping loss of the first candidate network when mapping the first modality image to a three-dimensional image space; the constraint loss value comprises a texture constraint loss value, a contour constraint loss value, a dimension conversion loss value, and a domain constraint loss value; wherein a first segmentation result corresponding to the first generated image is acquired, the first segmentation result is used to indicate a reference probability distribution of the first generated image corresponding to the first modality image; a second segmentation result corresponding to the first modality image is acquired; based on a segmentation difference between the first segmentation result and the second segmentation result, the texture constraint loss value is acquired, which is used to indicate a texture feature loss of the first candidate network when mapping the first modality image to the three-dimensional image space; In a training process from the first candidate network to the second candidate network, the first generated image is input into a discriminator corresponding to the first candidate network, and a first matching prediction result is output. The first matching prediction result is used to indicate a matching relationship between the first generated image and a second modality image. The second modality image is a pre-provided image of the second modality. The first generated image is input into a segmenter corresponding to the first candidate network, and a first segmentation result is output. The segmenter is used for region segmentation of an input image. Based on the first matching prediction result and the first segmentation result, a first contour constraint loss value is determined. The first contour constraint loss value is used to indicate a boundary feature loss of the first generated image in the three-dimensional image space. In a training process from the second candidate network to the first candidate network, a second generated image is input into a discriminator corresponding to the second candidate network, and a second matching prediction result is output. The second generated image is an image corresponding to the second modality image obtained by the second candidate network. The second matching prediction result is used to indicate a matching relationship between the second generated image and the first modality image. The second generated image is input into a segmenter corresponding to the second candidate network, and a second segmentation result is output. Based on the second matching prediction result and the second segmentation result, a second contour constraint loss value is determined. The second contour constraint loss value is used to indicate a boundary feature loss of the second generated image in the three-dimensional image space. The contour constraint loss value includes the first contour constraint loss value and the second contour constraint loss value. The segmenter in the first candidate network and the segmenter in the second candidate network are implemented as deconvolution operations. The dimension conversion loss value is obtained based on an image feature distribution difference between the first generated image and the second modality image, and based on an image feature distribution difference between the first modality image and the first restored image. The dimension conversion loss value is used to indicate a loss generated when the first candidate network performs image dimension conversion through the three-dimensional image space. The first feature distribution of the first modality image in the three-dimensional image space is obtained. The second feature distribution of the first generated image in the three-dimensional image space is obtained. The domain constraint loss value is obtained based on a distance between the first feature distribution and the second feature distribution. The domain constraint loss value is used to determine a feature similarity of the first feature distribution and the second feature distribution in the three-dimensional image space. The first candidate network is trained based on the constraint loss value, and an image conversion network is obtained. The image conversion network is used to perform modality conversion on an image belonging to the first modality, and obtain a three-dimensional image belonging to the second modality.

2. The method of claim 1, wherein, The dimension conversion loss value is obtained based on an image feature distribution difference between the first generated image and the second modality image, and based on an image feature distribution difference between the first modality image and the first restored image. determine a discriminative loss based on the image feature distribution difference between the second modality image and the first generated image; determine a generative loss based on the image feature distribution difference between the first modality image and the first restored image; take the generative loss and the discriminative loss as the dimension conversion loss value.

3. The method of claim 2, wherein, the first candidate network comprises a discriminator in a discriminative generative network architecture; the determining of the discriminative loss based on the image feature distribution difference between the second modality image and the first generated image comprises: inputting the second modality image into the discriminator to output a reference prediction result, the reference prediction result being used to indicate a probability that the second modality image is a reference image; inputting the first generated image into the discriminator to output a matching prediction result, the matching prediction result being used to indicate a matching relationship between the first generated image and the second modality image; determining the discriminative loss based on the reference prediction result and the matching prediction result.

4. The method of claim 2, wherein, the determining of the generative loss based on the image feature distribution difference between the first modality image and the first restored image comprises: determining a first feature representation corresponding to the first modality image; determining a second feature representation corresponding to the first restored image; determining the generative loss based on a feature representation distance between the first feature representation and the second feature representation.

5. The method of claim 4, wherein, the first candidate network comprises a generator in a generative adversarial network architecture; the modality conversion of the first modality image by the first candidate network to obtain the first generated image comprises: inputting the first modality image into the generator to output the first generated image.

6. The method according to any one of claims 1 to 5, characterized in that, the obtaining of the first segmentation result corresponding to the first generated image comprises: inputting the first generated image into a segmenter to output the first segmentation result, the segmenter being used for region segmentation of an input image.

7. An image generation apparatus characterized by comprising: the apparatus comprises: an acquisition module, configured to acquire a first modality image, the first modality image corresponding to a first modality; a conversion module, configured to perform modality conversion on the first modality image by a first candidate network to obtain a first generated image, the first generated image corresponding to a second modality, and the first generated image being a three-dimensional image, the first modality and the second modality being different modalities; the modality conversion comprising pixel point sequence reorganization, dimension conversion, feature fusion, and threshold conversion; a restoration module, configured to perform modality restoration on the first generated image by a second candidate network to obtain a first restored image, the first restored image corresponding to the first modality; the modality restoration comprising pixel point distribution restoration, dimension restoration, feature decomposition, and threshold restoration. The acquisition module is further configured to acquire a constraint loss value based on the first generated image and the first restored image, the constraint loss value being used to indicate a mapping loss when the first candidate network maps the first modality image to a three-dimensional image space; the constraint loss value includes a texture constraint loss value, a contour constraint loss value, a dimension conversion loss value, and a domain constraint loss value; wherein a first segmentation result corresponding to the first generated image is acquired, the first segmentation result being used to indicate a reference probability distribution of the first generated image corresponding to the first modality image; a second segmentation result corresponding to the first modality image is acquired; based on a segmentation difference between the first segmentation result and the second segmentation result, a texture feature loss value is acquired, the texture constraint loss value being used to indicate a texture feature loss when the first candidate network maps the first modality image to the three-dimensional image space; In the training process from the first candidate network to the second candidate network, the first generated image is input into a discriminator corresponding to the first candidate network, and a first matching prediction result is output, the first matching prediction result being used to indicate a matching relationship between the first generated image and a second modality image, the second modality image being a pre-provided image of the second modality; the first generated image is input into a segmenter corresponding to the first candidate network, and a first segmentation result is output, the segmenter being used for region segmentation of an input image; based on the matching prediction result and the first segmentation result, a first contour constraint loss value is determined, the first contour constraint loss value being used to indicate a boundary feature loss of the first generated image in the three-dimensional image space; in the training process from the second candidate network to the first candidate network, a second generated image is input into a discriminator corresponding to the second candidate network, and a second matching prediction result is output, the second generated image being an image corresponding to the second modality image acquired by the second candidate network; the second matching prediction result is used to indicate a matching relationship between the second generated image and the first modality image; the second generated image is input into a segmenter corresponding to the second candidate network, and a second segmentation result is output; based on the second matching prediction result and the second segmentation result, a second contour constraint loss value is determined, the second contour constraint loss value being used to indicate a boundary feature loss of the second generated image in the three-dimensional image space, the contour constraint loss value including the first contour constraint loss value and the second contour constraint loss value; the segmenter in the first candidate network and the segmenter in the second candidate network are implemented as deconvolution operations; The dimension conversion loss value is acquired based on an image feature distribution difference between the first generated image and the second modality image, and based on an image feature distribution difference between the first modality image and the first restored image, the dimension conversion loss value being used to indicate a loss generated when the first candidate network performs image dimension conversion through the three-dimensional image space; The first feature distribution of the first modality image in the three-dimensional image space is obtained, the second feature distribution of the first generated image in the three-dimensional image space is obtained, and the domain constraint loss value is obtained based on the distance between the first feature distribution and the second feature distribution, the domain constraint loss value being used to determine the feature similarity of the first feature distribution and the second feature distribution corresponding in the three-dimensional image space. The training module is configured to train the first candidate network based on the constraint loss value to obtain an image conversion network, the image conversion network being configured to perform modality conversion on an image belonging to the first modality to obtain a three-dimensional image belonging to the second modality.

8. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one program, which is loaded and executed by the processor to implement the image generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is loaded and executed by the processor to implement the image generation method according to any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product comprises computer instructions, which are executed by the processor to implement the image generation method according to any one of claims 1 to 6.

Images