Image generation method, model training method and related device

Abstract

The invention discloses an image generation method, a model training method and a related device, and the method comprises the steps: extracting the input information to obtain a style feature, and enabling the style feature to be used for representing the noise level of an image; generating a target noise map corresponding to the to-be-processed image based on the style features; and fusing the target noise map with the to-be-processed image to obtain the target image consistent with the expected image noise level, so that the problem of noise enhancement and weakening of the to-be-processed image can be solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence processing, and in particular to an image generation method, a model training method, and related devices. Background Art

[0002] Image noise refers to interference signals or errors generated during image acquisition, transmission, or processing. These interference signals or errors are superimposed on the image signal in unpredictable ways. For example, the image quality of computed tomography (CT) technology is directly proportional to the X-ray radiation dose used. CT images with low X-ray doses contain more noise and have lower image quality.

[0003] In current image processing methods, only a single noise adjustment can be performed on the image, for example, only noise reduction can be performed on the image, and it is impossible to simultaneously solve the noise addition and noise reduction processing of the image. Summary of the Invention

[0004] This application at least provides an image generation method, a model training method and related devices.

[0005] The present application provides an image generation method, comprising: extracting style features from input information, the style features being used to characterize the image noise level; generating a target noise map corresponding to the image to be processed based on the style features; and fusing the target noise map with the image to be processed to obtain a target image consistent with the expected image noise level.

[0006] Therefore, the input information is used as a reference for the image noise level, and a noise map is generated based on the style features of the input information and fused with the image to be processed to adjust the noise of the image to be processed. Adjusting the input information can solve the problem of noise enhancement and weakening of the image to be processed.

[0007] Among them, generating a target noise map corresponding to the image to be processed based on the style features includes: encoding the noise information in the image to be processed to obtain the original noise features corresponding to the image to be processed; using the style features to guide the original noise features to perform noise reconstruction to obtain the target noise map.

[0008] Therefore, the noise in the image to be processed is encoded, and the style features are used to guide the noise reconstruction, so that the noise level of the target noise map is related to the style features. The target noise map is then fused with the image to be processed to achieve noise adjustment of the image to be processed according to the input information.

[0009] Among them, using style features to guide original noise features to perform noise reconstruction to obtain a target noise map includes: performing several feature transformation processes and several decoding processes on the original noise features in sequence to obtain a target noise map; wherein the output features of the previous processing are used as input features of the next processing, and for at least one output feature, the output features are adjusted using style features, and the adjusted output features are used as input features of the next processing.

[0010] Therefore, the features are adjusted during the noise reconstruction process according to the style features, so that the reconstructed target noise map is correlated with the style features.

[0011] The adjusting of the output features by using the style features includes: transforming the style features into adjustment parameters; and adjusting the statistical values ​​of the output features by using the adjustment parameters.

[0012] Therefore, the style features are transformed into adjustment parameters to adjust the statistical values ​​of the features, so as to adjust the features according to the style features during the noise reconstruction process.

[0013] The input information includes a reference image, and extracting the style features from the input information includes: performing style encoding on the reference image to obtain the style features representing the image noise level of the reference image.

[0014] Therefore, the input information can be in the form of an image, so that the image can be used to guide the noise adjustment of the image to be processed.

[0015] The input information includes random Gaussian noise, and extracting the style feature from the input information includes mapping the random Gaussian noise into a style feature consistent with a preset noise level.

[0016] Therefore, the input information can be in the form of noise, so that the noise can be used to guide the noise adjustment of the image to be processed.

[0017] Before fusing the target noise map with the image to be processed to obtain a target image that matches the expected image noise level, the method further includes: transforming the target noise map to enhance or weaken the target noise map, and the transformation includes at least one of filtering, blurring, contrast adjustment, and brightness adjustment.

[0018] Therefore, transforming the target noise map will affect the noise level of the target image, so adjusting the transformation step is implemented to obtain target images with different noise levels.

[0019] The method is implemented using an image generation model, and further includes: training a discriminator for discriminating whether an input image is a real image or a generated image; extracting sample input style features from sample input information using the image generation model, generating a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, fusing the sample noise map with the sample to-be-processed image to obtain a sample generated image; discriminating the sample generated image using a discriminator to obtain a sample discrimination result; determining a target loss based on the sample generated image and the sample discrimination result; and adjusting parameters of the image generation model based on the target loss.

[0020] Therefore, by co-training the discriminator and the image generation model, there is no need for real annotation and paired sample images, which reduces the training cost.

[0021] Wherein, determining the target loss based on the sample generated image and the sample judgment result includes at least one of the following: extracting the sample to be processed style features from the sample to be processed image, generating a sample reconstructed image based on the sample to be processed style features and the sample generated image, and determining the reconstruction loss based on the difference between the sample reconstructed image and the sample to be processed image; extracting the sample generated style features from the sample generated image, and determining the style reconstruction loss based on the difference between the sample generated style features and the sample input style features; when the sample input information includes first sample input information and second sample input information, determining the discreteness loss based on the difference between a first sample noise map corresponding to the first sample input information and a second sample noise map corresponding to the second sample input information; determining the adversarial loss based on the sample discrimination result and the discrimination result corresponding to the real image.

[0022] Therefore, at least one of the reconstruction loss, adversarial loss, style reconstruction loss, and discreteness loss is used to adjust the model parameters to improve the generation effect of the image generation model.

[0023] The image to be processed is a computed tomography image.

[0024] Therefore, it is possible to add noise or reduce noise in a computed tomography image.

[0025] The present application provides a model training method, comprising: training a discriminator for discriminating whether an input image is a real image or a generated image; extracting sample input style features from sample input information using an image generation model, generating a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, fusing the sample noise map with the sample to-be-processed image to obtain a sample generated image; discriminating the sample generated image using a discriminator to obtain a sample discrimination result; determining a target loss based on the sample generated image and the sample discrimination result, and adjusting parameters of the image generation model based on the target loss.

[0026] Therefore, by co-training the discriminator and the image generation model, there is no need for real annotation and paired sample images, which reduces the training cost.

[0027] The present application provides an image generation device, which includes an extraction module, a generation module and a fusion module. The extraction module is used to extract style features from input information, and the style features are used to characterize the image noise level; the generation module is used to generate a target noise map corresponding to the image to be processed based on the style features; and the fusion module is used to fuse the target noise map with the image to be processed to obtain a target image consistent with the expected image noise level.

[0028] The present application provides a model training device, which includes a discriminator training module, an image generation module, a discrimination module and an adjustment module. The discriminator training module is used to train a discriminator for discriminating whether an input image is a real image or a generated image; the image generation module is used to extract sample input style features from sample input information using an image generation model, generate a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, and fuse the sample noise map with the sample to-be-processed image to obtain a sample generated image; the discrimination module is used to discriminate the sample generated image using the discriminator to obtain a sample discrimination result; the adjustment module is used to determine a target loss based on the sample generated image and the sample discrimination result, and adjust the parameters of the image generation model based on the target loss.

[0029] The present application provides an electronic device comprising a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement any of the above-mentioned image generation methods or model training methods.

[0030] The present application provides a computer-readable storage medium having program instructions stored thereon, which implement any of the above-mentioned image generation methods or model training methods when executed by a processor.

[0031] In the above scheme, the input information is used as a reference for the image noise level, and a noise map is generated based on the style features of the input information and fused with the image to be processed to adjust the noise of the image to be processed. Adjusting the input information can enhance or weaken the noise of the image to be processed.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0034] Figure 1This is a flow chart of an embodiment of the image generation method of the present application;

[0035] Figure 2 This is a flowchart of another embodiment of step S120 of the present application;

[0036] Figure 3 is a schematic diagram of another embodiment of the image generation method of the present application;

[0037] Figure 4 This is a flow chart of an embodiment of the model training method of the present application;

[0038] Figure 5 This is a schematic diagram of the framework of an embodiment of the image generating device of the present application;

[0039] Figure 6 This is a schematic diagram of the framework of an embodiment of the model training device of the present application;

[0040] Figure 7 This is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0041] Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0042] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0043] In the following description, for the purpose of explanation rather than limitation, specific details such as specific subsystem structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0044] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0045] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the image generation method of the present application.

[0046] Specifically, the method may include:

[0047] Step S110: extracting style features from input information.

[0048] It should be noted that the image generation method provided in this application can be used to perform noise adjustment on an image to be processed to generate a corresponding noise-adjusted target image. Input information can be used to guide how to perform noise adjustment.

[0049] The specific form of the input information can be set according to actual application needs. For example, the form of the input information can include images, noise, etc.

[0050] The style features are used to characterize the image noise level and can be used to guide subsequent noise adjustment to obtain a target image with an image noise level consistent with an expected image noise level. The expected image noise level can be the same as the image noise level represented by the style features, or the expected image noise level can be obtained by processing the image noise level represented by the style features.

[0051] In some embodiments, the input information may include random Gaussian noise, and extracting the style feature from the input information may include: mapping the random Gaussian noise into a style feature consistent with a preset noise level.

[0052] In some embodiments, the input information may include a reference image, and extracting the style feature from the input information may include: performing style encoding on the reference image to obtain the style feature representing the image noise level of the reference image.

[0053] In some embodiments, the style feature extracted from the input information may be in a vector form, and may represent the noise level of the image represented by the input information.

[0054] Step S120: generating a target noise map corresponding to the image to be processed based on the style features.

[0055] The target noise map is generated based on the style features, and its noise level is correlated with the image noise level represented by the style features. It is then fused with the image to be processed, thereby adjusting the noise of the processed image to produce a target image that matches the expected image noise level.

[0056] The expected image noise level is related to the image noise level represented by the style feature. For example, it can be the image noise level represented by the style feature, or it can be obtained by processing the image noise level represented by the style feature.

[0057] In some embodiments, the target noise map may be directly fused with the image to be processed to obtain a target image, and the image noise level of the target image is consistent with the image noise level represented by the style features.

[0058] In some embodiments, before fusion, the target noise map may be transformed to enhance or weaken the target noise map, and the transformed target noise map is fused with the image to be processed to obtain a target image. The image noise level of the target image is related to the image noise level represented by the style features and the aforementioned transformation.

[0059] Step S130: Fusing the target noise map with the image to be processed to obtain a target image consistent with the expected image noise level.

[0060] The method of fusing the target noise map with the image to be processed may be set according to actual application, and may include but is not limited to adding the two.

[0061] The image noise level represented by the style features changes with the input information, thus affecting the noise adjustment of the target image. Compared to the original image to be processed, the target image has a different noise level. This change can be noise enhancement or noise reduction. The input information can be adjusted to select the noise enhancement or noise reduction method.

[0062] In some embodiments, the target noise map may be transformed before fusion to enhance or weaken the target noise map. In addition to the input information affecting whether noise enhancement or noise reduction is performed, the transformation of the target noise map may also affect whether noise enhancement or noise reduction is performed.

[0063] In a specific application scenario, the transformation of the target noise image may include at least one of filtering, blurring, contrast adjustment, and brightness adjustment. Of course, other transformations may also be provided according to actual application needs.

[0064] In some application scenarios, the image to be processed may be a CT image, and the above solution can be used to adjust the noise of the CT image.

[0065] It is understood that CT images can be divided into low-dose CT (LDCT) and normal-dose CT (NDCT), and the image noise levels of the two differ. NDCT images have high imaging quality but high radiation risk, while LDCT images have poor imaging quality but low radiation risk. The above methods can be used to adjust the noise of CT images. For example, NDCT images can be used to generate LDCT images, or LDCT images can be used to generate NDCT images, or a higher-noise NDCT image can be generated from an NDCT image, and so on.

[0066] The above scheme can provide a noise reduction solution for medical impact data, including but not limited to CT images. In addition, it can also add noise to images as a data augmentation strategy for downstream model training.

[0067] See also Figure 2 , Figure 2 1 is a flow chart of another embodiment of step S120 of the present application. Specifically, step S120 includes:

[0068] Step S221: Encode the noise information in the image to be processed to obtain the original noise features corresponding to the image to be processed.

[0069] It can be understood that the target noise map is fused with the original noise of the processed image to obtain an image with the expected image noise level. In order to make the target image style consistent with the expected image noise level, in addition to using style features, the noise of the processed image itself can also be considered when generating the target noise map.

[0070] The original noise feature corresponding to the image to be processed may represent the feature of the noise of the image to be processed itself, which is obtained by encoding the noise information in the image to be processed and does not include the image content information in the image to be processed.

[0071] Step S222: Use the style features to guide the original noise features to perform noise reconstruction to obtain a target noise map.

[0072] Noise reconstruction reconstructs a noise map based on the original noise features. The style features can be used as a reference for noise reconstruction to fuse with the image to produce a target image with the expected image noise level. The target noise map does not contain the image content information in the image to be processed.

[0073] In some embodiments, the original noise features are encoded features, and the noise reconstruction process may include: performing several feature transformation processes and several decoding processes on the original noise features in sequence to obtain a target noise map, and the output features of the previous process are used as the input features of the next process.

[0074] Wherein, for at least one output feature, the style feature is used to adjust the output feature, and the adjusted output feature is used as the input feature for the next processing.

[0075] In some embodiments, adjusting the output feature using the style feature may include: transforming the style feature into an adjustment parameter, and adjusting the statistical value of the output feature using the adjustment parameter.

[0076] In one implementation scenario, the mean and variance of the output features may be adjusted using adjustment parameters.

[0077] See also Figure 3 , Figure 3 It is a schematic diagram of another embodiment of the image generation method of the present application.

[0078] In this embodiment, the image generation method is implemented using an image generation model. The image generation model includes an input information processing module and a generator. The input information processing module is used to process input information to obtain style features. The specific submodules included in the input information processing module can be configured based on the form of the input information. The generator is used to generate a target noise map and a target image.

[0079] In this embodiment, the input information in the form of images and noise is used as an example for illustration. The input information processing module includes a style encoder and a mapping network. The style encoder is used to process the input information in the form of images, and the mapping network is used to process the input information in the form of noise.

[0080] In one implementation scenario, the input information includes a reference image, and the style encoder is used to perform style encoding on the reference image to obtain a style feature representing an image noise level of the reference image.

[0081] In a specific application scenario, the network details of the style encoder are as follows:

[0082] piece Input size Output size 3*3 convolution (1,128,128) (32,128,128) 3*3 convolution + downsampling convolution + 3*3 convolution (32,128,128) (64,64,64) 3*3 convolution + downsampling convolution + 3*3 convolution (64,64,64) (128,32,32) 3*3 convolution + downsampling convolution + 3*3 convolution (128,32,32) (256,16,16) 3*3 convolution + downsampling convolution + 3*3 convolution (256,16,16) (256,8,8) 3*3 convolution + downsampling convolution + 3*3 convolution (256,8,8) (256,4,4) 4*4 convolution (256,4,4) (256,1,1) Fully connected layer 256 64

[0083] Table 1 Network details of style encoder

[0084] Among them, the downsampling module is implemented using a 3x3 convolution kernel (sliding step size is 2), and the convolution layer is followed by IN (instance normalization) and LeakyReLU (LeakyRect ifi ed L i nearUn it) for regularization and activation.

[0085] Style Encoder E s Take reference image I s (image blocks of size 128x128) as input, and finally output 64-dimensional style features, namely V E =E s (I s ). Style Features V E Representation reference image I s Style, V E Able to represent the reference image I s noise level.

[0086] In a specific application scenario, the above image generation method can be used to process CT (Computed Tomography) images.

[0087] In one implementation scenario, the input information includes random Gaussian noise, and the mapping network is used to map the random Gaussian noise into style features consistent with a preset noise level.

[0088] In a specific application scenario, the above image generation method can be used to process CT (Computed Tomography) images. CT images can be divided into low-dose CT (LDCT) images and normal-dose CT (NDCT) images, which have different image noise levels. Based on this, the noise level can be divided into a noise level range corresponding to LDCT images and a noise level range corresponding to NDCT images. The preset noise level can be selected from one of these two ranges.

[0089] In a specific application scenario, a noise level identifier can be used to represent the preset noise level. For example, Y is used as the noise level identifier. Y is 1, indicating that the preset noise level is the noise level range corresponding to the LDCT image. Y is 0, indicating that the preset noise level is the noise level range corresponding to the NDCT image.

[0090] In a specific application scenario, the mapping network takes random Gaussian noise N(0,1) as input and has two output terminals, with the corresponding output terminal selected based on the value of Y. When Y is 0, the first output terminal is selected, and the image noise level represented by the output style features falls within the noise level range corresponding to the NDCT image. When Y is 1, the second output terminal is selected, and the image noise level represented by the output style features falls within the noise level range corresponding to the LDCT image. Of course, the mapping network can also use a single output terminal to output style features consistent with a preset noise level.

[0091] In a specific application scenario, mapping network V M To achieve V M =M(N|Y). The mapping network consists of a fully connected layer and uses ReLU as the activation function. The input random noise N is a 16-dimensional vector, and the output of the convolutional intermediate layer is a 512-dimensional vector, and the final output is a 64-dimensional style feature V. M , select the appropriate output port through Y according to the required style.

[0092] The network details of the mapped network are as follows:

[0093]

[0094] Table 2 Network details of the mapped network

[0095] like Figure 3As shown, the generator includes an encoder, a bottleneck layer and a decoder. The encoder is used to process the image I in The bottleneck module and decoder are used to use style features to guide the original noise features for noise reconstruction, resulting in a target noise map. The target noise map is then added to the image to be processed to obtain the target image.

[0096] like Figure 3 As shown, Figure 3 The blocks in the encoder, bottleneck layer, and decoder are explained in

[15] . Among them, the first module is a 3×3 convolutional layer (Conv), the second module includes instance normalization layer (IN, instance normalization) + LeakyReLU activation function + convolution layer (Conv) + downsampling layer (Down) + instance normalization layer + LeakyReLU activation function + convolution layer, the third module includes instance normalization layer (IN, instance normalization) + LeakyReLU activation function + convolution layer (Conv) + instance normalization layer (IN, instance normalization) + LeakyReLU activation function + convolution layer (Conv), and the fourth module includes adaptive instance normalization layer (AdaI N, Adaptive Instance Normalization) + LeakyReLU activation function + convolution layer (Conv) + adaptive instance normalization layer (AdaI N, Adaptive Instance Normalization) ion)+LeakyReLU activation function+convolution layer (Conv), the fifth module includes adaptive instance normalization layer (AdaIN, Adaptive Instance Normalization) + LeakyReLU activation function + convolution layer (Conv) + upsampling layer (Up) + adaptive instance normalization layer (AdaIN, Adaptive Instance Normalization) + LeakyReLU activation function + convolution layer (Conv).

[0097] The required style features V are received in the bottleneck layer and decoder through AdaIN (Adaptive Instance Normalization) in (V E or V M) and reconstruct the corresponding noise image, and add it to the original image to get the final output result, that is, I out =G(I in |V in )+I in . According to different V in It can realize the noise reduction of low-dose CT images and also realize the simulation generation of low-dose CT images.

[0098] Specifically, the style features are transformed into adjustment parameters, and the adjustment parameters are used to adjust the statistical values ​​of the output features. Figure 3 As shown in , the style features can be converted into adjustment parameters gamma and beta using linear transformation. IN processing (Instance Normalization) is performed on the input features, and the results of the IN processing are adjusted using the adjustment parameters, and the adjusted results are output.

[0099] It should be noted that the output of the network is a noise image, which does not contain image content information. The generated image is obtained by adding it to the original image. This method further ensures that the original information can be retained to the greatest extent.

[0100] In a specific application scenario, low-dose CT is subjected to noise reduction. An LDCT image is input and noise reduction of the LDCT image is achieved based on the style features of the NDCT image or the style features of zero standard deviation (noise-free).

[0101] In a specific application scenario, LDCT images with diverse noise levels can be generated as a data augmentation strategy for downstream models. This can be achieved in a variety of ways: based on reference images with varying noise levels, or based on Gaussian random noise maps with varying standard deviations, or based on a target noise map and attenuating / enhancing it, and then adding it to the original image to achieve quantitative and diverse image generation.

[0102] See also Figure 4 , Figure 4 It is a flow chart of an embodiment of the model training method of the present application.

[0103] Specifically, the model training method of the present application is used to train an image generation model, and the trained image generation model can be used to implement the aforementioned image generation method. The model training method may specifically include:

[0104] Step S410: training a discriminator for determining whether an input image is a real image or a generated image.

[0105] It should be noted that step S410 is the step of training the discriminator. During this process, the parameters of the image generation model are fixed. Steps S420-S450 are the steps of training the image generation model. During this process, the discriminator parameters are fixed. During the entire training process, the steps of training the discriminator and training the image generation model can be performed alternately multiple times.

[0106] Among them, the discriminator is used to determine whether the input image is a real image or an image generated by the image generation model.

[0107] In one implementation scenario, the discriminator may have two output terminals, one for determining the authenticity of the LDCT image and the other for determining the authenticity of the NDCT image. The discriminator may determine which of the LDCT image and the NDCT image to discriminate based on a set discrimination flag and select the corresponding output terminal to perform the discrimination. The discrimination flag may be set based on the image input to the discriminator.

[0108] In a specific application scenario, the Y value is set based on the image input to the discriminator (Y = 0 for NDCT images, and Y = 1 for LDCT images). The discriminator's final output has two outputs, namely two fully connected layers, used to determine the authenticity of the NDCT image and the LDCT image, respectively. Of course, in some cases, the discriminator can also include only a single output to determine the authenticity of the input image.

[0109] In a specific application scenario, I in represents the real image, the generated image I out The training process of the discriminator D uses a discriminative loss. Specifically, the discriminative loss may include at least one of a real image loss for discriminating a real image, a pseudo image loss for discriminating a pseudo image, and a gradient loss of the discriminator to the real image.

[0110] In a specific application scenario, the discriminant loss can be obtained as follows:

[0111] L D =L real +L fake +L R1

[0112] L real =2||D(I in |Y1)-1||2

[0113] L fake =||D(G(I in |V E )|Y2)||2+||D(G(I in |V M )|Y3)||2

[0114]

[0115] Among them, L D represents the discrimination loss, L real represents the real image loss, L fake represents the pseudo image loss, Represents the gradient loss. Y1 corresponds to I in Image category (NDCT image or LDCT image), Y2 corresponds to I s The image category, Y3 corresponds to the output category of the mapping network.

[0116] Step S420: Generate a sample generated image based on the sample input information and the sample to-be-processed image using the image generation model.

[0117] Specifically, the image generation model can extract sample input style features from the sample input information, generate a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, and fuse the sample noise map with the sample to-be-processed image to obtain the sample generated image. For details, please refer to the description of generating the target image in the previous embodiment and will not be repeated here.

[0118] Step S430: using the discriminator to discriminate the sample generated image to obtain a sample discrimination result.

[0119] Among them, the sample discrimination result indicates whether the sample generated image is a real image or a generated image.

[0120] In a specific application scenario, the sample identification result can be 0 or 1, where 0 indicates that the sample generated image is identified as a generated image, and 1 indicates that the sample generated image is identified as a real image.

[0121] Step S440: Determine the target loss based on the sample generation image and the sample discrimination result.

[0122] Generally speaking, paired NDCT and LDCT images have the same image content but different image noise levels. These images can be used to construct sample data, with one serving as the sample image to be processed and the other as the corresponding true label, which is then used as sample data for image denoising or image noise addition tasks. However, the acquisition cost of paired NDCT and LDCT images is very high. The model training method of this application utilizes an unsupervised generative model, eliminating the need for paired NDCT and LDCT images and reducing the cost of model training.

[0123] In some embodiments, the target loss may include at least one of an adversarial loss, a reconstruction loss, a discreteness loss, and a style reconstruction loss.

[0124] In some embodiments, after generating a sample generated image, the sample to-be-processed image is used as new sample input information to guide the sample generated image in generating a sample reconstructed image. Specifically, style features of the sample to-be-processed image can be extracted from the sample to-be-processed image, and a sample reconstructed image can be generated based on the style features of the sample to-be-processed image and the sample generated image. A reconstruction loss can be determined based on the difference between the sample reconstructed image and the sample to-be-processed image.

[0125] In a specific application scenario, the reconstruction loss can be obtained by the following formula:

[0126] I out =G(I in |V in )+I in

[0127] V E =E S (I in )

[0128] I rec =G(I out |V E )+I out

[0129] L cycle =||I rec -I in ||1

[0130] Among them, V in Represents the sample input style feature, which can be extracted using a style encoder or mapping network. in Represents the sample image to be processed, I out Represents the sample generated image. V E Represents the style features of the sample to be processed, I rec Represents the sample reconstructed image. L cycle represents the reconstruction loss.

[0131] The reconstruction loss ensures that no information is lost during the conversion process, while ensuring that the style encoder can correctly encode the style of the original image and reconstruct it.

[0132] In some embodiments, after generating a sample generated image, the sample generated image may be extracted to obtain a sample generated style feature, and a style reconstruction loss may be determined based on a difference between the sample generated style feature and the sample input style feature.

[0133] In a specific application scenario, the style reconstruction loss can be obtained by the following formula:

[0134] V rec =E S (G(I in|V in )+I in )

[0135] L sty =||V rec -V in ||1

[0136] Among them, V in Represents the sample input style feature, which can be extracted using a style encoder or mapping network. in Represents the sample image to be processed, V rec Represents the sample generation style feature, L sty Represents the style reconstruction loss. The style reconstruction loss ensures that the style of the generated image is consistent with the expected one.

[0137] In some embodiments, when the sample input information includes first sample input information and second sample input information, a first sample noise map is extracted corresponding to the first sample input information, and a second sample noise map is extracted corresponding to the second sample input information. A discreteness loss is determined based on the difference between the first sample noise map and the second sample noise map. It is understood that noise maps obtained from different input information should differ, and discreteness loss can indirectly constrain the differences in style features encoded from different input information.

[0138] In a specific application scenario, the discreteness loss can be obtained by the following formula:

[0139]

[0140] L ds =-||G(I in |V1)-G(I in |V2)||1

[0141] Among them, the first sample input information is Or N1, the first sample input information is Or N2, V1 represents the first sample input style feature corresponding to the first sample input information, V2 represents the second sample input style feature corresponding to the second sample input information, L ds Indicates the loss of discreteness.

[0142] By setting a discreteness loss on the noise maps generated by different style features, we ensure diversity and improve the style encoder's ability to encode noise levels. This loss indirectly constrains the style features encoded from different-dose CT images to be distinct, enabling them to characterize the noise level of the input style image, rather than simply providing a qualitative description of NDCT and LDCT images.

[0143] In some embodiments, when training the generator, it is expected that the sample generated image should be judged as a real image by the discriminator, and the adversarial loss can be determined based on the discrimination result of the sample and the discrimination result corresponding to the real image.

[0144] In a specific application scenario, the adversarial loss can be obtained as follows:

[0145]

[0146] Among them, V E V represents the sample input style feature extracted from the sample input information in the form of an image. M Represents the sample input style feature extracted from the sample input information in the form of noise, G(I in |V E ) can represent the sample generated image, Represents resistance to loss.

[0147] In some embodiments, the above various losses can be combined to obtain the final target loss. For example, it can be obtained by the following formula:

[0148]

[0149] Step S450: Adjust the parameters of the image generation model based on the target loss.

[0150] In a specific application scenario, the model was optimized for 200k iterations, using Adam as the optimizer, with the first-order momentum parameter beta1 set to 0 and the second-order momentum parameter beta2 set to 0.99.

[0151] In some embodiments, the image generation method may also include the above model training step, which is performed before using the model to generate images.

[0152] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of the image generating device of the present application.

[0153] In this embodiment, the image generation device 50 includes an extraction module 51, a generation module 52 and a fusion module 53. The extraction module 51 is used to extract style features from input information, and the style features are used to characterize the image noise level; the generation module 52 is used to generate a target noise map corresponding to the image to be processed based on the style features; the fusion module 53 is used to fuse the target noise map with the image to be processed to obtain a target image consistent with the expected image noise level.

[0154] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of the model training device of the present application.

[0155] In this embodiment, the model training device 60 includes a discriminator training module 61, an image generation module 62, a discrimination module 63 and an adjustment module 64. The discriminator training module 61 is used to train a discriminator for discriminating whether an input image is a real image or a generated image; the image generation module 62 is used to use an image generation model to extract sample input style features from sample input information, generate a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, and fuse the sample noise map with the sample to-be-processed image to obtain a sample generated image; the discrimination module 63 is used to use a discriminator to discriminate the sample generated image to obtain a sample discrimination result; the adjustment module 64 is used to determine the target loss based on the sample generated image and the sample discrimination result, and adjust the parameters of the image generation model based on the target loss.

[0156] See also Figure 7 , Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application.

[0157] In this embodiment, electronic device 70 includes memory 71 and processor 72. Processor 72 is configured to execute program instructions stored in memory 71 to implement any of the above-described image generation methods or model training methods. In a specific implementation scenario, electronic device 70 may include, but is not limited to, computer equipment, electrical equipment, microcomputers, desktop computers, and servers. Furthermore, electronic device 70 may also include mobile devices such as laptop computers and tablet computers, which are not limited herein.

[0158] Specifically, the processor 72 is used to control itself and the memory 71 to implement any of the above-mentioned image generation methods or model training methods. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0159] See also Figure 8 , Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0160] The computer-readable storage medium 80 provided in this embodiment stores program instructions 81 that can be run by a processor. When the program instructions 81 are executed by the processor, they are used to implement any of the above-mentioned image generation methods or model training methods.

[0161] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another subsystem, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0163] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. An image generation method, characterized in that: The method comprises: Extracting style features from input information, where the style features are used to characterize image noise levels; Generating a target noise map corresponding to the image to be processed based on the style feature; The target noise map is fused with the image to be processed to obtain a target image consistent with the expected image noise level.

2. The method according to claim 1, characterized in that Generating a target noise map corresponding to the image to be processed based on the style feature includes: Encoding the noise information in the image to be processed to obtain the original noise features corresponding to the image to be processed; The style feature is used to guide the original noise feature to perform noise reconstruction to obtain the target noise map.

3. The method according to claim 2, characterized in that The using the style feature to guide the original noise feature to perform noise reconstruction to obtain the target noise map comprises: The original noise features are sequentially subjected to several feature transformation processes and several decoding processes to obtain the target noise map; wherein the output features of the previous process are used as the input features of the next process, and for at least one of the output features, the output features are adjusted using the style features, and the adjusted output features are used as the input features of the next process.

4. The method according to claim 3, characterized in that The adjusting the output feature by using the style feature comprises: transforming the style features into adjustment parameters; The statistical value of the output feature is adjusted using the adjustment parameter.

5. The method according to any one of claims 1 to 4, characterized in that The input information includes a reference image, and the style features extracted from the input information include: Style encoding is performed on the reference image to obtain the style feature representing the image noise level of the reference image.

6. The method according to any one of claims 1 to 4, characterized in that The input information includes random Gaussian noise, and the style features extracted from the input information include: The random Gaussian noise is mapped to the style feature consistent with a preset noise level.

7. The method according to any one of claims 1 to 6, characterized in that Before fusing the target noise map with the image to be processed to obtain a target image consistent with the expected image noise level, the method further includes: The target noise map is transformed to enhance or weaken the target noise map, wherein the transformation comprises at least one of filtering, blurring, contrast adjustment, and brightness adjustment.

8. The method according to any one of claims 1 to 7, characterized in that The method is implemented using an image generation model, and the method further comprises: Train a discriminator to distinguish whether the input image is a real image or a generated image; Extracting sample input style features from sample input information using the image generation model, generating a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, and fusing the sample noise map with the sample to-be-processed image to obtain a sample generated image; Using the discriminator to discriminate the sample generated image to obtain a sample discrimination result; Determining target loss based on the sample generated image and the sample discrimination result; Based on the target loss, parameters of the image generation model are adjusted.

9. The method according to claim 8, characterized in that The determining of the target loss based on the sample generated image and the sample discrimination result includes at least one of the following: Extracting a sample to-be-processed style feature from the sample to-be-processed image, generating a sample reconstructed image based on the sample to-be-processed style feature and the sample generated image, and determining a reconstruction loss based on a difference between the sample reconstructed image and the sample to-be-processed image; Extracting a sample generated style feature from the sample generated image, and determining a style reconstruction loss based on a difference between the sample generated style feature and the sample input style feature; In a case where the sample input information includes first sample input information and second sample input information, determining a dispersion loss based on a difference between a first sample noise map corresponding to the first sample input information and a second sample noise map corresponding to the second sample input information; The adversarial loss is determined based on the sample discrimination result and the discrimination result corresponding to the real image.

10. The method according to any one of claims 1 to 9, characterized in that The image to be processed is a computer tomography image.

11. A model training method, characterized in that: The method comprises: Train a discriminator to distinguish whether the input image is a real image or a generated image; Extracting sample input style features from sample input information using the image generation model, generating a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, and fusing the sample noise map with the sample to-be-processed image to obtain a sample generated image; Using the discriminator to discriminate the sample generated image to obtain a sample discrimination result; Based on the sample generated image and the sample discrimination result, a target loss is determined, and based on the target loss, parameters of the image generation model are adjusted.

12. An image generating device, characterized in that: The device comprises: An extraction module, used to extract style features from input information, wherein the style features are used to characterize the image noise level; A generating module, used for generating a target noise map corresponding to the image to be processed based on the style feature; The fusion module is used to fuse the target noise map with the image to be processed to obtain a target image consistent with the expected image noise level.

13. A model training device, characterized in that: The device comprises: A discriminator training module, used to train a discriminator for distinguishing whether an input image is a real image or a generated image; An image generation module is used to extract sample input style features from sample input information using the image generation model, generate a sample noise map corresponding to the sample to-be-processed image based on the sample input style features, and fuse the sample noise map with the sample to-be-processed image to obtain a sample generated image; A discrimination module, used for discriminating the sample generated image using the discriminator to obtain a sample discrimination result; An adjustment module is used to determine a target loss based on the sample generated image and the sample discrimination result, and adjust the parameters of the image generation model based on the target loss.

14. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images