A training method for an image generation model and an image generation method

Abstract

The embodiment of the present application provides a training method for an image generation model and an image generation method, which relate to the field of image processing technology. The training method includes: inputting a first sample image into an image generation model, allowing the image generation model to encode the first sample image, based on the obtained image feature vector, using a clean image prior codebook and a rainbow pattern image prior codebook to respectively predict the clean image code and the rainbow pattern image code, performing encoding fusion on the background layer code and the rainbow pattern layer code obtained by comparing the clean image code and the rainbow pattern image code, decoding the obtained target rainbow pattern image code, and obtaining a rainbow pattern image; based on the first sample image and the rainbow pattern image, calculating a first loss value, and adjusting the model parameters of the image generation model based on the first loss value. As can be seen from the above, by applying the solution provided in the embodiment of the present application, the obtained image generation model can be trained to generate a rainbow pattern window image with higher fidelity.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for an image generation model and an image generation method. Background Art

[0002] Usually, in order to block the heat generated by infrared rays and reduce the harm of ultraviolet rays to people in the space, a film can be applied on the inside of the glass installed on the light-transmitting windows set in the space. For example, in order to improve the riding experience of passengers, a film can be applied on the inside of the front windshield of the vehicle. Furthermore, when the image of the filmed window is collected, due to the refraction and reflection of natural light, rainbow patterns may often be formed in the collected image to block the facial image, object image, etc. in the image, reducing the accuracy of image recognition.

[0003] Take a car window with a film as an example. When natural light shines on the window surface, part of the light is reflected on the window surface, while the other part passes through the window glass and reaches the film on the inside of the window. Among them, the light reaching the film is partially polarized light, part of which is reflected back to the outside of the car after crystal birefringence, and the other part passes through the film into the car, and is reflected back to the outside after being reflected by objects inside the car. The light reflected back to the outside of the car can be received by the lens of the image acquisition device. When the light reflected back to the outside of the car after crystal birefringence passes through the IR (Infrared) polarizer set by the lens, interference can occur to form rainbow patterns in the collected window image.

[0004] Generally, an image without rainbow streaks can be called a clean image, while an image with rainbow streaks can be called a rainbow streak image. For example, FIG3 (a) is a clean image, while FIG3 (b) and FIG3 (c) are rainbow streak images. Since the rainbow streaks in a rainbow streak image will block the details of objects such as people and objects in the background layer of the image, in an image recognition scenario, in order to improve the accuracy of image recognition, it is necessary to train an image processing model for eliminating rainbow streaks in rainbow streak images. In order to train the above image processing model, it is necessary to obtain a large number of rainbow streak images in advance. Based on this, how to train the obtained image generation model to generate rainbow streak images with high fidelity has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiment of the present application is to provide a training method for an image generation model and an image generation method, so as to generate a rainbow pattern window image with high fidelity by training an image generation model. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for training an image generation model, the method comprising:

[0007] Acquire a first sample image; wherein the first sample image is a clean image;

[0008] Inputting the first sample image into the image generation model to be trained, so that the image generation model encodes the first sample image to obtain an image feature vector, predicting a clean image code based on the image feature vector using a preset clean image priori codebook, and predicting a rainbow image code based on the image feature vector using a preset rainbow image priori codebook, comparing the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, performing encoding fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, and decoding the target rainbow image code to obtain a rainbow image;

[0009] Calculating a first loss value based on the first sample image and the rainbow pattern image, and adjusting a model parameter of the image generation model based on the first loss value;

[0010] Among them, the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0011] Optionally, in a specific implementation, the image generation model includes a Transforemer module;

[0012] The image generation model predicts clean image coding based on the image feature vector using a preset clean image prior codebook, and predicts rainbow image coding based on the image feature vector using a preset rainbow image prior codebook, including:

[0013] Using the Transforemer module to perform feature calculation on the image feature vector to obtain a coding sequence corresponding to the image feature vector;

[0014] For each coding sequence, in a preset clean image priori codebook, a clean window code item with the highest similarity to the coding sequence is retrieved, and the coding sequence is predicted using the retrieved clean window code item to obtain a clean image code;

[0015] For each coding sequence, the rainbow window code item with the highest similarity to the coding sequence is retrieved in the preset rainbow image priori codebook, and the coding sequence is predicted using the retrieved rainbow window code item to obtain the rainbow image code.

[0016] Optionally, in a specific implementation, the Transforemer module includes: a plurality of self-attention submodules;

[0017] The using the Transforemer module to perform feature calculation on the image feature vector to obtain a coding sequence corresponding to the image feature vector includes:

[0018] Based on the image feature vector, sequentially use the multiple self-attention submodules to perform specified calculations to obtain a calculation result output by the last self-attention module;

[0019] The calculation result is projected to a specific dimension matching the dimension of the image feature vector using a linear layer to obtain a coding sequence corresponding to the image feature vector; wherein the specific dimension is: , express and The dimension of the image feature vector is .

[0020] Optionally, in a specific implementation, each self-attention submodule performs a specified calculation based on the image feature vector, including:

[0021] Each self-attention submodule performs a specified calculation based on the image feature vector using a first formula; wherein the first formula is:

[0022] ;

[0023] in, is the image feature vector, is the calculation result of the s-th self-attention module, is the calculation result of the s+1th self-attention submodule, 0≤s≤M-1, M is the number of the multiple self-attention submodules; for Query value, for Key value, for The Value value.

[0024] Optionally, in a specific implementation, the image generation model compares the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, and performs code fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, including:

[0025] Using the second formula, the target rainbow pattern image code that meets the optimization target is calculated;

[0026] Wherein, the second formula is:

[0027] ;

[0028] in, Encode the target rainbow pattern image, The first A coding sequence, is the first A coding sequence, The first A coding sequence, is the first A coding sequence, is a preset parameter, and N is a preset codebook dimension;

[0029] The optimization goal is:

[0030] + ;

[0031] in, Encode the target rainbow pattern image, is the pre-set self-consistent data fidelity and , is the clean image prior codebook, is the rainbow image prior codebook, for The weight coefficient of for The weight coefficient of .

[0032] Optionally, in a specific implementation, the training method of the discrete vector generation model includes:

[0033] Acquire a second sample image; wherein the second sample image includes: a clean image and a rainbow pattern image;

[0034] For each second sample image, perform the following steps:

[0035] The second sample image is input into a discrete vector generation model to be trained, so that the discrete vector generation model encodes the second sample image to obtain an intermediate feature vector of the second sample image, and a discrete vector closest to each encoding position of the intermediate feature vector is retrieved from a preset discrete vector codebook, a discrete encoding vector of the second sample image is generated by using the retrieved discrete vector, and the discrete encoding vector is decoded to obtain a reconstructed image corresponding to the second sample image; wherein the preset discrete vector codebook records the discrete encoding vector of the second sample image that has been generated;

[0036] A second loss value is calculated based on the reconstructed image and the second sample image, and a model parameter of the discrete vector generation model is adjusted based on the second loss value.

[0037] Optionally, in a specific implementation manner, before adjusting the model parameters of the image generation model based on the first loss value, the method further includes:

[0038] Inputting the rainbow pattern image into an image classification model so that the image classification model outputs a confidence loss value of the rainbow pattern image; wherein the image classification model is obtained by training a classification model to be trained based on a pre-acquired sample clean image and a sample rainbow pattern image;

[0039] The adjusting the model parameters of the image generation model based on the first loss value includes:

[0040] Based on the first loss value and the confidence loss value, adjust the model parameters of the image generation model.

[0041] In a second aspect, an embodiment of the present application provides an image generation method, the method comprising:

[0042] Acquire an initial image; wherein the initial image is a clean image;

[0043] Input the initial image into a preset image generation model so that the image generation model encodes the initial image to obtain an initial image feature vector, predict the initial clean image code based on the initial image feature vector using a preset clean image priori codebook, and predict the initial rainbow image code based on the initial image feature vector using a rainbow image priori codebook, compare the initial clean image code with the initial rainbow image code to obtain an initial background layer code and an initial rainbow layer code, perform encoding fusion on the initial background layer code and the initial rainbow layer code, determine the rainbow image code to be generated, decode the rainbow image code to be generated, and obtain a target rainbow image;

[0044] Acquire the target rainbow pattern image output by the image generation model;

[0045] Among them, the image generation model is: a model trained based on any training method provided in the first aspect above.

[0046] In a third aspect, an embodiment of the present application provides a training device for an image generation model, the device comprising:

[0047] A sample image acquisition module, used to acquire a first sample image; wherein the first sample image is a clean image;

[0048] A first input module is used to input the first sample image into the image generation model to be trained, so that the image generation model encodes the first sample image to obtain an image feature vector, predicts a clean image code based on the image feature vector using a preset clean image priori codebook, and predicts a rainbow image code based on the image feature vector using a preset rainbow image priori codebook, compares the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, performs encoding fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, and decodes the target rainbow image code to obtain a rainbow image;

[0049] A first calculation module, configured to calculate a first loss value based on the first sample image and the rainbow pattern image, and adjust a model parameter of the image generation model based on the first loss value;

[0050] Among them, the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0051] In a fourth aspect, an embodiment of the present application provides an image generating device, the device comprising:

[0052] A first initial image acquisition module, used to acquire an initial image; wherein the initial image is a clean image;

[0053] A second input module is used to input the initial image into a preset image generation model so that the image generation model encodes the initial image to obtain an initial image feature vector, predict an initial clean image code based on the initial image feature vector using a preset clean image priori codebook, and predict an initial rainbow image code based on the initial image feature vector using a rainbow image priori codebook, compare the initial clean image code with the initial rainbow image code to obtain an initial background layer code and an initial rainbow layer code, perform code fusion on the initial background layer code and the initial rainbow layer code, determine a rainbow image code to be generated, decode the rainbow image code to be generated, and obtain a target rainbow image;

[0054] A first target image acquisition module, used to acquire the target rainbow pattern image output by the image generation model;

[0055] Among them, the image generation model is: a model trained based on any training method provided in the first aspect above.

[0056] In a fifth aspect, an embodiment of the present application provides an image generation method, the method comprising:

[0057] Acquire an initial image; wherein the initial image is a clean image;

[0058] Inputting the initial image into a trained image generation model, so that the image generation model generates a target rainbow pattern image with the initial image as a background layer based on the initial image, a preset clean image priori codebook and a preset rainbow pattern image priori codebook;

[0059] Acquire the target rainbow pattern image output by the image generation model;

[0060] Among them, the image generation model is obtained by training a preset image generation model based on a first sample image, the clean image prior codebook and the rainbow pattern image prior codebook, and the first sample image is a clean image; the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0061] In a sixth aspect, an embodiment of the present application provides an image generating device, the device comprising:

[0062] A second initial acquisition module, used to acquire an initial image; wherein the initial image is a clean image;

[0063] A third input module is used to input the initial image into the trained image generation model, so that the image generation model generates a target rainbow pattern image with the initial image as the background layer based on the initial image, a preset clean image prior codebook and a preset rainbow pattern image prior codebook;

[0064] A second target image acquisition module, used to acquire the target rainbow pattern image output by the image generation model;

[0065] Among them, the image generation model is obtained by training a preset image generation model based on a first sample image, the clean image prior codebook and the rainbow pattern image prior codebook, and the first sample image is a clean image; the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0066] In a seventh aspect, an embodiment of the present application provides an electronic device, including:

[0067] Memory, used to store computer programs;

[0068] The processor is used to implement any one of the methods provided in the first aspect, the second aspect and / or the fifth aspect when executing the program stored in the memory.

[0069] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods provided in the first aspect and / or the second aspect is implemented.

[0070] In a ninth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods provided in the first and / or second aspects above.

[0071] Beneficial effects of the embodiments of the present application:

[0072] As can be seen from the above, by applying the solution provided by the embodiment of the present application, a clean image prior codebook and the rainbow pattern image prior codebook can be pre-constructed, and then, the clean image as the sample image, as well as the clean image prior codebook and the rainbow pattern image prior codebook can be used to train the image generation model to be trained. In the training process, the image generation model encodes the sample image to obtain an image feature vector, and based on the image feature vector, the clean image code is predicted using the preset clean image prior codebook, and based on the image feature vector, the rainbow pattern image code is predicted using the preset rainbow pattern image prior codebook, and the clean image code and the rainbow pattern image code are compared to obtain the background layer code and the rainbow pattern layer code, and the background layer code and the rainbow pattern layer code are coded and fused, and the target rainbow pattern image code is decoded to obtain a rainbow pattern image. Furthermore, a first loss value can be calculated based on the obtained rainbow pattern image and the sample image, and the model parameters of the above image generation model can be adjusted based on the first loss value.

[0073] Based on this, when the above-mentioned image generation model converges, a trained image generation model can be obtained, and the trained image generation model can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity. In this way, by applying the scheme provided by the embodiment of the present application, an image generation model can be trained based on an unsupervised training method to achieve the degeneration of the clean image to a rainbow pattern image with rainbow patterns, and the difference between the rainbow pattern distribution in the degraded rainbow pattern image and the rainbow pattern distribution of the real rainbow pattern image is small, and at the same time, the background layer of the generated rainbow pattern image has a high fidelity with the clean image, that is, the trained image generation model can be used to generate a rainbow pattern image with high fidelity using a clean image.

[0074] In addition, compared with the related technology of rainbow pattern processing, which separates the rainbow pattern layer in the rainbow pattern image by manually adjusting parameters and extracting individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image. Therefore, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image can be performed without being limited to the display of the rainbow pattern shape, rainbow pattern color, etc. Moreover, in the embodiment of the present application, when training the above-mentioned image generation model, it does not rely on the paired data of the clean image and the rainbow pattern image for image model training, so that the trained image generation model can have higher generalization, and the use of discrete vector coding compression technology can reduce the computing resource consumption of the image generation model, so that the image generation model can be trained on a large number of data sets, increase the number of model iterations and model iteration efficiency, and improve the reasoning effect of the model.

[0075] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0077] Figure 1 A flowchart of a training method for a discrete vector generation model provided in an embodiment of the present application;

[0078] Figure 2 A schematic diagram of a network structure of a VQGAN as a discrete vector generation model provided in an embodiment of the present application;

[0079] Figure 3 (a) is a clean image;

[0080] FIG3( b ) is a rainbow pattern image generated by the image generation model provided in an embodiment of the present application based on FIG3( a );

[0081] FIG3 (c) is another rainbow pattern image generated by the image generation model provided in an embodiment of the present application based on FIG3 (a);

[0082] FIG3( d ) is an image obtained by removing the rainbow pattern from FIG3( c );

[0083] Figure 4 A flowchart of a method for training an image generation model provided in an embodiment of the present application;

[0084] Figure 5A schematic diagram of the structure of an image generation model provided in an embodiment of the present application;

[0085] Figure 6 A schematic diagram of a flow chart of another method for training an image generation model provided in an embodiment of the present application;

[0086] Figure 7 A schematic diagram of a training method for an image generation model provided in an embodiment of the present application;

[0087] Figure 8 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application;

[0088] Fig. 9 A schematic diagram of a flow chart of another image generation method provided in an embodiment of the present application;

[0089] Fig.10 A schematic diagram of the structure of a training device for an image generation model provided in an embodiment of the present application;

[0090] Fig.11 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0091] Fig.12 A schematic diagram of the structure of another image generating device provided in an embodiment of the present application;

[0092] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0094] Currently, in image recognition scenarios, in order to improve the accuracy of image recognition, it is necessary to train an image processing model for eliminating rainbow patterns in rainbow images. In order to train the above image processing model, it is necessary to obtain a large number of rainbow images in advance. Based on this, how to train the obtained image generation model to generate rainbow patterns with higher fidelity has become a technical problem that needs to be solved urgently.

[0095] In order to solve the above technical problems, an embodiment of the present application provides a training method for an image generation model.

[0096] Among them, the training method of the image generation model can be applied to various scenarios that require the generation of rainbow pattern images, for example, generating rainbow pattern images of car windows to train image processing models for eliminating rainbow patterns, improve the recognition accuracy of car window images, etc. Moreover, the execution subject of the training method of the image generation model can be various electronic devices, which are specifically used for training models. In specific applications, the electronic device can be a server, a desktop computer, etc., and the electronic device can be a single electronic device or an electronic device cluster composed of multiple electronic devices. Based on this, the present application does not specifically limit the application scenarios and execution subjects of the training method of the image generation model.

[0097] The present invention provides a method for training an image generation model, which may include the following steps:

[0098] Acquire a first sample image; wherein the first sample image is a clean image;

[0099] Inputting the first sample image into the image generation model to be trained, so that the image generation model encodes the first sample image to obtain an image feature vector, predicting a clean image code based on the image feature vector using a preset clean image priori codebook, and predicting a rainbow image code based on the image feature vector using a preset rainbow image priori codebook, comparing the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, performing encoding fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, and decoding the target rainbow image code to obtain a rainbow image;

[0100] Calculating a first loss value based on the sample image and the rainbow pattern image, and adjusting a model parameter of the image generation model based on the first loss value;

[0101] Among them, the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; wherein the second sample image includes: a clean image and a rainbow pattern image.

[0102] As can be seen from the above, by applying the solution provided by the embodiment of the present application, a clean image prior codebook and the rainbow pattern image prior codebook can be pre-constructed, and then, the clean image as the sample image, as well as the clean image prior codebook and the rainbow pattern image prior codebook can be used to train the image generation model to be trained. In the training process, the image generation model encodes the sample image to obtain an image feature vector, and based on the image feature vector, the clean image code is predicted using the preset clean image prior codebook, and based on the image feature vector, the rainbow pattern image code is predicted using the preset rainbow pattern image prior codebook, and the clean image code and the rainbow pattern image code are compared to obtain the background layer code and the rainbow pattern layer code, and the background layer code and the rainbow pattern layer code are coded and fused to obtain the target rainbow pattern image code, and the target rainbow pattern image code is decoded to obtain the rainbow pattern image. Furthermore, based on the obtained rainbow pattern image and the sample image, a first loss value can be calculated, and the model parameters of the above image generation model can be adjusted based on the first loss value.

[0103] Based on this, when the above-mentioned image generation model converges, a trained image generation model can be obtained, and the trained image generation model can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity. In this way, by applying the scheme provided by the embodiment of the present application, an image generation model can be trained based on an unsupervised training method to achieve the degeneration of the clean image to a rainbow pattern image with rainbow patterns, and the difference between the rainbow pattern distribution in the degraded rainbow pattern image and the rainbow pattern distribution of the real rainbow pattern image is small, and at the same time, the background layer of the generated rainbow pattern image has a high fidelity with the clean image, that is, the trained image generation model can be used to generate a rainbow pattern image with high fidelity using a clean image.

[0104] In addition, compared with the related technology of rainbow pattern processing, which separates the rainbow pattern layer in the rainbow pattern image by manually adjusting parameters and extracting individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image. Therefore, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image can be performed without being limited to the display of the rainbow pattern shape, rainbow pattern color, etc. Moreover, in the embodiment of the present application, when training the above-mentioned image generation model, it does not rely on the paired data of the clean image and the rainbow pattern image for image model training, so that the trained image generation model can have higher generalization, and the use of discrete vector coding compression technology can reduce the computing resource consumption of the image generation model, so that the image generation model can be trained on a large number of data sets, increase the number of model iterations and model iteration efficiency, and improve the reasoning effect of the model.

[0105] In an embodiment of the present invention, before performing image generation model training, it is necessary to first obtain a clean image prior codebook and a rainbow pattern image prior codebook, and the above-mentioned clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by the discrete vector generation model, and the above-mentioned rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; and the above-mentioned discrete vector generation model is a second sample image, which is obtained by training the discrete vector generation model to be trained, wherein the above-mentioned second sample image includes a clean image and a rainbow pattern image, that is, the above-mentioned discrete vector generation model is obtained by training the discrete vector generation model to be trained using the clean image and the rainbow pattern image.

[0106] As described above, in an embodiment of the present invention, a second sample image including a clean image and a rainbow pattern image is first obtained, and then the discrete vector generation model to be trained is trained using the second sample image to obtain a trained discrete vector generation model; thereafter, the clean image is encoded using the trained discrete vector generation model to obtain a clean image prior codebook, and based on a preset discrete vector codebook, the rainbow pattern image is encoded using the trained discrete vector generation model to obtain a rainbow pattern image prior codebook.

[0107] Among them, the above-mentioned discrete vector generation model is also called discretization vector generation model, discrete coding vector generation model, etc.

[0108] Based on this, before specifically describing a training method for an image generation model provided by an embodiment of the present invention, the training method for the above-mentioned discrete vector generation model is first described in detail.

[0109] Optionally, in a specific implementation, such as Figure 1As shown, the training method of the above discrete vector generation model may include the following steps S101-S103.

[0110] S101: Acquire a second sample image;

[0111] The second sample image includes: a clean image and a rainbow pattern image.

[0112] In this specific implementation, at least one real clean image and at least one real rainbow pattern image may be first obtained as the second sample image, wherein the so-called real clean image and real rainbow pattern image refer to: a real clean image and a real rainbow pattern image acquired by an image acquisition device, rather than a clean image and a rainbow pattern image generated by an image generation model or the like.

[0113] S102: For each second sample image, the second sample image is input into the discrete vector generation model to be trained, so that the discrete vector generation model encodes the second sample image to obtain an intermediate feature vector of the second sample image, and in a preset discrete vector codebook, a discrete vector closest to each encoding position of the intermediate feature vector is retrieved, and a discrete encoding vector of the second sample image is generated using the retrieved discrete vector, and the discrete encoding vector is decoded to obtain a reconstructed image corresponding to the second sample image.

[0114] The preset discrete vector codebook records the generated discrete coding vector of the second sample image.

[0115] For each second sample image, the second sample image can be input into the discrete vector generation model to be trained. In this way, the discrete vector generation model to be trained can encode the second sample image to obtain the encoding vector of the sample image, which is called the intermediate feature vector of the second sample image; wherein the discrete vector generation model can be provided with an encoder, so that the discrete vector generation model uses the encoder to encode the second sample image to obtain the intermediate feature vector.

[0116] Afterwards, the discrete vector generation model can retrieve the discrete vector closest to each coding position of the intermediate feature vector in the preset discrete vector codebook. The so-called coding position is the coding sequence composed of each row or column in the vector, and the so-called distance is the mean square error loss. In this way, for the second sample image, the same number of discrete vectors as the coding positions of the intermediate feature vector can be obtained, so that the retrieved discrete vectors can be used to generate discrete coding vectors.

[0117] The above-mentioned preset discrete vector codebook records the discrete coding vector of the generated second sample image. Specifically, for the second sample image first input into the discrete vector generation model to be trained, the intermediate feature vector obtained by encoding it can be used as a discrete coding vector and recorded in the preset discrete vector codebook; for the second sample image input into the discrete vector generation model to be trained again, after encoding it to obtain the intermediate feature vector, the discrete vector closest to each coding position of the above-mentioned intermediate feature vector can be retrieved from the discrete coding vector of the generated second sample image recorded in the preset discrete vector codebook. Correspondingly, for the second sample image input into the discrete vector generation model to be trained again, after the discrete coding vector of the second sample image is generated using the retrieved discrete vector, the discrete coding vector of the second sample image is recorded in the above-mentioned preset discrete vector codebook.

[0118] Afterwards, for each second sample image, the discrete coding vector of the second sample image obtained above can be decoded to obtain the reconstructed image corresponding to the second sample image; wherein the discrete vector generation model can be provided with a decoder, so that the discrete vector generation model uses the above decoder to decode the discrete coding vector to obtain the above reconstructed image.

[0119] As can be seen from the above, the processing process of the above-mentioned discrete vector generation model to be trained for each second sample image is essentially: information discretization modeling of the image data of the second sample image, that is, for each second sample image, the continuous image data is represented as a discretized vector. The above-mentioned discrete vector retrieval and discrete coding vector generation process can be called quantization.

[0120] In order to realize the information discretization modeling of the image data of the second sample image, the above-mentioned discrete vector generation model can be designed based on the VQGAN (Vector Quantization Generative Adversarial Network), which mainly uses VQ (Vector Quantization) data compression technology to represent continuous data as discretized vectors. Therefore, the input image of the above-mentioned discrete vector generation model can be mapped to the discretized vector representation in the VQ space.

[0121] For example, Figure 2 As shown in Figure 1, it is a schematic diagram of the network structure of VQGAN as a discrete vector generation model. Figure 2For example, the continuous data represented in the form of a sequence expresses the components of the image, and the complexity of the information discretization modeling of image data is not based on a single pixel, but through a method of using a learning discrete vector codebook to convert any RGB image data input to the discrete vector generation model , represented by a discrete vector with the same dimension as the discrete vector in the learned discrete vector codebook. middle, and are the height and width of the RGB image data, and 3 is the number of channels of the RGB image data. That is, for RGB image data, the dimension of the entire image pixel of the RGB image data is , and 3 corresponds to the dimension of each pixel, that is, each pixel has an R (red) channel value, a G (green) channel value, and a B (blue) channel value.

[0122] Among them, Figure 2 In the VQGAN shown in the figure, the Resnet (Residual Network) block is used to build the encoder network structure to obtain the CNN encoder After the input image (i.e., the second sample image) is input into the discrete vector generation model, it can be passed through the CNN encoder Encode the input image and get the intermediate feature vector Among them, the above intermediate eigenvector is the above CNN encoder The discrete vector set obtained by compressing the input image pixels can also be called the codebook of the input image. and is the codebook dimension of the intermediate feature vector; in addition, is the vector dimension of each discrete vector in the above discrete vector set, which can also be called the vector dimension of the intermediate feature vector. For example, Can be , Furthermore, in order to be able to use the preset discrete vector codebook in the subsequent implementation, the above intermediate feature vector Discretize the code into a discrete code vector , the codebook dimension of the preset discrete vector codebook is the same as the codebook dimension of the above intermediate feature vector, and the vector dimension of the discrete vectors in the preset discrete vector codebook is the same as the vector dimension of each discrete vector in the above discrete vector set, that is, and is the codebook dimension of the preset discrete vector codebook, is the vector dimension of the discrete vector in the preset discrete vector codebook.

[0123] At this time, the preset discrete vector codebook is introduced ,Right now Figure 2 In this way, for the intermediate feature vector For each coding position, the nearest neighbor search method is used to find the preset discrete vector codebook Retrieve the discrete vector closest to the encoding position In this way, after all discrete vectors are retrieved, all the coded values ​​retrieved are used to generate A vector with the same dimension as the input image is obtained by dividing it into discrete encoding vectors. The above-mentioned discrete vector retrieval and discrete coding vector generation process can be referred to as a discrete coding process, which is quantization.

[0124] exist Figure 2 In the VQGAN shown in the figure, the intermediate feature vector Further discretize the code into a discrete code vector After that, the discrete coding vector Input CNN decoder, then CNN decoder Decode and obtain the generated image (i.e. reconstructed image) corresponding to the input image Among them, the above , and The dimensions of a single encoding vector in are ,For example, Can be , wait.

[0125] S103: For each second sample image, calculate a second loss value based on the reconstructed image and the second sample image, and adjust model parameters of the discrete vector generation model based on the second loss value.

[0126] As mentioned above, in this specific implementation, continuous image data is represented by discretized vectors, and the discretized vectors are further encoded into continuous image data. Therefore, for each second sample image, the second sample image and the reconstructed image corresponding to the second sample image should be the same image. Under ideal conditions, the second sample image and the reconstructed image corresponding to the second sample image should be exactly the same.

[0127] Based on this, in specific applications, it is often hoped that the second sample image and the reconstructed image corresponding to the second sample image are as similar as possible, and the smaller the difference between the two, the better. Therefore, in this specific implementation, in order to make the discrete vector generation model gradually converge to complete the training of the discrete vector generation model, for each second sample image, after obtaining the reconstructed image corresponding to the second sample image, the second loss value can be calculated based on the generated image and the second sample image, that is, the function value of the preset loss function of the discrete vector generation model is calculated. In this way, the model parameters of the above discrete vector generation model can be adjusted based on the second loss value.

[0128] That is, when it is determined based on the second loss value that the discrete vector generation model has not converged, the model parameters of the discrete vector generation model are adjusted, and for the next sample image, the step S101 is returned to again to continuously train the discrete vector generation model. Exemplarily, the model parameters of the discrete vector generation model can be updated after multiple iterations based on the back propagation algorithm to optimize the discrete vector generation model.

[0129] It should be noted that if the calculated second loss value does not exceed the preset loss threshold, it can be considered that the quantization effect for the second sample image is ideal, that is, the quantization effect of the discrete vector generation model meets the requirements, and the trained discrete vector generation model is obtained.

[0130] Optional, with Figure 2 As an example, the VQGAN shown in the figure discretizes the continuous data represented in the form of a sequence into a discrete encoding vector. Therefore, it is hoped that , so we can set the adversarial loss CNN discriminator, then for each second sample image, after obtaining the reconstructed image corresponding to the second sample image (i.e. Figure 2 After the generated image in the figure), the adversarial loss of the second sample image and the reconstructed image corresponding to the second sample image can be calculated. , as the second loss value mentioned above. Among them, the adversarial loss As shown below:

[0131] .

[0132] Among them, the above-mentioned confrontation loss In the calculation formula, represents the encoder in the discrete vector generation model, represents the generator in the discrete vector generation model, is the preset discrete vector codebook, represents the discriminator in the discrete vector generation model, It is a judgment result obtained by the discriminator in the discrete vector generation model as to whether the reconstructed image corresponding to the second sample image is a real image.

[0133] Among them, the above CNN discriminator can be based on the calculated adversarial loss , determines whether the reconstructed image corresponding to the second sample image is a rainbow pattern image, and outputs a determination result of Real (true) or Fake (false). When the reconstructed image is determined to be a rainbow pattern image, the CNN discriminator outputs Real, and correspondingly, when the reconstructed image is determined not to be a rainbow pattern image, the CNN discriminator outputs Fake.

[0134] Optionally, due to the discretization operation, the reconstructed image corresponding to the second sample image The backpropagation of non-differentiable operations in is implemented by a straight-through gradient estimator, which simply copies the gradients of the CNN decoder to the CNN encoder. , so that VQGAN can be trained end-to-end through the loss function. Based on this, the second loss value can be: codebook gradient update self-supervised loss .

[0135] + .

[0136] in, To rebuild the losses; For shrinking CNN encoders The vector distance between the output intermediate feature vector and the discrete vector in the preset discrete vector codebook; It is a gradient termination operation, and its purpose is to ensure that the neural network gradient can be back-propagated normally and is not affected by discrete coding. Used to constrain the CNN encoder Keep consistent with the quantization codebook space, that is, used to constrain the CNN encoder The obtained intermediate feature vector has a similarity as close as possible to the discrete vector in the preset discrete vector codebook.

[0137] Optionally, in order to further improve the similarity between the input image (sample image) and the reconstructed image, the perceptual loss may be added to the second loss value. In some cases, considering that the image may contain features of a person's face, in order to improve the fidelity of the person's face, when preprocessing the sample data, facial region detection is added and the detection results are saved. Thus, in the training stage of the above discrete vector generation model, the facial feature similarity loss can be increased. , then the loss function of VQGAN includes: adversarial loss, codebook gradient update self-supervision loss, perceptual loss and facial feature similarity loss, so the above second loss value is as follows, where, is the calculated second loss value.

[0138] + + .

[0139] As mentioned above, Figure 2 Taking the VQGAN shown in FIG. 1 as an example, after 800,000 iterations, the discrete coding vector generation model converged and a trained discrete coding vector generation model was obtained.

[0140] After obtaining the above-mentioned trained discrete coding vector generation model, the above-mentioned trained discrete coding vector generation model can be used to quantize the pre-acquired clean images, for example, the clean images serving as the second sample images, etc., so as to obtain the discrete coding vectors of the clean images generated by the above-mentioned trained discrete coding vector generation model as the clean image prior codebook, and the above-mentioned trained discrete coding vector generation model can be used to quantize the pre-acquired rainbow images, for example, the rainbow images serving as the second sample images, etc., so as to obtain the discrete coding vectors of the rainbow images generated by the above-mentioned trained discrete coding vector generation model as the rainbow image prior codebook.

[0141] On the basis of obtaining the above-mentioned clean image priori codebook and rainbow pattern image priori codebook, a training method for an image generation model provided by an embodiment of the present invention is specifically described below in conjunction with the accompanying drawings.

[0142] Figure 4 A flowchart of a training method for an image generation model provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method may include the following steps S401-S404.

[0143] S401: Acquire a first sample image.

[0144] The first sample image is a clean image.

[0145] Since it is relatively easy to obtain a clean image without rainbow streaks in a specific application, in the embodiment of the present application, the trained image generation model is used to generate a rainbow streak image using a clean image, so that when training the above image generation model, a clean image can be first obtained as the first sample image. The above clean image as the first sample image can be a real clean image captured by an image acquisition device.

[0146] S402: Input the first sample image into the image generation model to be trained so that the image generation model encodes the first sample image to obtain an image feature vector, and based on the image feature vector, use a preset clean image prior codebook to predict the clean image code, and based on the image feature vector, use a preset rainbow image prior codebook to predict the rainbow image code, compare the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, perform encoding fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, and decode the target rainbow image code to obtain a rainbow image.

[0147] Among them, the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by a discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using the second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0148] After obtaining the clean image as the first sample image, the first sample image is input into the image generation model to be trained.

[0149] In this way, the image generation model encodes the first sample image to obtain an image feature vector. The image generation model may be provided with an encoder, so that the image generation model uses the encoder to encode the first sample image to obtain an image feature vector. Moreover, the structure of the encoder provided by the image generation model and the encoder provided by the discrete vector generation model may be the same.

[0150] Afterwards, the image generation model predicts the clean image code based on the image feature vector and the clean image prior codebook, and predicts the rainbow image code based on the image feature vector and the rainbow image prior codebook. Since the clean image prior codebook and the rainbow image prior codebook are discrete vector codebooks, and their contents are discrete vectors, the clean image code and the rainbow image code obtained by the prediction are also discrete vectors, and the clean image code and the rainbow image code, as well as the discrete vectors in the clean image prior codebook and the rainbow image prior codebook have the same vector dimensions.

[0151] Optionally, the above-mentioned image generation model can adopt a nearest neighbor search method to retrieve the discrete vector closest to each coding position in the above-mentioned image feature vector in the above-mentioned clean image prior codebook, thereby generating a clean image code using the retrieved discrete vector, and for each coding position in the above-mentioned image feature vector, retrieve the discrete vector closest to the coding position in the above-mentioned rainbow image prior codebook, thereby generating a rainbow image code using the retrieved discrete vector.

[0152] Optionally, in order to further reduce the coding deviation caused by the nearest neighbor search method and improve the accuracy of the predicted clean image coding and rainbow image coding, Transforemer (time series model) can be used to perform the above-mentioned image coding prediction. Since Transforemer can learn the contextual relationship of the data sequence, when using Transforemer for image coding, it is possible to start from the global perspective of the above-mentioned image feature vector and consider the relationship between the various coding positions of the image feature vector, thereby obtaining better image coding prediction results.

[0153] Based on this, optionally, in a specific implementation, the above-mentioned image generation model includes a Transforemer module.

[0154] Correspondingly, in this specific implementation, in the above step S402, the image generation model predicts the clean image coding based on the image feature vector using the preset clean image prior codebook, and predicts the rainbow image coding based on the image feature vector using the preset rainbow image prior codebook, which may include the following steps 1-3.

[0155] Step 1: Use the Transforemer module to calculate the features of the image feature vector and obtain the encoding sequence corresponding to the image feature vector.

[0156] In this specific implementation, when the image generation model obtains the above image feature vector, the above image feature vector is expanded into a discrete vector of a specified dimension; then, the above image feature vector is input into the Transforemer module, so that the Transforemer module performs feature calculation on the above image feature vector to obtain the coding sequence corresponding to the above image feature vector. The number of the obtained coding sequences corresponds to the specified dimension of the image feature vector. For example, the specified dimension of the image feature vector is , where a and b are both positive integers, the number of obtained coding sequences is the product of a and b.

[0157] Optionally, the Transforemer module may include: multiple self-attention submodules. Thus, the step 1 may include the following steps 11 and 12.

[0158] Step 11: Based on the image feature vector, use multiple self-attention sub-modules in turn to perform specified calculations to obtain the calculation result output by the last self-attention module.

[0159] Step 12: Use a linear layer to project the calculation result to a specific dimension that matches the dimension of the image feature vector to obtain a coding sequence corresponding to the image feature vector; the specific dimension is: , express and The dimension of the image feature vector is .

[0160] Among them, when the above-mentioned image feature vector is input into the Transforemer module, the above-mentioned image feature vector can be directly input into the first self-attention submodule among the above-mentioned multiple self-attention submodules, and the first self-attention submodule performs a specified calculation on the above-mentioned image feature vector to obtain a calculation result, and inputs the calculation result into the second self-attention submodule, and the second self-attention submodule performs a specified calculation on the obtained input result, obtains a calculation result again, and inputs the calculation result into the third self-attention submodule. In this way, by analogy, the above-mentioned Transforemer module can perform specified calculations based on the above-mentioned image feature vector, and sequentially use multiple self-attention submodules to perform specified calculations, thereby obtaining the calculation result output by the last self-attention module.

[0161] That is to say, the above-mentioned Transforemer module is composed of a plurality of self-attention sub-modules stacked together, and the above-mentioned Transforemer module has adjustable parameters to adjust the number of self-attention sub-modules included therein. For example, the above-mentioned Transforemer module may include 9 self-attention sub-modules.

[0162] Optionally, the manner in which each self-attention submodule performs a specified calculation based on the image feature vector may include the following step 111.

[0163] Step 111: Each self-attention submodule performs a specified calculation based on the image feature vector using the first formula; wherein the first formula is:

[0164] ;

[0165] in, is the image feature vector, is the calculation result of the s-th self-attention module, is the calculation result of the s+1th self-attention module, 0≤s≤M-1, M is the number of multiple self-attention sub-modules; for Query value, for Key value, for The Value value.

[0166] The calculation result of the first self-attention submodule is the above-mentioned feature vector, and starting from the second self-attention submodule, each self-attention submodule first calculates the Query (query, Q) value, Key (key, K) value and (value, V) Value value of the obtained calculation result when obtaining the calculation result of the previous self-attention submodule; wherein, the above-mentioned Query value, Key value and Value value can be calculated through the linear layer included in the Transforemer module. Afterwards, the calculated Query value, Key value and Value value, as well as the obtained calculation result, are used to Operations are performed to obtain the output results, and finally the calculation results output by the last self-attention module are obtained.

[0167] Optionally, in the above step 111, in order to improve the prediction ability of the Transforemer module, sinusoidal position coding may be added to the above calculated Query value and Key value to improve the expression ability of the Transforemer module for discrete vector coding of the above image feature vector.

[0168] After obtaining the calculation result of the last self-attention module output, the Transforemer module can use the linear layer to project the above calculation result to Dimension, thus, we get coding sequences, where each coding sequence is represented in the form of a probability of N coding vectors.

[0169] Step 2: For each coding sequence, retrieve the clean window code item with the highest similarity to the coding sequence in the preset clean image prior codebook, and use the retrieved clean window code item to predict the coding sequence to obtain the clean image code.

[0170] Step 3: For each coding sequence, retrieve the rainbow window code item with the highest similarity to the coding sequence in the preset rainbow image prior codebook, and use the retrieved rainbow window code item to predict the coding sequence to obtain the rainbow image code.

[0171] After obtaining the coding sequence corresponding to the above-mentioned image feature vector, for each coding sequence, the image generation model retrieves the clean image code item with the highest similarity to the coding sequence in the preset clean image prior codebook, and uses the retrieved clean image code item to predict the coding sequence to obtain the clean image coding corresponding to the clean image; correspondingly, for each coding sequence, the image generation model retrieves the rainbow image code item with the highest similarity to the coding sequence in the preset rainbow image prior codebook, and uses the retrieved rainbow image code item to predict the coding sequence to obtain the rainbow image coding corresponding to the clean image.

[0172] Among them, the so-called prediction of the coding sequence using the clean image code item means: replacing the coding sequence with the clean image code item; or, fine-tuning the coding sequence with the clean image code item to make it closer to the retrieved clean image code item; correspondingly, the so-called prediction of the coding sequence using the rainbow pattern image code item means: replacing the coding sequence with the rainbow pattern image code item; or, fine-tuning the coding sequence with the rainbow pattern image code item to make it closer to the retrieved rainbow pattern image code item.

[0173] After obtaining the above clean image code and the rainbow image code, the image generation model can further compare the above clean image code and the above rainbow image code to obtain the background layer code and the rainbow layer code of the rainbow image to be generated, and then the above background layer code and the rainbow layer code can be fused to obtain the target rainbow image code. The above process of obtaining the target rainbow image code can be called comparative learning of the clean image code and the rainbow image code.

[0174] Among them, in the embodiment of the present application, the purpose of training the above-mentioned image generation model is to generate a rainbow image using a clean image through the image generation model. That is to say, what the above-mentioned image generation model realizes is to generate a rainbow layer, and to generate a rainbow image with a clean image as the background layer and superimposed with the generated rainbow layer. The above-mentioned clean image code and the rainbow image code are generated for the same first sample image, so the above-mentioned rainbow image code can be understood as an image code obtained by superimposing the rainbow layer on the basis of the above-mentioned clean image code as the background layer. In this way, for the above-mentioned rainbow image code and the clean image code, there are similar or even identical parts and different parts, and the above-mentioned similar or even identical parts can be understood as the code of the image area belonging to the background layer, and the different parts can be understood as the code of the image area superimposed with rainbow stripes. Therefore, by comparing the clean image code and the rainbow image code, the similar parts in the above-mentioned rainbow image code and the clean image code can be determined as the background layer code of the rainbow image to be generated, and the different parts in the above-mentioned rainbow image code and the clean image code can be determined as the rainbow layer code of the rainbow image to be generated. Afterwards, the rainbow layer code is superimposed on the background layer code, that is, the rainbow layer code and the background layer code are coded and fused to obtain the target rainbow image code.

[0175] Therefore, the process of generating rainbow patterns by the trained image generation model can be understood as: comparing the clean image code and the rainbow image code obtained above, thereby using the code that meets the similarity requirements between the clean image code and the rainbow image code as the background layer code of the rainbow image to be generated, and using the code that does not meet the similarity requirements between the clean image code and the rainbow image code as the rainbow layer code of the rainbow image to be generated, and then, superimposing the rainbow layer code on the background layer code, and obtaining the final rainbow image to be generated through code fusion.

[0176] Based on this, the above-mentioned image generation model can compare the above-mentioned clean image coding and rainbow pattern image coding to obtain background layer coding and rainbow pattern layer coding, and encode and fuse the above-mentioned background layer coding and rainbow pattern layer coding to obtain the target rainbow pattern image coding.

[0177] Optionally, in a specific implementation, the above-mentioned image generation model compares the clean image coding and the rainbow image coding to obtain the background layer coding and the rainbow layer coding, and performs coding fusion on the background layer coding and the rainbow layer coding to obtain the target rainbow image coding, which may include the following step 4.

[0178] Step 4: Using the second formula, calculate the target rainbow pattern image code that meets the optimization target;

[0179] Among them, the second formula is:

[0180] ;

[0181] in, Encode the target rainbow pattern image, The first A coding sequence, It is the first A coding sequence, For the clean image encoding A coding sequence, It is the first A coding sequence, is a preset parameter, and N is a preset codebook dimension;

[0182] The optimization goal is:

[0183] + ;

[0184] in, is the preset loss value, is the self-consistent data fidelity and , is the clean image prior codebook, is the rainbow pattern image prior codebook, for The weight coefficient of for The weight coefficient of .

[0185] It can be seen from the above that the obtained target rainbow image coding satisfies the optimization target, which means that the obtained target rainbow image coding minimizes the preset loss value in the optimization target.

[0186] It can be understood that, given a rainbow image O, the rainbow image can be decomposed into a clean background layer B and a rainbow layer R. The purpose to be achieved by the embodiment of the present application can be understood as the degradation process of the above decomposition process, and the degradation process can be defined as: O = B + R. Therefore, the above optimization target can be defined.

[0187] Among them, the so-called self-consistent data fidelity refers to: the consistency and non-contradiction of the data in internal logic, that is, the data itself can form an internally logically consistent and non-contradictory whole, and data self-consistency can improve the degree of assurance.

[0188] Based on this, through the above-mentioned second formula and optimization target, the different parts in the above-mentioned clean image coding and rainbow image coding can be further separated, and the similar parts can be clustered more tightly, so that the fine-grained compact differences can be further captured, and the distinguishing representation between the rainbow layer in the above-mentioned rainbow image coding and the above-mentioned clean image coding can be improved, thereby improving the accuracy of the obtained target rainbow image coding.

[0189] After obtaining the above-mentioned target rainbow image code, the image generation model can decode the above-mentioned target rainbow image code to obtain a rainbow image; wherein the above-mentioned image generation model can be provided with a decoder, so that the image generation model uses the above-mentioned decoder to decode the target rainbow image code to obtain a rainbow image.

[0190] S403: Calculate a first loss value based on the first sample image and the rainbow pattern image.

[0191] S404: Adjust model parameters of the image generation model based on the first loss value.

[0192] As mentioned above, in an embodiment of the present application, the image generation model generates a rainbow image using a clean image. Therefore, for the clean image to be trained, the background layer of the clean image and the generated rainbow image is still the same image. Under ideal conditions, the background layer of the clean image and the generated rainbow image should be exactly the same.

[0193] Based on this, in specific applications, it is often hoped that the background layer of the generated rainbow pattern image is as similar as possible to the clean image, and the smaller the difference between the two, the better. Therefore, in order to make the image generation model gradually converge to complete the training of the image generation model, for each first sample image, after obtaining the rainbow pattern image, the first loss value can be calculated based on the first sample image and the rainbow pattern image, that is, the function value of the preset loss function of the image generation model is calculated. In this way, the model parameters of the above-mentioned image generation model can be adjusted based on the first loss value.

[0194] That is, when it is determined based on the first loss value that the image generation model has not converged, the model parameters of the image generation model are adjusted, and for the next clean image as the first sample image, the step S501 is returned to again to continuously train the image generation model. Exemplarily, the model parameters of the image generation model can be updated after multiple iterations based on the back propagation algorithm to optimize the image generation model.

[0195] It should be noted that if the calculated first loss value does not exceed the preset loss threshold, it can be considered that the fidelity of the rainbow image obtained for the clean image serving as the first sample image is ideal, that is, the accuracy of the rainbow image generated by the image generation model meets the requirements, and the trained image generation model is obtained at this time.

[0196] Optionally, in order to constrain the consistency between the first sample image and the background layer of the generated rainbow pattern image, the first loss value may include: reconstruction loss , based on the perceptual loss of VGG (Visual Geometry Group Network) 16 Similarity loss to face In this way, by using the reconstruction loss and the perceptual loss, the model parameters of the image generation model are adjusted, so that the background layer of the rainbow pattern image obtained by the trained image generation model has a high similarity with the clean image used to generate the rainbow pattern image; and by using the facial similarity loss, the model parameters of the image generation model are adjusted, so that the fidelity of the facial image in the rainbow pattern image obtained by the trained image generation model can be improved.

[0197] In addition, in the embodiment of the present application, the purpose of using the image generation model to generate a rainbow image is to use the generated rainbow image to train an image processing model for removing rainbow patterns in a real rainbow image. Therefore, in the embodiment of the present application, it is also hoped that the distribution of rainbow patterns in the rainbow image generated by the image generation model is highly consistent with the distribution of rainbow patterns in the real rainbow image captured by the image acquisition device.

[0198] Based on this, optionally, in the training stage of the above-mentioned discrete vector generation model, when the CNN discriminator is used to calculate the above-mentioned second loss value, in the training stage of the above-mentioned image generation model, the same CNN discriminator can also be used to calculate the above-mentioned first loss value. At this time, the above-mentioned first loss value is used to constrain the distribution of rainbow patterns in the rainbow pattern image generated by the image generation model to be consistent with the distribution of rainbow patterns in the real rainbow pattern image captured by the image acquisition device.

[0199] As mentioned above, optionally, the first loss value can simultaneously constrain the consistency between the clean image to be trained and the background layer of the generated rainbow pattern image, and the consistency between the distribution of rainbow patterns in the rainbow pattern image generated by the image generation model and the distribution of rainbow patterns in the real rainbow pattern image collected by the image acquisition device. Therefore, the first loss value can include: the reconstruction loss , Perceptual Loss , face similarity loss And the loss value calculated by the above CNN discriminator. Then the above first loss value is as follows, where, is the calculated first loss value, , , , are the coefficients for adjusting the proportion of loss, Loss value calculated for the above CNN discriminator.

[0200] ;

[0201] Among them, when calculating the above-mentioned first loss value, the loss value calculated by the above-mentioned CNN discriminator is the same as the target rainbow image code calculated by the above-mentioned second formula. Usually, the target rainbow image code calculated by the above-mentioned second formula can be substituted into the above-mentioned first loss value calculation formula as the loss value calculated by the above-mentioned CNN discriminator to calculate the above-mentioned first loss value.

[0202] Based on this, when the above-mentioned image generation model converges, a trained image generation model can be obtained, and the trained image generation model can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity. In this way, by applying the scheme provided by the embodiment of the present application, an image generation model can be trained based on an unsupervised training method to achieve the degeneration of the clean image to a rainbow pattern image with rainbow patterns, and the difference between the rainbow pattern distribution in the degraded rainbow pattern image and the rainbow pattern distribution of the real rainbow pattern image is small, and at the same time, the background layer of the generated rainbow pattern image has a high fidelity with the clean image, that is, the trained image generation model can be used to generate a rainbow pattern image with high fidelity using a clean image.

[0203] In addition, compared with the related technology of rainbow pattern processing, which separates the rainbow pattern layer in the rainbow pattern image by manually adjusting parameters and extracting individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image. Therefore, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image can be performed without being limited to the display of the rainbow pattern shape, rainbow pattern color, etc. Moreover, in the embodiment of the present application, when training the above-mentioned image generation model, it does not rely on the paired data of the clean image and the rainbow pattern image for image model training, so that the trained image generation model can have higher generalization, and the use of discrete vector coding compression technology can reduce the computing resource consumption of the image generation model, so that the image generation model can be trained on a large number of data sets, increase the number of model iterations and model iteration efficiency, and improve the reasoning effect of the model.

[0204] Below, Figure 5 Taking the structural diagram of the image generation model shown in the figure and the car window image recognition scenario as examples, the training method of the above image generation model is illustrated.

[0205] Among them, Figure 5 As shown in the figure, the image generation model adopts an unsupervised training method, and its network architecture is an adversarial network architecture. Figure 5 The encoder E, decoder D and CNN discriminator in Figure 2 The CNN encoder in the discrete vector generation model shown , the structures of CNN decoder and CNN discriminator are the same. Figure 5 In the proposed method, the Transformer module can more accurately predict the clean image coding and the rainbow image coding, and based on contrastive learning, the predicted clean image coding and the rainbow image coding can be accurately separated to achieve the goals of intra-class cohesion and inter-class separation.

[0206] like Figure 5 As shown, a clear car window X (i.e., a clean image to be trained) is obtained. After the clear car window X is input into the image generation model to be trained, the clear car window X can be encoded by the encoder E to obtain the image feature vector .in, and is the codebook dimension of the preset clean image prior codebook and rainbow pattern image prior codebook, is the vector dimension of the image feature vector, the encoder E can be Figure 2 CNN encoder in After fine-tuning, it can be expressed as .

[0207] In the image generation model to be trained, a Transformer module containing nine self-attention submodules is set after the encoder E. Then, when the image feature vector When the image feature vector The expanded dimension is The discrete vector , then Input the above Transformer module. The s-th self-attention submodule in the Transformer module is calculated according to the first formula above.

[0208] In order to improve the prediction ability of the Transforemer module, sinusoidal position coding can be added to the Query value and Key value calculated above. , in order to improve the ability of the Transforemer module to express the above feature vectors in discrete vector encoding. After obtaining the calculation results output by the last self-attention submodule, a linear layer is used to project the above calculation results to That is, the eigenvector As the input of the above Transformer module, Transformer takes The probability form of the prediction of the code items A code sequence.

[0209] Then, for the above For each code sequence in the code sequences, the rainbow pattern window code item with the highest similarity to each code sequence is retrieved from the rainbow pattern window prior knowledge code book (i.e., the rainbow pattern image prior code book), and each code sequence is predicted using the retrieved rainbow pattern window code item to obtain the rainbow pattern window discrete coding vector (i.e., rainbow pattern image coding) of the clear window X. , for the above For each code sequence in the code sequences, retrieve the clean window code item with the highest similarity to each code sequence from the clear window prior knowledge code book (i.e., clean image prior code book), and use the retrieved clean window code item to predict each code sequence to obtain the clear window discrete code vector (i.e., clean image code) of the clear window X. .

[0210] Then, the above second formula and optimization target are used to calculate the rainbow pattern window discrete coding vector of the clear window X: , and the clear window discrete encoding vector of the clear window X Perform contrastive learning, and fuse the background layer coding and rainbow pattern layer coding obtained by contrastive learning to obtain a new rainbow pattern window discrete coding vector (i.e., target rainbow pattern image coding) , and use the decoder D to discretize the above new rainbow pattern window encoding vector Decode and get the window image with rainbow pattern, that is, Figure 5 The generated image in is used as the target rainbow pattern image.

[0211] After that, we can use the CNN discriminator based on Figure 5 The clear car window X and the generated image in the image are used to calculate the first loss value, and the first loss value is used to adjust Figure 5 The model parameters of the image generation model shown in . Among them, Figure 5 The CNN discriminator in Figure 2 The CNN discriminator in is the same discriminator. Figure 5 The image generation model shown in the figure has obviously converged after 400,000 iterations of training, and a trained image generation model has been obtained.

[0212] Taking the clean image shown in FIG3 (a) as an example, after the image generation model is trained, the clean image shown in FIG3 (a) can be input into the trained image generation model to generate a rainbow pattern image. In particular, since the first sample image and / or the second sample image used in training the image generation model are different, an image generation model that can generate rainbow pattern images with rainbow patterns of different styles can be obtained. Therefore, as shown in FIG3 (b) and FIG3 (c), rainbow pattern images with rainbow patterns of different styles are generated using different trained image generation models. FIG3 (d) is an image obtained by removing the rainbow pattern from the rainbow pattern image shown in FIG3 (c).

[0213] Based on this, in the embodiment of the present application, the use of Transformer is effective and efficient in accurately generating the global correlation of the rainbow image encoding, and the use of VQGAN-based prior codebook construction and contrast-based learning is effective in separating the rainbow layer and background layer of the rainbow image encoding. Therefore, the unsupervised method provided in the embodiment of the present application has a good effect in generating the authenticity and background fidelity of the rainbow image, and it is feasible to construct paired data for the generated rainbow image and the input clean image, thereby providing data support for the subsequent removal of rainbow in the rainbow image.

[0214] During the training process of the above-mentioned image generation model, a classification model for classifying clean images and rainbow pattern images can be introduced to utilize the prior knowledge of the classification of clean images and rainbow pattern images to guide the gradient descent direction of the image generation model in the early stage of training to be biased towards the rainbow pattern image, thereby accelerating the convergence speed of the image generation model.

[0215] Based on this, optionally, in a specific implementation, such as Figure 6 As shown, in the above step S404, before adjusting the model parameters of the image generation model based on the first loss value, a training method for an image generation model provided in an embodiment of the present application may also include the following step S405.

[0216] S405: Input the rainbow pattern image into the image classification model so that the image classification model outputs a confidence loss value of the rainbow pattern image.

[0217] The image classification model is obtained by training the classification model to be trained based on the pre-acquired clean images and rainbow pattern images.

[0218] Accordingly, in this specific implementation, the above step S404 may include the following step S4041.

[0219] S4041: Adjust model parameters of the image generation model based on the first loss value and the confidence loss value.

[0220] In this specific implementation, the sample clean images and sample rainbow images acquired in advance can be used to train the classification model to be trained, and a trained image classification model can be obtained. In some cases, in the rainbow image generated by the above-mentioned image generation model, only a small part of the image area has rainbow patterns. Therefore, when using the above-mentioned discriminator to calculate the above-mentioned first loss value, it is easy to mistake the above-mentioned rainbow image for a clean image, which affects the reliability of the calculated first loss value, thereby affecting the reasoning effect of the trained image generation model. Based on this, an image classification model can be introduced, and the confidence loss value obtained by the above-mentioned first loss value and the image classification model can be used to jointly adjust the model parameters of the image generation model, so as to guide the image generation model to be biased towards the rainbow image in the direction of gradient descent at the initial stage of training, accelerate the convergence speed of the image generation model, and improve the reasoning effect of the image generation model.

[0221] In this way, during the training process of the above-mentioned image generation model, after obtaining the rainbow pattern image generated by the above-mentioned image generation model, the above-mentioned rainbow pattern image can be input into the above-mentioned image classification model, so that the above-mentioned image classification model performs classification calculation on the above-mentioned rainbow pattern image to determine whether the rainbow pattern image generated by the above-mentioned image classification model is really the confidence of the rainbow pattern image, thereby obtaining the confidence loss value output by the above-mentioned image classification model. Then, the model parameters of the image generation model can be adjusted based on the first loss value and the confidence loss value.

[0222] Optionally, the model parameters of the image generation model may be adjusted based on the sum of the first loss value and the confidence loss value. For example, when it is determined based on the sum of the first loss value and the confidence loss value that the image generation model has not converged, the model parameters of the image generation model are adjusted.

[0223] Optionally, the mobilenetv3-small network framework can be used as the backbone network of the classification model. Its lightweight structure can quickly complete reasoning. In order to improve the accuracy of classification, the data enhancement method mainly uses saturation, brightness, blur, inversion and noise addition to improve the model accuracy and generalization performance. To address the problem of low prediction accuracy of difficult samples, the Focal loss method is used to add a coefficient factor on the basis of the standard cross entropy loss, thereby weakening the learning of easy samples and strengthening the learning of difficult samples, thereby improving the classification ability of the model. The formula is as follows:

[0224] ;

[0225] in, To predict how close the sample is to the label, The larger it is, the higher the classification confidence is. is an adjustable factor.

[0226] In addition to the above methods to improve accuracy, the CoordAttention (coordinate attention) attention mechanism can be introduced to replace the SE channel attention mechanism in mobilenetv3. When calculating attention, CoordAttention not only considers the input feature information, but also the position information of each pixel point to better capture the local and global relationships of image features.

[0227] Among them, through the optimization of the above method, the above image classification model was trained for 200 epochs, and the obtained image classification model achieved an accuracy of 95% on the rainbow pattern image and clean image validation set (2000 samples). Among them, the output of the above image classification model can use softmax to obtain the floating-point value of the confidence of each category, and set the category loss value of the rainbow pattern image generation model according to the confidence, that is, the above confidence loss value.

[0228] Thus, in this specific implementation, the use of discrete vectorized encoding prior knowledge, rainbow pattern image confidence guidance, and an unsupervised image generation model based on contrastive learning can generate rainbow pattern images with high background fidelity and realistic rainbow patterns without relying on paired data, providing important data support for the subsequent construction of a paired data set for rainbow pattern removal.

[0229] Based on this, taking the car window image as an example, in this specific implementation, the function of the image generation model is to generate a rainbow pattern car window image with rainbow patterns based on a clean car window image without rainbow patterns, so that the background layer of the generated rainbow pattern car window image (i.e., the scene image inside the car window) has high fidelity, and the distribution of rainbow patterns is highly similar to the distribution of rainbow patterns in the real rainbow pattern car window image. And it can meet the pairing data requirements of the generated rainbow pattern car window image and the clean car window image without rainbow patterns used to generate the rainbow pattern car window image, which is of great help in removing rainbow patterns from the rainbow pattern car window image. In this specific implementation, the training method of the above-mentioned image generation model mainly includes the following three steps:

[0230] Step 1: Generate adversarial networks based on vector quantization technology to discretize information modeling of rainbow pattern window images and clean window images. That is, based on the discrete vector generation model, discrete vector encoding is performed on the pre-acquired sample clean images and sample rainbow pattern images, and the clean image prior codebook and rainbow pattern image prior codebook are generated.

[0231] Step 2: Design a high-precision image classification model for classifying clean window images and rainbow window images, which is used to guide the image generation model to bias the gradient descent direction towards the rainbow image in the early stage of training, that is, to guide the image generation model to generate a rainbow layer when generating a rainbow image based on a clean image.

[0232] Step 3: Based on the above-mentioned clean image prior codebook and rainbow pattern image prior codebook, as well as the image classification model, an unsupervised rainbow pattern window generation network of asymmetric contrast learning is designed, that is, an unsupervised image generation model of asymmetric contrast learning is designed.

[0233] For example, Figure 7As shown in the figure, a generative adversarial network structure based on the Transformer model and CNN is designed, and the network loss function is designed using an unsupervised learning method, which can generate a rainbow window image with rainbow patterns based on a clean window image.

[0234] First, obtain the prior knowledge of positive and negative samples. Based on a large number of rainbow pattern car window (i.e., rainbow pattern car window images, as positive samples) data sets and a large number of clear car window (i.e., clear car window images, as negative samples) data sets, train VQGAN (discretized vector generation model). At the same time, based on the trained VQGAN, construct vector codebooks for positive and negative samples respectively as the generation priors of positive and negative samples, that is, obtain the rainbow pattern car window discrete coding vector codebook (i.e., rainbow pattern image prior codebook) and clear car window discrete coding vector codebook (i.e., clean image prior codebook). Among them, using VQGAN to discretize and vectorize the image can better obtain the feature compression information of positive and negative samples, and the above-mentioned rainbow pattern car window discrete coding vector codebook and clear car window discrete coding vector codebook can be used as rainbow pattern car window prior knowledge and clear car window prior knowledge respectively by the rainbow pattern car window generation model (i.e., image generation model).

[0235] Secondly, learn the rainbow pattern window classification model. Based on the CNN model, the rainbow pattern window and clear window data are classified and trained to obtain the rainbow pattern window classification probability model, which is used as the rainbow pattern window classification model (i.e., image classification model). The trained model outputs: the confidence that the input window image is a rainbow pattern window image and a clear window image. The above trained model is mainly used to guide the rainbow pattern window generation model during the training process, which can make the rainbow pattern window generation model converge better and faster. Among them, the clear window image is a clean image.

[0236] Finally, a rainbow pattern window generation model is designed, such as Figure 7 As shown in the structure of the rainbow pattern car window generation model, the rainbow pattern car window generation model mainly consists of three parts: encoder, decoder, and transformer-based codebook index prediction (i.e., Transformer prediction module). Among them, the structure of the encoder and decoder is the same as the network structure of the VQGAN model. The transformer-based codebook index prediction module uses the transformer multi-head attention mechanism to capture the dependencies between different positions in the sequence and better understand the global information. Its main function is to predict the training of the rainbow pattern car window prior knowledge and the clear car window prior knowledge, model the negative sample (i.e., clean image) input, and use it for the discrete representation of the rainbow pattern car window image, that is, Figure 7 The rainbow pattern discrete vector in helps to improve the quality and fidelity of the generated rainbow pattern window image. In addition, Figure 7 The confidence loss in is the confidence loss value in the embodiment of the present application.

[0237] Corresponding to the training method of an image generation model provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides an image generation method.

[0238] Figure 8 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application, such as Figure 8 As shown, the image generating method may include the following steps S801-S803.

[0239] S801: Acquire an initial image.

[0240] Among them, the initial image is a clean image.

[0241] S802: Input the initial image into a preset image generation model so that the image generation model encodes the initial image to obtain an initial image feature vector, predict the initial clean image code based on the initial image feature vector using a preset clean image prior codebook, and predict the initial rainbow image code based on the initial image feature vector using a rainbow image prior codebook, compare the initial clean image code with the initial rainbow image code to obtain an initial background layer code and an initial rainbow layer code, perform encoding fusion on the initial background layer code and the initial rainbow layer code, determine the rainbow image code to be generated, decode the rainbow image code to be generated, and obtain a target rainbow image.

[0242] S803: Obtain the target rainbow pattern image output by the image generation model.

[0243] Among them, the image generation model is: a model trained based on the training method of any image generation model provided in the above-mentioned embodiments of the present application.

[0244] Based on this, the image generation model trained by the training method of any image generation model provided in the embodiments of the present application can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity.

[0245] In addition, compared to the related technology of rainbow pattern processing, in which the rainbow pattern layers in the rainbow pattern image are separated by manual parameter adjustment to extract individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image, that is, in the embodiment of the present application, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image is not limited to the display of rainbow pattern shape, rainbow pattern color, etc.

[0246] Corresponding to the training method of an image generation model provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides an image generation method.

[0247] Fig. 9 A schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application, such as Fig. 9 As shown, the image generating method may include the following steps S901-S903.

[0248] S901: Acquire an initial image.

[0249] Among them, the initial image is a clean image.

[0250] S902: Inputting the initial image into the trained image generation model, so that the image generation model generates a target rainbow image with the initial image as the background layer based on the initial image, a preset clean image prior codebook and a preset rainbow image prior codebook.

[0251] S903: Obtain the target rainbow pattern image output by the image generation model.

[0252] Among them, the image generation model is obtained by training a preset image generation model based on a first sample image, a clean image prior codebook and a rainbow pattern image prior codebook, the first sample image is a clean image; the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by a discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using the second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0253] Optional, in the above Fig. 9 In the image generation method shown, when the preset image generation model is trained based on the first sample image, the clean image prior codebook and the rainbow pattern image prior codebook to obtain the trained image generation model, any image generation model training method provided in the above embodiment of the present application can be used. Fig. 9In the image generation method shown, the image generation model can be a model trained based on any image generation model training method provided in the above-mentioned embodiments of the present application.

[0254] Based on this, the image generation model trained by the training method of any image generation model provided in the embodiments of the present application can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity.

[0255] In addition, compared to the related technology of rainbow pattern processing, in which the rainbow pattern layers in the rainbow pattern image are separated by manual parameter adjustment to extract individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image, that is, in the embodiment of the present application, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image is not limited to the display of rainbow pattern shape, rainbow pattern color, etc.

[0256] Corresponding to the training method of an image generation model provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides a training device for an image generation model.

[0257] Fig.10 A structural diagram of a training device for an image generation model provided in an embodiment of the present application, such as Fig.10 As shown, the device comprises:

[0258] The sample image acquisition module 1001 is used to acquire a first sample image; wherein the sample image is a clean image;

[0259] A first input module 1002 is used to input the first sample image into the image generation model to be trained, so that the image generation model encodes the first sample image to obtain an image feature vector, predicts a clean image code based on the image feature vector using a preset clean image priori codebook, and predicts a rainbow image code based on the image feature vector using a preset rainbow image priori codebook, compares the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, performs code fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, and decodes the target rainbow image code to obtain a rainbow image;

[0260] A first calculation module 1003, configured to calculate a first loss value based on the first sample image and the rainbow pattern image, and adjust a model parameter of the image generation model based on the first loss value;

[0261] Among them, the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0262] Based on this, when the above-mentioned image generation model converges, a trained image generation model can be obtained, and the trained image generation model can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity. In this way, by applying the scheme provided by the embodiment of the present application, an image generation model can be trained based on an unsupervised training method to achieve the degeneration of the clean image to a rainbow pattern image with rainbow patterns, and the difference between the rainbow pattern distribution in the degraded rainbow pattern image and the rainbow pattern distribution of the real rainbow pattern image is small, and at the same time, the background layer of the generated rainbow pattern image has a high fidelity with the clean image, that is, the trained image generation model can be used to generate a rainbow pattern image with high fidelity using a clean image.

[0263] In addition, compared with the related technology of rainbow pattern processing, which separates the rainbow pattern layer in the rainbow pattern image by manually adjusting parameters and extracting individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image. Therefore, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image can be performed without being limited to the display of the rainbow pattern shape, rainbow pattern color, etc. Moreover, in the embodiment of the present application, when training the above-mentioned image generation model, it does not rely on the paired data of the clean image and the rainbow pattern image for image model training, so that the trained image generation model can have higher generalization, and the use of discrete vector coding compression technology can reduce the computing resource consumption of the image generation model, so that the image generation model can be trained on a large number of data sets, increase the number of model iterations and model iteration efficiency, and improve the reasoning effect of the model.

[0264] Corresponding to the image generating method provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides an image generating device.

[0265] Fig.11 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the device comprises:

[0266] A first initial image acquisition module 1101 is used to acquire an initial image; wherein the initial image is a clean image;

[0267] The second input module 1102 is used to input the initial image into a preset image generation model so that the image generation model encodes the initial image to obtain an initial image feature vector, predict an initial clean image code based on the initial image feature vector using a preset clean image priori codebook, and predict an initial rainbow image code based on the initial image feature vector using a rainbow image priori codebook, compare the initial clean image code with the initial rainbow image code to obtain an initial background layer code and an initial rainbow layer code, perform code fusion on the initial background layer code and the initial rainbow layer code, determine a rainbow image code to be generated, and decode the rainbow image code to be generated to obtain a target rainbow image;

[0268] A first target image acquisition module 1103 is used to acquire the target rainbow pattern image output by the image generation model;

[0269] Among them, the image generation model is: a model trained based on the training method of any image generation model provided in the above-mentioned embodiments of the present application.

[0270] Based on this, the image generation model trained by the training method of any image generation model provided in the embodiments of the present application can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity.

[0271] In addition, compared to the related technology of rainbow pattern processing, in which the rainbow pattern layers in the rainbow pattern image are separated by manual parameter adjustment to extract individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image, that is, in the embodiment of the present application, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image is not limited to the display of rainbow pattern shape, rainbow pattern color, etc.

[0272] Corresponding to the image generating method provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides an image generating device.

[0273] Fig.12 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the device comprises:

[0274] The second initial image acquisition module 1201 is used to acquire an initial image; wherein the initial image is a clean image;

[0275] The third input module 1202 is used to input the initial image into the trained image generation model, so that the image generation model generates a target rainbow pattern image with the initial image as the background layer based on the initial image, a preset clean image priori codebook and a preset rainbow pattern image priori codebook;

[0276] A second target image acquisition module 1203, used to acquire the target rainbow pattern image output by the image generation model;

[0277] Among them, the image generation model is obtained by training a preset image generation model based on a first sample image, the clean image prior codebook and the rainbow pattern image prior codebook, and the first sample image is a clean image; the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

[0278] Based on this, the image generation model trained by the training method of any image generation model provided in the embodiments of the present application can generate a rainbow pattern image with the clean image as the background layer based on the clean image. Among them, since the training of the above-mentioned image generation model is based on the prior knowledge of the clean image and the rainbow pattern image (the clean image prior codebook and the rainbow pattern image prior codebook), when the above-mentioned image generation model is used to generate the rainbow pattern image, the above-mentioned prior knowledge will still be used, so that the rainbow pattern layer of the generated rainbow pattern image can have high fidelity.

[0279] In addition, compared to the related technology of rainbow pattern processing, in which the rainbow pattern layers in the rainbow pattern image are separated by manual parameter adjustment to extract individual features such as rainbow pattern shape and rainbow pattern color, in the embodiment of the present application, the above-mentioned image generation model directly uses the clean image to generate a complete rainbow pattern image, that is, in the embodiment of the present application, the generation of the rainbow pattern image is end-to-end, and the generation of the rainbow pattern image is not limited to the display of rainbow pattern shape, rainbow pattern color, etc.

[0280] The present application also provides an electronic device, such as Fig.13 As shown, including:

[0281] Memory 1301, used for storing computer programs;

[0282] The processor 1302 is used to implement the steps of the training method of any image generation model provided in the above-mentioned embodiments of the present application when executing the program stored in the memory 1301, and / or implement the steps of any image generation method provided in the above-mentioned embodiments of the present application.

[0283] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1302, the communication interface, and the memory 1301 communicate with each other via the communication bus.

[0284] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0285] The communication interface is used for communication between the above electronic device and other devices.

[0286] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0287] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0288] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the training method of any image generation model provided in the above-mentioned embodiments of the present application, and / or implements the steps of any image generation method provided in the above-mentioned embodiments of the present application.

[0289] In another embodiment provided in the present application, a computer program product comprising instructions is also provided. When the computer program product is run on a computer, the computer executes the steps of the training method of any image generation model provided in the above-mentioned embodiments of the present application, and / or executes the steps of any image generation method provided in the above-mentioned embodiments of the present application.

[0290] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0291] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0292] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0293] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Acquire a first sample image; wherein the first sample image is a clean image; Inputting the first sample image into the image generation model to be trained, so that the image generation model encodes the first sample image to obtain an image feature vector, predicting a clean image code based on the image feature vector using a preset clean image priori codebook, and predicting a rainbow image code based on the image feature vector using a preset rainbow image priori codebook, comparing the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, performing encoding fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, and decoding the target rainbow image code to obtain a rainbow image; Calculating a first loss value based on the first sample image and the rainbow pattern image, and adjusting a model parameter of the image generation model based on the first loss value; Among them, the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

2. The method according to claim 1, characterized in that The image generation model includes a Transforemer module; The image generation model predicts clean image coding based on the image feature vector using a preset clean image prior codebook, and predicts rainbow image coding based on the image feature vector using a preset rainbow image prior codebook, including: Using the Transforemer module to perform feature calculation on the image feature vector to obtain a coding sequence corresponding to the image feature vector; For each coding sequence, in a preset clean image priori codebook, a clean window code item with the highest similarity to the coding sequence is retrieved, and the coding sequence is predicted using the retrieved clean window code item to obtain a clean image code; For each coding sequence, the rainbow window code item with the highest similarity to the coding sequence is retrieved in the preset rainbow image priori codebook, and the coding sequence is predicted using the retrieved rainbow window code item to obtain the rainbow image code.

3. The method according to claim 2, characterized in that The Transforemer module includes: a plurality of self-attention submodules; The using the Transforemer module to perform feature calculation on the image feature vector to obtain a coding sequence corresponding to the image feature vector includes: Based on the image feature vector, sequentially use the multiple self-attention submodules to perform specified calculations to obtain a calculation result output by the last self-attention module; The calculation result is projected to a specific dimension matching the dimension of the image feature vector using a linear layer to obtain a coding sequence corresponding to the image feature vector; wherein the specific dimension is: , express and The dimension of the image feature vector is .

4. The method according to claim 3, characterized in that: Each self-attention submodule performs a specified calculation based on the image feature vector, including: Each self-attention submodule performs a specified calculation based on the image feature vector using a first formula; wherein the first formula is: ； in, is the image feature vector, is the calculation result of the s-th self-attention module, is the calculation result of the s+1th self-attention submodule, 0≤s≤M-1, M is the number of the multiple self-attention submodules; for Query value, for Key value, for The Value value.

5. The method according to claim 1, characterized in that The image generation model compares the clean image code with the rainbow pattern image code to obtain a background layer code and a rainbow pattern layer code, and performs code fusion on the background layer code and the rainbow pattern layer code to obtain a target rainbow pattern image code, including: Using the second formula, the target rainbow pattern image code that meets the optimization target is calculated; Wherein, the second formula is: ； in, Encode the target rainbow pattern image, The first A coding sequence, is the first A coding sequence, The first A coding sequence, is the first A coding sequence, is a preset parameter, and N is a preset codebook dimension; The optimization goal is: + ； in, Encode the target rainbow pattern image, is the pre-set self-consistent data fidelity and , is the clean image prior codebook, is the rainbow image prior codebook, for The weight coefficient of for The weight coefficient of .

6. The method according to any one of claims 1 to 5, characterized in that: The training method of the discrete vector generation model includes: Acquire a second sample image; wherein the second sample image includes: a clean image and a rainbow pattern image; For each second sample image, perform the following steps: The second sample image is input into a discrete vector generation model to be trained, so that the discrete vector generation model encodes the second sample image to obtain an intermediate feature vector of the second sample image, and a discrete vector closest to each encoding position of the intermediate feature vector is retrieved from a preset discrete vector codebook, a discrete encoding vector of the second sample image is generated by using the retrieved discrete vector, and the discrete encoding vector is decoded to obtain a reconstructed image corresponding to the second sample image; wherein the preset discrete vector codebook records the discrete encoding vector of the second sample image that has been generated; A second loss value is calculated based on the reconstructed image and the second sample image, and a model parameter of the discrete vector generation model is adjusted based on the second loss value.

7. The method according to claim 1, characterized in that Before adjusting the model parameters of the image generation model based on the first loss value, the method further includes: Inputting the rainbow pattern image into an image classification model so that the image classification model outputs a confidence loss value of the rainbow pattern image; wherein the image classification model is obtained by training a classification model to be trained based on a pre-acquired sample clean image and a sample rainbow pattern image; The adjusting the model parameters of the image generation model based on the first loss value includes: Based on the first loss value and the confidence loss value, adjust the model parameters of the image generation model.

8. An image generation method, characterized in that: The method comprises: Acquire an initial image; wherein the initial image is a clean image; Input the initial image into a preset image generation model so that the image generation model encodes the initial image to obtain an initial image feature vector, predict an initial clean image code based on the initial image feature vector using a preset clean image priori codebook, and predict an initial rainbow image code based on the initial image feature vector using a preset rainbow image priori codebook, compare the initial clean image code with the initial rainbow image code to obtain an initial background layer code and an initial rainbow layer code, perform encoding fusion on the initial background layer code and the initial rainbow layer code, determine a rainbow image code to be generated, decode the rainbow image code to be generated, and obtain a target rainbow image; Acquire the target rainbow pattern image output by the image generation model; Wherein, the image generation model is: a model trained based on the training method described in any one of claims 1-7.

9. A training device for an image generation model, characterized in that: The device comprises: A sample image acquisition module, used to acquire a first sample image; wherein the first sample image is a clean image; A first input module is used to input the first sample image into the image generation model to be trained, so that the image generation model encodes the first sample image to obtain an image feature vector, predicts a clean image code based on the image feature vector using a preset clean image priori codebook, and predicts a rainbow image code based on the image feature vector using a preset rainbow image priori codebook, compares the clean image code with the rainbow image code to obtain a background layer code and a rainbow layer code, performs encoding fusion on the background layer code and the rainbow layer code to obtain a target rainbow image code, and decodes the target rainbow image code to obtain a rainbow image; A first calculation module, configured to calculate a first loss value based on the first sample image and the rainbow pattern image, and adjust a model parameter of the image generation model based on the first loss value; Among them, the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

10. An image generating device, characterized in that: The device comprises: A first initial image acquisition module, used to acquire an initial image; wherein the initial image is a clean image; A second input module is used to input the initial image into a preset image generation model so that the image generation model encodes the initial image to obtain an initial image feature vector, predict an initial clean image code based on the initial image feature vector using a preset clean image priori codebook, and predict an initial rainbow image code based on the initial image feature vector using a preset rainbow image priori codebook, compare the initial clean image code with the initial rainbow image code to obtain an initial background layer code and an initial rainbow layer code, perform code fusion on the initial background layer code and the initial rainbow layer code, determine a rainbow image code to be generated, decode the rainbow image code to be generated, and obtain a target rainbow image; A first target image acquisition module, used to acquire the target rainbow pattern image output by the image generation model; Wherein, the image generation model is: a model trained based on the training method described in any one of claims 1-7.

11. An image generation method, characterized in that: The method comprises: Acquire an initial image; wherein the initial image is a clean image; Inputting the initial image into a trained image generation model, so that the image generation model generates a target rainbow pattern image with the initial image as a background layer based on the initial image, a preset clean image priori codebook and a preset rainbow pattern image priori codebook; Acquire the target rainbow pattern image output by the image generation model; Among them, the image generation model is obtained by training a preset image generation model based on a first sample image, the clean image prior codebook and the rainbow pattern image prior codebook, and the first sample image is a clean image; the clean image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired clean image by a discrete vector generation model, and the rainbow pattern image prior codebook is a discrete vector codebook obtained by discrete vector encoding a pre-acquired rainbow pattern image by the discrete vector generation model; the discrete vector generation model is obtained by training the discrete vector generation model to be trained using a second sample image; the second sample image includes: a clean image and a rainbow pattern image.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images