Training method of image generation model and image processing method and device

Abstract

The invention discloses a training method of an image generation model, and an image processing method and device. Relates to the field of artificial intelligence, and the method comprises the steps: obtaining a training sample image set which comprises a plurality of spliced feature maps; the plurality of spliced feature maps are input into an initial student model to obtain a first prediction result output by the initial student model, the plurality of spliced feature maps are input into a teacher model to obtain a second prediction result output by the teacher model, and the initial student model is obtained by initializing parameters of the teacher model; the teacher model is a pre-trained text graph generative model; and constructing a target loss function based on the first prediction result and the second prediction result, and training parameters of the initial student model based on the target loss function to obtain a target image generation model. The technical problem of low image generation efficiency caused by high reasoning delay due to more reasoning step number and parameter quantity of an image generation model in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically, to a training method for an image generation model, an image processing method and a device. Background Art

[0002] In the process of e-commerce operations, merchants need to prepare a large number of image materials for displaying goods. Since different platforms and activities have different requirements for image material resolution and aspect ratio, merchants are often required to provide multiple sets of image materials of different proportions for the same product, which consumes a lot of labor costs. With the development of large models, it has become possible to automatically scale products based on artificial intelligence generated content (AIGC) technology. At present, the relevant technology mainly uses the diffusion generation model (Stable Diffusion XL, SDXL for short) for adaptive image content extension. However, the model has a large number of inference steps and parameters, and the deployment cost and inference delay are high, resulting in low efficiency of image generation.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a training method, an image processing method and a device for an image generation model, so as to at least solve the technical problems in the related art that the image generation model has a large number of inference steps and parameters, and has a high inference delay, resulting in relatively low efficiency of image generation.

[0005] According to one aspect of an embodiment of the present application, a method for training an image generation model is provided, comprising: obtaining a training sample image set, wherein the training sample image set includes a plurality of spliced ​​feature maps; inputting the plurality of spliced ​​feature maps into an initial student model to obtain a first prediction result output by the initial student model, inputting the plurality of spliced ​​feature maps into a teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is initialized based on parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generation model; constructing a target loss function based on the first prediction result and the second prediction result, and based on the target loss function, training the parameters of the initial student model to obtain a target image generation model.

[0006] Furthermore, multiple spliced ​​feature maps are input into the initial student model to obtain a first prediction result output by the initial student model, and multiple spliced ​​feature maps are input into the teacher model to obtain a second prediction result output by the teacher model, including: performing a single-step denoising operation on multiple spliced ​​feature maps through the initial student model to obtain a first prediction result; performing a multi-step iterative denoising operation on multiple spliced ​​feature maps through the teacher model to obtain a second prediction result.

[0007] Furthermore, the target loss function includes at least one of the following: a first loss function, a second loss function, and a third loss function. The first loss function, the second loss function, and the third loss function have different corresponding training targets. Constructing the target loss function based on the first prediction result and the second prediction result includes: constructing the first loss function based on the first prediction result and the second prediction result; constructing the second loss function based on the first prediction result and the latent space feature map of the initial sample image, wherein the latent space feature map of the initial sample image is used to generate a spliced ​​feature map; mapping the first prediction result from the latent space to the pixel space through a preset decoder to obtain a first generated image, and constructing the third loss function based on the first generated image and the initial sample image.

[0008] Furthermore, based on the target loss function, the parameters of the initial student model are trained to obtain the target image generation model, including: if the difference between the first prediction result and the second prediction result is less than the first threshold as the training target, the parameters of the initial student model are trained according to the first loss function; if the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than the second threshold as the training target, the parameters of the initial student model are trained according to the second loss function; if the distance between the features of the first generated image and the features of the initial sample image is less than the third threshold as the training target, the parameters of the initial student model are trained according to the third loss function until the preset conditions are met to obtain the target image generation model.

[0009] Furthermore, before obtaining the training sample image set, the method also includes: obtaining an initial sample image, a first sample image and a first mask image, wherein the first sample image is obtained by processing the initial sample image according to a sample mask image corresponding to the initial sample image, and the first mask image is obtained by processing the sample mask image; mapping the initial sample image from a pixel space to a latent space through a preset encoder to obtain a latent space feature map of the initial sample image, mapping the first sample image from a pixel space to a latent space through a preset encoder to obtain a latent space feature map of the first sample image; randomly adding Gaussian noise to the latent space feature map of the initial sample image to obtain a noise feature map of the initial sample image; splicing the noise feature map of the initial sample image, the latent space feature map of the first sample image and the first mask image to obtain a spliced ​​feature map, and generating a training sample image set based on multiple spliced ​​feature maps.

[0010] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: obtaining an image to be processed, a first image and a target mask image, wherein the image to be processed is obtained by performing pixel expansion processing on the initial image to be processed according to target size information, the first image is obtained by processing the image to be processed according to the mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image; mapping the image to be processed from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the image to be processed, mapping the first image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first image; inputting the latent space feature map of the image to be processed, the latent space feature map of the first image and the target mask image into a target image generation model to obtain a target image, wherein the target image generation model is trained using any of the above-mentioned image generation model training methods.

[0011] Furthermore, the latent space feature map of the image to be processed, the latent space feature map of the first image, and the target mask image are input into the target image generation model to obtain the target image, including: randomly adding Gaussian noise to the latent space feature map of the image to be processed to obtain the noise feature map of the image to be processed; splicing the noise feature map of the image to be processed, the latent space feature map of the first image, and the target mask image to obtain a target spliced ​​feature map; and inputting the target spliced ​​feature map into the target image generation model to obtain the target image.

[0012] Furthermore, the target splicing feature map is input into the target image generation model to obtain the target image, including: repeatedly executing the target image generation model to randomly add the first Gaussian noise to the target splicing feature map according to the time step of the current denoising step to obtain the first noise feature map; predicting the second Gaussian noise, and determining the second noise feature map according to the second Gaussian noise and the first noise feature map; determining the first target noise feature map and the second target noise feature map according to the second noise feature map, wherein the first target noise feature map and the second target noise feature map contain different Gaussian noises; weightedly fusing the first target noise feature map and the second target noise feature map to obtain the noise feature map of the next denoising step of the current denoising step, until the current denoising step is a preset value, to obtain the latent space feature map of the target image; mapping the latent space feature map of the target image from the latent space to the pixel space through a preset decoder to obtain the target image.

[0013] Furthermore, determining the first target noise characteristic map and the second target noise characteristic map based on the second noise characteristic map includes: randomly adding a third Gaussian noise to the second noise characteristic map according to the next time step of the current denoising step to obtain a third noise characteristic map; determining the first target noise characteristic map based on the second noise characteristic map and the third noise characteristic map; determining the fourth noise characteristic map based on the second noise characteristic map and the second Gaussian noise, and determining the fifth noise characteristic map based on the fourth noise characteristic map and the third Gaussian noise; and determining the second target noise characteristic map based on the fourth noise characteristic map and the fifth noise characteristic map.

[0014] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: obtaining an image to be processed, a first image and a target mask image uploaded by a client, wherein the image to be processed is obtained by performing pixel expansion processing on the initial image to be processed according to target size information, the first image is obtained by processing the image to be processed according to the mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image; in a cloud server, mapping the image to be processed from a pixel space to a latent space through a preset encoder to obtain a latent space feature map of the image to be processed, and mapping the first image from a pixel space to a latent space through a preset encoder to obtain a latent space feature map of the first image; inputting the latent space feature map of the image to be processed, the latent space feature map of the first image and the target mask image into a target image generation model to obtain a target image, wherein the target image generation model is trained using any one of the above-mentioned image generation model training methods; and feeding back the target image to the client.

[0015] According to another aspect of an embodiment of the present application, a training device for an image generation model is also provided, including: a first acquisition unit, used to acquire a training sample image set, wherein the training sample image set includes multiple spliced ​​feature maps; a first processing unit, used to input the multiple spliced ​​feature maps into an initial student model to obtain a first prediction result output by the initial student model, and input the multiple spliced ​​feature maps into a teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is initialized based on the parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generation model; a second processing unit, used to construct a target loss function based on the first prediction result and the second prediction result, and based on the target loss function, train the parameters of the initial student model to obtain a target image generation model.

[0016] Furthermore, the first processing unit includes: a first processing sub-unit, used to perform a single-step denoising operation on multiple spliced ​​feature maps through an initial student model to obtain a first prediction result; a second processing sub-unit, used to perform a multi-step iterative denoising operation on multiple spliced ​​feature maps through a teacher model to obtain a second prediction result.

[0017] Furthermore, the second processing unit includes: a first construction subunit, used to construct a first loss function based on the first prediction result and the second prediction result; a second construction subunit, used to construct a second loss function based on the first prediction result and the latent space feature map of the initial sample image, wherein the latent space feature map of the initial sample image is used to generate a spliced ​​feature map; a third construction subunit, used to map the first prediction result from the latent space to the pixel space through a preset decoder to obtain a first generated image, and construct a third loss function based on the first generated image and the initial sample image.

[0018] Furthermore, the second processing unit also includes: a third processing sub-unit, which is used to train the parameters of the initial student model according to the first loss function if the difference between the first prediction result and the second prediction result is less than the first threshold as the training target; a fourth processing sub-unit, which is used to train the parameters of the initial student model according to the second loss function if the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than the second threshold as the training target; and a fifth processing sub-unit, which is used to train the parameters of the initial student model according to the third loss function if the distance between the features of the first generated image and the features of the initial sample image is less than the third threshold as the training target, until the preset conditions are met to obtain the target image generation model.

[0019] Furthermore, the device also includes: a second acquisition unit, used to acquire an initial sample image, a first sample image and a first mask image before acquiring the training sample image set, wherein the first sample image is obtained by processing the initial sample image according to the sample mask image corresponding to the initial sample image, and the first mask image is obtained by processing the sample mask image; a third processing unit, used to map the initial sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the initial sample image, and map the first sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first sample image; a fourth processing unit, used to randomly add Gaussian noise to the latent space feature map of the initial sample image to obtain a noise feature map of the initial sample image; a fifth processing unit, used to splice the noise feature map of the initial sample image, the latent space feature map of the first sample image and the first mask image to obtain a spliced ​​feature map, and generate a training sample image set based on multiple spliced ​​feature maps.

[0020] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: a third acquisition unit, used to acquire an image to be processed, a first image and a target mask image, wherein the image to be processed is obtained by performing pixel expansion processing on the initial image to be processed according to target size information, the first image is obtained by processing the image to be processed according to the mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image; a sixth processing unit, used to map the image to be processed from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the image to be processed, and map the first image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first image; a seventh processing unit, used to input the latent space feature map of the image to be processed, the latent space feature map of the first image and the target mask image into a target image generation model to obtain a target image, wherein the target image generation model is trained using any one of the above-mentioned image generation model training methods.

[0021] Furthermore, the seventh processing unit includes: a sixth processing sub-unit, used to randomly add Gaussian noise to the latent space feature map of the image to be processed to obtain a noise feature map of the image to be processed; a seventh processing sub-unit, used to splice the noise feature map of the image to be processed, the latent space feature map of the first image and the target mask image to obtain a target spliced ​​feature map; an eighth processing sub-unit, used to input the target spliced ​​feature map into a target image generation model to obtain a target image.

[0022] Furthermore, the eighth processing sub-unit includes: a first processing module, which is used to repeatedly execute the target image generation model to randomly add the first Gaussian noise to the target splicing feature map according to the time step of the current denoising step to obtain a first noise feature map; a first determination module, which is used to predict the second Gaussian noise, and determine the second noise feature map according to the second Gaussian noise and the first noise feature map; a second determination module, which is used to determine the first target noise feature map and the second target noise feature map according to the second noise feature map, wherein the first target noise feature map and the second target noise feature map contain different Gaussian noises; a second processing module, which is used to weightedly fuse the first target noise feature map and the second target noise feature map to obtain the noise feature map of the next denoising step of the current denoising step, until the current denoising step is a preset value, and obtain the latent space feature map of the target image; a third processing module, which is used to map the latent space feature map of the target image from the latent space to the pixel space through a preset decoder to obtain the target image.

[0023] Furthermore, the second determination module includes: a processing submodule, used to randomly add a third Gaussian noise to the second noise characteristic map according to the next time step of the current denoising step time step to obtain a third noise characteristic map; a first determination submodule, used to determine the first target noise characteristic map according to the second noise characteristic map and the third noise characteristic map; a second determination submodule, used to determine the fourth noise characteristic map according to the second noise characteristic map and the second Gaussian noise, and determine the fifth noise characteristic map according to the fourth noise characteristic map and the third Gaussian noise; a third determination submodule, used to determine the second target noise characteristic map according to the fourth noise characteristic map and the fifth noise characteristic map.

[0024] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein when the program is running, any one of the above-mentioned methods for training an image generation model is executed.

[0025] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned image generation model training methods.

[0026] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a computer program or instructions, which implements any of the above-mentioned image generation model training methods when executed by a processor.

[0027] In an embodiment of the present application, a training sample image set is obtained, wherein the training sample image set includes multiple spliced ​​feature maps; the multiple spliced ​​feature maps are input into an initial student model to obtain a first prediction result output by the initial student model; the multiple spliced ​​feature maps are input into a teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is obtained based on parameter initialization of the teacher model, and the teacher model is a pre-trained Vincent graph generation model; a target loss function is constructed based on the first prediction result and the second prediction result, and based on the target loss function, the parameters of the initial student model are trained to obtain a target image generation model, thereby realizing model distillation accelerated training, reducing the number of inference steps and the amount of parameters, reducing machine resource costs and merchant production costs, and being able to improve merchant usage experience and operational efficiency, achieving the purpose of reducing model deployment costs and inference delays, thereby achieving the technical effect of improving the efficiency of image generation, and further solving the technical problem in the related technology that the image generation model has a large number of inference steps and parameters, and has a high inference delay, resulting in relatively low image generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 is a schematic diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0030] Figure 2 is a flowchart of a training method for an image generation model provided in Example 1 of the present application;

[0031] Figure 3 is a schematic diagram of an optional model distillation acceleration framework provided according to Example 1 of the present application;

[0032] Figure 4 is a flowchart of an image processing method provided according to Embodiment 2 of the present application;

[0033] Figure 5 is a flowchart of an image processing method provided according to Embodiment 3 of the present application;

[0034] Figure 6 is a schematic diagram of a training device for an image generation model provided in Embodiment 4 of the present application;

[0035] Figure 7 is a schematic diagram of an image processing device provided according to Embodiment 5 of the present application;

[0036] Figure 8This is a structural block diagram of an electronic device provided according to Embodiment 6 of the present application. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] Example 1

[0041] According to an embodiment of the present application, a training method for an image generation model is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a training method for an image generation model. Figure 1As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (Field Programmable Gate Array, referred to as FPGA), and the processor set 102 may include a processor set, Figure 1 102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuits may be a single independent processing module, or may be integrated in whole or in part into any of the other components in the computer terminal 10 (or mobile device).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the training method of the image generation model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the training method of the image generation model described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] The display may be a touch screen type liquid crystal display that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0047] In the process of e-commerce operations, merchants need to prepare a large number of image materials for displaying products. Since different platforms and activities have different requirements for image material resolution and aspect ratio, merchants are often required to provide multiple sets of image materials of different proportions for the same product, which consumes a lot of labor costs. With the development of large models, it has become possible to automatically scale products based on AIGC technology. At present, the SDXL model is mainly used in related technologies for adaptive image content extension. However, the model has a large number of inference steps and parameters, and the deployment cost and inference delay are high, resulting in low efficiency of image generation.

[0048] In the above technical background, the present application provides Figure 2 The training method of the image generation model shown. Figure 2 : is a flow chart of a training method for an image generation model provided in Example 1 of the present application. The method comprises:

[0049] Step S201, obtaining a training sample image set, wherein the training sample image set includes a plurality of spliced ​​feature maps.

[0050] In order to obtain a training sample image set, in the training method of the image generation model provided in Example 1 of the present application, before obtaining the training sample image set, an initial sample image, a first sample image and a first mask image are obtained, wherein the first sample image is obtained by processing the initial sample image according to the sample mask image corresponding to the initial sample image, and the first mask image is obtained by processing the sample mask image; the initial sample image is mapped from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the initial sample image, and the first sample image is mapped from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first sample image; Gaussian noise is randomly added to the latent space feature map of the initial sample image to obtain a noise feature map of the initial sample image; the noise feature map of the initial sample image, the latent space feature map of the first sample image and the first mask image are spliced ​​to obtain a spliced ​​feature map, and a training sample image set is generated based on multiple spliced ​​feature maps.

[0051] Optionally, first obtain an initial sample image (i.e., original image) and a sample mask image corresponding to the initial sample image, and process the original image according to the sample mask image to obtain a masked image (i.e., first sample image). The sample mask image is downsampled to obtain a first mask image. The original image is mapped from pixel space to latent space by a preset encoder (e.g., an encoder in a pre-trained variational autoencoder) to obtain a latent space feature map of the original image, and the masked image is mapped from pixel space to latent space by an encoder in a pre-trained variational autoencoder to obtain a latent space feature map of the masked image. Gaussian noise is randomly added to the latent space feature map of the original image to obtain a noise feature map of the original image. The noise feature map of the original image, the latent space feature map of the masked image, and the first mask image are spliced ​​to obtain a spliced ​​feature map, and then a training sample image set can be generated based on multiple spliced ​​feature maps.

[0052] In an optional embodiment, it is possible to use Figure 3 The schematic diagram shown realizes the distillation accelerated training of the model. Figure 3 is a schematic diagram of an optional model distillation acceleration framework provided according to the first embodiment of the present application, and the sample mask image can be Figure 3 The mask image shown, the first mask image can be Figure 3 The image after downsampling the mask image in , the latent space feature map of the original image can be Figure 3 In the image after the encoder in the variational autoencoder maps the original image from the pixel space to the latent space, the noise feature map of the original image can be Figure 3 The image after adding noise to the latent space feature map of the original image in the latent space feature map of the masked image can be Figure 3In the example, the encoder in the variational autoencoder maps the masked image from the pixel space to the image in the latent space. Figure 3 As shown, the noise feature map of the original image, the latent space feature map of the masked image, and the first mask image are concatenated and input into the initial student model and the teacher model respectively.

[0053] Step S202, input multiple spliced ​​feature maps into the initial student model to obtain a first prediction result output by the initial student model, and input multiple spliced ​​feature maps into the teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is initialized based on the parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generative model.

[0054] The goal of distillation acceleration training is to train a student model (i.e., target image generation model) based on SDXL so that the student model can obtain similar results with fewer sampling steps (e.g., 8 steps) as the teacher model which needs to execute more steps (e.g., more than 30 steps). Optionally, the teacher model is a pre-trained Vincent graph generation model (e.g., a pre-trained SDXL extended model). For the student model, the same U-Net network structure as the teacher model is used and initialized with the parameters of the teacher model. During training, the parameters of the teacher model remain fixed and only the parameters of the student model are trained.

[0055] For example, the noise feature map of the original image, the latent space feature map of the masked image, and the first mask image are concatenated in the channel dimension and input into the initial student model and the teacher model respectively. The initial student model directly performs one-step denoising based on the input to obtain a prediction of the latent space feature map of the original image (i.e., the first prediction result), and the teacher model performs multi-step iterative denoising based on the input to obtain a prediction of the latent space feature map of the original image (i.e., the second prediction result).

[0056] Step S203, constructing a target loss function based on the first prediction result and the second prediction result, and training the parameters of the initial student model based on the target loss function to obtain a target image generation model.

[0057] Optionally, a target loss function is constructed based on the one-step prediction of the student model and the multi-step prediction of the teacher model, such as distillation loss, adversarial loss, and perceptual loss, wherein distillation loss and adversarial loss are used in latent space, and perceptual loss is constrained in the original pixel space to increase the details of the generated image. Based on the target loss function, the parameters of the initial student model are trained until the preset conditions are met (for example, the preset number of training steps is reached, etc.), and the trained student model (i.e., the target image generation model) can be obtained.

[0058] In this solution, the distillation of the model is implemented to accelerate training, reduce the number of inference steps and parameters, reduce machine resource costs and merchant production costs, improve merchant usage experience and operational efficiency, and achieve the purpose of reducing the deployment cost and inference delay of the model, thereby achieving the technical effect of improving the efficiency of image generation, and further solving the technical problem in related technologies that the image generation model has a large number of inference steps and parameters, and has a high inference delay, resulting in relatively low image generation efficiency.

[0059] In order to achieve accelerated training of the model by distillation, in the training method of the image generation model provided in Example 1 of the present application, multiple spliced ​​feature maps are input into the initial student model to obtain a first prediction result output by the initial student model, and multiple spliced ​​feature maps are input into the teacher model to obtain a second prediction result output by the teacher model, including: performing a single-step denoising operation on the multiple spliced ​​feature maps through the initial student model to obtain a first prediction result; performing a multi-step iterative denoising operation on the multiple spliced ​​feature maps through the teacher model to obtain a second prediction result.

[0060] For example, the first prediction result is Figure 3 The image obtained by one-step denoising of the concatenated feature map by the initial student model, the second prediction result is Figure 3 The image is obtained by performing multi-step iterative denoising on the concatenated feature map through the teacher model.

[0061] In order to better train the model, in the training method of the image generation model provided in Example 1 of the present application, the target loss function includes at least one of the following: a first loss function, a second loss function, and a third loss function. The first loss function, the second loss function, and the third loss function correspond to different training targets. Constructing the target loss function based on the first prediction result and the second prediction result includes: constructing a first loss function based on the first prediction result and the second prediction result; constructing a second loss function based on the first prediction result and a latent space feature map of the initial sample image, wherein the latent space feature map of the initial sample image is used to generate a spliced ​​feature map; mapping the first prediction result from the latent space to the pixel space through a preset decoder to obtain a first generated image, and constructing a third loss function based on the first generated image and the initial sample image.

[0062] Optionally, a distillation loss (i.e., the first loss function) is used to make the one-step prediction result of the student model consistent with the multi-step prediction result of the teacher model. Figure 3 As shown, the distillation loss is constructed based on the one-step prediction result of the student model (i.e., the first prediction result) and the multi-step prediction result of the teacher model (i.e., the second prediction result), which is expressed as follows:

[0063]

[0064] in, represents the one-step denoising prediction result of the student model, Represents the multi-step denoising prediction result of the teacher model. For example, the latent space feature map of the original image x is recorded as z0. During training, a time step t is first randomly sampled in the predefined time step range [0, T], and then Gaussian noise ∈ is added to z0 based on the time step t. t Get the noise feature map z of the original image t :

[0065]

[0066] Among them, α t and σ t Predefined noise intensity control coefficients for SDXL.

[0067] Then the splicing is performed, and the spliced ​​feature map is input into the student model to predict the Gaussian noise ∈ θ , thus obtaining:

[0068]

[0069] Optionally, the teacher model uses a solver to predict using multi-step iterative sampling

[0070] Optionally, an adversarial loss (i.e., the second loss function) is used to make the distribution of the result predicted by the student model in one step consistent with the latent space feature distribution of the real image (i.e., the original image). Figure 3 As shown in the figure, the adversarial loss is constructed based on the one-step prediction result of the student model (i.e., the first prediction result) and the latent space feature map of the original image. Since the adversarial loss requires the introduction of a new network as the discriminant network D, in order to reduce the training cost, it can be constructed by adding two convolutional layers on the basis of the U-Net encoder of the student model, and only the parameters of the newly added convolutional layers are trained during training. Based on this, the generator loss L included in the adversarial loss adv and the discriminator loss L disc They are respectively expressed as follows:

[0071]

[0072]

[0073] Optionally, a perceptual loss (i.e., the third loss function) is used to make the predicted image consistent with the perceptual features of the real image in the pixel space. Figure 3 As shown, the first prediction result is mapped from the latent space to the pixel space through a preset decoder (for example, a decoder in a pre-trained variational autoencoder) to obtain a first generated image, and a perceptual loss is constructed based on the first generated image and the original image.

[0074] For example, the student model is denoised in one step. The decoder in the pre-trained variational autoencoder maps the latent space to the pixel space to obtain the first generated image Then a perceptual loss is applied in pixel space. Optionally, a pre-trained Visual Geometry Group (VGG) network is used to extract multi-layer features of the image predicted by the student model in one step and the original image, respectively, and then the distance between the predicted image and the original image features is calculated. The perceptual loss is expressed as follows:

[0075]

[0076] Among them, H l and W l represents the height and width of the feature image of the lth layer, h and w represent the feature map index, and w l is a learnable vector, and the f function is the VGG network.

[0077] By making the one-step prediction result of the student model approach the multi-step prediction result of the teacher model, the cumulative error can be effectively avoided, and there is no need to adopt multi-stage restart training, and the training cost is low; by introducing adversarial loss, the quality of images generated by the model can be enhanced; by introducing perceptual loss, the details of the images generated by the model can be enhanced.

[0078] In order to obtain a trained student model, in the training method of the image generation model provided in Example 1 of the present application, the parameters of the initial student model are trained based on the target loss function to obtain the target image generation model, including: if the difference between the first prediction result and the second prediction result is less than the first threshold as the training target, the parameters of the initial student model are trained according to the first loss function; if the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than the second threshold as the training target, the parameters of the initial student model are trained according to the second loss function; if the distance between the features of the first generated image and the features of the initial sample image is less than the third threshold as the training target, the parameters of the initial student model are trained according to the third loss function until the preset conditions are met to obtain the target image generation model.

[0079] Optionally, the distillation acceleration training strategy of the model adopts three losses for training as a whole, wherein the distillation loss is used to make the one-step prediction result of the student model consistent with the multi-step prediction result of the teacher model, that is, the difference between the first prediction result and the second prediction result is less than a first threshold as the training target; the adversarial loss is used to make the result distribution of the one-step prediction of the student model consistent with the latent space feature distribution of the real image, that is, the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than a second threshold as the training target; the perceptual loss is used to make the perceptual features of the predicted image and the real image consistent in the pixel space, that is, the distance between the features of the first generated image and the features of the initial sample image is less than a third threshold as the training target, until the preset conditions are met (for example, the preset number of training steps is reached, etc.), and the trained student model (that is, the target image generation model) can be obtained.

[0080] In an embodiment of the present application, a training sample image set is obtained, wherein the training sample image set includes multiple spliced ​​feature maps; the multiple spliced ​​feature maps are input into an initial student model to obtain a first prediction result output by the initial student model; the multiple spliced ​​feature maps are input into a teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is obtained based on parameter initialization of the teacher model, and the teacher model is a pre-trained Vincent graph generation model; a target loss function is constructed based on the first prediction result and the second prediction result, and based on the target loss function, the parameters of the initial student model are trained to obtain a target image generation model, thereby realizing model distillation accelerated training, reducing the number of inference steps and the amount of parameters, reducing machine resource costs and merchant production costs, and being able to improve merchant usage experience and operational efficiency, achieving the purpose of reducing model deployment costs and inference delays, thereby achieving the technical effect of improving the efficiency of image generation, and further solving the technical problem in the related technology that the image generation model has a large number of inference steps and parameters, and has a high inference delay, resulting in relatively low image generation efficiency.

[0081] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0083] Example 2

[0084] According to an embodiment of the present application, an image processing method is also provided, such as Figure 4 As shown, the method includes:

[0085] Step S401, obtaining an image to be processed, a first image and a target mask image, wherein the image to be processed is obtained by performing pixel expansion processing on the initial image to be processed according to target size information, the first image is obtained by processing the image to be processed according to the mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image;

[0086] Step S402, mapping the image to be processed from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the image to be processed, and mapping the first image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first image;

[0087] Step S403, inputting the latent space feature map of the image to be processed, the latent space feature map of the first image and the target mask image into a target image generation model to obtain a target image, wherein the target image generation model is trained using any of the above-mentioned image generation model training methods.

[0088] Optionally, this solution can be applied to provide intelligent image extension services. For example, the original image to be extended (i.e., the initial image to be processed) is first extended to the target size to be extended by copying the edge pixels of the image. For example, the width and height of the original image are 800*800, and the extension target size (i.e., the target size information) is 800*1200. The original image can be expanded to a size of 800*1200 by copying the upper and lower edge pixels respectively, and the expanded image is used as the image to be processed. The target mask image is obtained by obtaining the corresponding mask image for downsampling, and the masked image (i.e., the first image) is obtained by processing the image to be processed with the mask image. Then, consistent with the training, the image to be processed is mapped from the pixel space to the latent space through a preset encoder (for example, the encoder in the pre-trained variational autoencoder) to obtain a latent space feature map of the image to be processed, and the first image is mapped from the pixel space to the latent space through the preset encoder to obtain a latent space feature map of the first image. Then, the latent space feature map of the image to be processed, the latent space feature map of the masked image, and the target mask image are input into the target image generation model to obtain the extended image (i.e., the target image).

[0089] In order to obtain an extended image, in the image processing method provided in Example 2 of the present application, the latent space feature map of the image to be processed, the latent space feature map of the first image, and the target mask image are input into a target image generation model to obtain the target image, including: randomly adding Gaussian noise to the latent space feature map of the image to be processed to obtain a noise feature map of the image to be processed; splicing the noise feature map of the image to be processed, the latent space feature map of the first image, and the target mask image to obtain a target spliced ​​feature map; and inputting the target spliced ​​feature map into the target image generation model to obtain the target image.

[0090] Optionally, consistent with the training, Gaussian noise is randomly added to the latent space feature map of the image to be processed to obtain a noise feature map of the image to be processed. The noise feature map of the image to be processed, the latent space feature map of the masked image, and the target mask image are spliced ​​to obtain a target spliced ​​feature map, and then the target spliced ​​feature map is input into the target image generation model to obtain the target image.

[0091] In order to obtain an extended image, in the image processing method provided in Example 2 of the present application, the target splicing feature map is input into the target image generation model, and obtaining the target image includes: repeatedly executing the target image generation model to randomly add the first Gaussian noise to the target splicing feature map according to the time step of the current denoising step to obtain the first noise feature map; predicting the second Gaussian noise, and determining the second noise feature map according to the second Gaussian noise and the first noise feature map; determining the first target noise feature map and the second target noise feature map according to the second noise feature map, wherein the first target noise feature map and the second target noise feature map contain different Gaussian noises; weightedly fusion the first target noise feature map and the second target noise feature map to obtain the noise feature map of the next denoising step of the current denoising step, until the current denoising step is a preset value m, to obtain the latent space feature map of the target image; mapping the latent space feature map of the target image from the latent space to the pixel space through a preset decoder to obtain the target image.

[0092] In order to obtain an extended image, in the image processing method provided in Example 2 of the present application, determining the first target noise characteristic map and the second target noise characteristic map based on the second noise characteristic map includes: randomly adding a third Gaussian noise to the second noise characteristic map according to the next time step of the current denoising step to obtain a third noise characteristic map; determining the first target noise characteristic map based on the second noise characteristic map and the third noise characteristic map; determining the fourth noise characteristic map based on the second noise characteristic map and the second Gaussian noise, and determining the fifth noise characteristic map based on the fourth noise characteristic map and the third Gaussian noise; determining the second target noise characteristic map based on the fourth noise characteristic map and the fifth noise characteristic map.

[0093] Optionally, in order to adapt the distillation acceleration training algorithm proposed in this solution, a multi-step sampling algorithm is proposed. Assume that the number of sampling steps during inference is k, and the time steps of k samples are recorded as t1, t2, ..., t k .

[0094] Step 1: Based on the time step t of the current denoising step k k Randomly add the first Gaussian noise ∈ to the target concatenated feature map z0 k Get the first noise characteristic map z k ;

[0095] Step 2: Predict the second Gaussian noise According to the second Gaussian noise and the first noise characteristic map z k Determine the second noise signature It is expressed as follows:

[0096]

[0097] Among them, σ k and αk The value of is consistent with that during training. Then, based on the next time step t of the current denoising step k-1 For the second noise characteristic diagram Randomly add a third Gaussian noise ∈ k-1 The third noise characteristic graph is obtained, which is expressed as follows:

[0098]

[0099] The first target noise characteristic graph is determined according to the second noise characteristic graph and the third noise characteristic graph, which is expressed as follows:

[0100]

[0101] The fourth noise characteristic graph is determined according to the second noise characteristic graph and the second Gaussian noise, and is expressed as follows:

[0102]

[0103] The fifth noise characteristic graph is determined according to the fourth noise characteristic graph and the third Gaussian noise, which is expressed as follows:

[0104]

[0105] The second target noise characteristic graph is determined according to the fourth noise characteristic graph and the fifth noise characteristic graph, which is expressed as follows:

[0106]

[0107] Step 3: The first target noise feature map and the second target noise signature Perform weighted fusion to obtain the noise feature map z of the next denoising step k-1 of the current denoising step k-1 , which is expressed as follows:

[0108]

[0109] Among them, 0<w<1 is a predefined weighting coefficient, and then the current denoising time step k is set to k-1.

[0110] Step 4: Repeat the above three steps until k=1, and then get the latent space feature map of the target image, which is expressed as follows:

[0111]

[0112] Then, the latent space feature map of the target image is mapped from the latent space to the pixel space through the decoder in the pre-trained variational autoencoder to obtain the target image.

[0113] In an embodiment of the present application, in order to reduce the number of inference steps and the amount of parameters, a distillation acceleration training algorithm is proposed. In order to adapt the distillation acceleration training algorithm, a multi-step sampling algorithm during inference is proposed. The multi-step sampling algorithm combined with the distillation acceleration training algorithm can produce better image quality and obtain more detailed and natural image extension results.

[0114] Example 3

[0115] According to an embodiment of the present application, an image processing method is also provided, such as Figure 5 As shown, the method includes:

[0116] Step S501, obtaining an image to be processed, a first image and a target mask image uploaded by a client, wherein the image to be processed is obtained by performing pixel expansion processing on the initial image to be processed according to target size information, the first image is obtained by processing the image to be processed according to the mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image;

[0117] Step S502, in the cloud server, mapping the image to be processed from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the image to be processed, mapping the first image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first image; inputting the latent space feature map of the image to be processed, the latent space feature map of the first image, and the target mask image into a target image generation model to obtain a target image, wherein the target image generation model is trained using any one of the above-mentioned image generation model training methods;

[0118] Step S503: Feedback the target image to the client.

[0119] In the cloud server, the specific method for image processing is the same as that in the second embodiment, and will not be repeated here.

[0120] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0122] Example 4

[0123] According to an embodiment of the present application, a training device for an image generation model for implementing the training method of the above-mentioned image generation model is also provided, such as Figure 6 As shown, the device includes: a first acquisition unit 601, a first processing unit 602, and a second processing unit 603.

[0124] A first acquisition unit 601 is used to acquire a training sample image set, wherein the training sample image set includes a plurality of spliced ​​feature maps;

[0125] The first processing unit 602 is used to input the multiple spliced ​​feature maps into the initial student model to obtain a first prediction result output by the initial student model, and input the multiple spliced ​​feature maps into the teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is obtained by initializing the teacher model according to the parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generative model;

[0126] The second processing unit 603 is used to construct a target loss function based on the first prediction result and the second prediction result, and train the parameters of the initial student model based on the target loss function to obtain a target image generation model.

[0127] In the training device of the image generation model provided in the fourth embodiment of the present application, a training sample image set is obtained by a first acquisition unit 601, wherein the training sample image set includes a plurality of spliced ​​feature maps; a first processing unit 602 inputs the plurality of spliced ​​feature maps into an initial student model to obtain a first prediction result output by the initial student model, and inputs the plurality of spliced ​​feature maps into a teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is obtained by initializing the parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generation model; a second processing unit 603 constructs a target loss function based on the first prediction result and the second prediction result, and trains the parameters of the initial student model based on the target loss function to obtain a target image generation model. In this solution, the distillation acceleration training of the model is realized, the number of inference steps and the amount of parameters are reduced, the machine resource cost and the merchant production cost are reduced, the merchant use experience and operation efficiency can be improved, and the purpose of reducing the deployment cost and inference delay of the model is achieved, thereby achieving the technical effect of improving the efficiency of image generation, and thus solving the technical problem that the number of inference steps and the amount of parameters of the image generation model in the related technology are large, and the inference delay is high, resulting in low efficiency of image generation.

[0128] Optionally, in the training device of the image generation model provided in Example 4 of the present application, the first processing unit 602 includes: a first processing sub-unit, used to perform a single-step denoising operation on multiple spliced ​​feature maps through an initial student model to obtain a first prediction result; a second processing sub-unit, used to perform a multi-step iterative denoising operation on multiple spliced ​​feature maps through a teacher model to obtain a second prediction result.

[0129] Optionally, in the training device of the image generation model provided in Example 4 of the present application, the second processing unit 603 includes: a first construction subunit, used to construct a first loss function based on the first prediction result and the second prediction result; a second construction subunit, used to construct a second loss function based on the first prediction result and the latent space feature map of the initial sample image, wherein the latent space feature map of the initial sample image is used to generate a spliced ​​feature map; a third construction subunit, used to map the first prediction result from the latent space to the pixel space through a preset decoder to obtain a first generated image, and construct a third loss function based on the first generated image and the initial sample image.

[0130] Optionally, in the training device of the image generation model provided in Example 4 of the present application, the second processing unit 603 also includes: a third processing sub-unit, used to train the parameters of the initial student model according to the first loss function if the difference between the first prediction result and the second prediction result is less than the first threshold as the training target; a fourth processing sub-unit, used to train the parameters of the initial student model according to the second loss function if the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than the second threshold as the training target; a fifth processing sub-unit, used to train the parameters of the initial student model according to the third loss function if the distance between the features of the first generated image and the features of the initial sample image is less than the third threshold as the training target, until the preset conditions are met to obtain the target image generation model.

[0131] Optionally, in the training device of the image generation model provided in Example 4 of the present application, the device also includes: a second acquisition unit, used to acquire an initial sample image, a first sample image and a first mask image before acquiring the training sample image set, wherein the first sample image is obtained by processing the initial sample image according to the sample mask image corresponding to the initial sample image, and the first mask image is obtained by processing the sample mask image; a third processing unit, used to map the initial sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the initial sample image, and map the first sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first sample image; a fourth processing unit, used to randomly add Gaussian noise to the latent space feature map of the initial sample image to obtain a noise feature map of the initial sample image; a fifth processing unit, used to splice the noise feature map of the initial sample image, the latent space feature map of the first sample image and the first mask image to obtain a spliced ​​feature map, and generate a training sample image set based on multiple spliced ​​feature maps.

[0132] It should be noted that the above-mentioned first acquisition unit 601, first processing unit 602, and second processing unit 603 correspond to steps S201 to S203 in Example 1, and the above-mentioned units and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0133] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0134] Example 5

[0135] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Figure 7 As shown, the device includes: a third acquisition unit 701, a sixth processing unit 702, and a seventh processing unit 703.

[0136] The third acquisition unit 701 is used to acquire the image to be processed, the first image and the target mask image, wherein the image to be processed is obtained by performing pixel expansion processing on the initial image to be processed according to the target size information, the first image is obtained by processing the image to be processed according to the mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image;

[0137] The sixth processing unit 702 is used to map the image to be processed from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the image to be processed, and map the first image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first image;

[0138] The seventh processing unit 703 is used to input the latent space feature map of the image to be processed, the latent space feature map of the first image and the target mask image into the target image generation model to obtain the target image, wherein the target image generation model is trained using any of the above-mentioned image generation model training methods.

[0139] Optionally, in the image processing device provided in Example 5 of the present application, the seventh processing unit includes: a sixth processing sub-unit, used to randomly add Gaussian noise to the latent space feature map of the image to be processed to obtain a noise feature map of the image to be processed; a seventh processing sub-unit, used to splice the noise feature map of the image to be processed, the latent space feature map of the first image, and the target mask image to obtain a target spliced ​​feature map; and an eighth processing sub-unit, used to input the target spliced ​​feature map into a target image generation model to obtain a target image.

[0140] Optionally, in the image processing device provided in Example 5 of the present application, the eighth processing sub-unit includes: a first processing module, used to repeatedly execute the target image generation model to randomly add the first Gaussian noise to the target splicing feature map according to the time step of the current denoising step to obtain a first noise feature map; a first determination module, used to predict the second Gaussian noise, and determine the second noise feature map based on the second Gaussian noise and the first noise feature map; a second determination module, used to determine the first target noise feature map and the second target noise feature map based on the second noise feature map, wherein the first target noise feature map and the second target noise feature map contain different Gaussian noises; a second processing module, used to weightedly fuse the first target noise feature map and the second target noise feature map to obtain the noise feature map of the next denoising step of the current denoising step, until the current denoising step is a preset value, and obtain the latent space feature map of the target image; a third processing module, used to map the latent space feature map of the target image from the latent space to the pixel space through a preset decoder to obtain the target image.

[0141] Optionally, in the image processing device provided in Example 5 of the present application, the second determination module includes: a processing submodule, used to randomly add a third Gaussian noise to the second noise characteristic map according to the next time step of the current denoising step time step to obtain a third noise characteristic map; a first determination submodule, used to determine the first target noise characteristic map based on the second noise characteristic map and the third noise characteristic map; a second determination submodule, used to determine the fourth noise characteristic map based on the second noise characteristic map and the second Gaussian noise, and determine the fifth noise characteristic map based on the fourth noise characteristic map and the third Gaussian noise; a third determination submodule, used to determine the second target noise characteristic map based on the fourth noise characteristic map and the fifth noise characteristic map.

[0142] It should be noted here that the above-mentioned third acquisition unit 701, sixth processing unit 702, and seventh processing unit 703 correspond to steps S401 to S403 in Example 2, and the instances and application scenarios implemented by the above-mentioned units and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 2.

[0143] Example 6

[0144] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0145] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0146] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the training method of the image generation model: obtaining a training sample image set, wherein the training sample image set includes multiple spliced ​​feature maps; inputting the multiple spliced ​​feature maps into the initial student model to obtain a first prediction result output by the initial student model, and inputting the multiple spliced ​​feature maps into the teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is initialized based on the parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generation model; constructing a target loss function based on the first prediction result and the second prediction result, and based on the target loss function, training the parameters of the initial student model to obtain a target image generation model.

[0147] The above-mentioned electronic device can also execute the program code of the following steps in the training method of the image generation model: performing a single-step denoising operation on multiple spliced ​​feature maps through the initial student model to obtain a first prediction result; performing a multi-step iterative denoising operation on multiple spliced ​​feature maps through the teacher model to obtain a second prediction result.

[0148] The above-mentioned electronic device can also execute the program code of the following steps in the training method of the image generation model: constructing a first loss function based on the first prediction result and the second prediction result; constructing a second loss function based on the first prediction result and the latent space feature map of the initial sample image, wherein the latent space feature map of the initial sample image is used to generate a spliced ​​feature map; mapping the first prediction result from the latent space to the pixel space through a preset decoder to obtain a first generated image, and constructing a third loss function based on the first generated image and the initial sample image.

[0149] The above-mentioned electronic device can also execute the program code of the following steps in the training method of the image generation model: if the difference between the first prediction result and the second prediction result is less than the first threshold value as the training target, the parameters of the initial student model are trained according to the first loss function; if the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than the second threshold value as the training target, the parameters of the initial student model are trained according to the second loss function; if the distance between the features of the first generated image and the features of the initial sample image is less than the third threshold value as the training target, the parameters of the initial student model are trained according to the third loss function until the preset conditions are met to obtain the target image generation model.

[0150] The above-mentioned electronic device can also execute the program code of the following steps in the training method of the image generation model: before obtaining the training sample image set, obtain an initial sample image, a first sample image and a first mask image, wherein the first sample image is obtained by processing the initial sample image according to the sample mask image corresponding to the initial sample image, and the first mask image is obtained by processing the sample mask image; mapping the initial sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the initial sample image, mapping the first sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first sample image; randomly adding Gaussian noise to the latent space feature map of the initial sample image to obtain a noise feature map of the initial sample image; splicing the noise feature map of the initial sample image, the latent space feature map of the first sample image and the first mask image to obtain a spliced ​​feature map, and generating a training sample image set based on multiple spliced ​​feature maps.

[0151] Optionally, Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device 80 may include: one or more ( Figure 8 Only one is shown in the figure) processor 802 and memory 804. The electronic device 80 may also include a storage controller, through which the memory 804 is controlled and managed; the electronic device 80 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0152] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method and device of the image generation model in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the training method of the above-mentioned image generation model is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a training sample image set, wherein the training sample image set includes multiple spliced ​​feature maps; input the multiple spliced ​​feature maps into the initial student model to obtain a first prediction result output by the initial student model, and input the multiple spliced ​​feature maps into the teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is initialized based on the parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generation model; construct a target loss function based on the first prediction result and the second prediction result, and based on the target loss function, train the parameters of the initial student model to obtain a target image generation model.

[0154] Optionally, the processor may also execute the program code of the following steps: performing a single-step denoising operation on multiple spliced ​​feature maps through an initial student model to obtain a first prediction result; performing a multi-step iterative denoising operation on multiple spliced ​​feature maps through a teacher model to obtain a second prediction result.

[0155] Optionally, the processor may also execute the program code of the following steps: constructing a first loss function based on the first prediction result and the second prediction result; constructing a second loss function based on the first prediction result and the latent space feature map of the initial sample image, wherein the latent space feature map of the initial sample image is used to generate a spliced ​​feature map; mapping the first prediction result from the latent space to the pixel space through a preset decoder to obtain a first generated image, and constructing a third loss function based on the first generated image and the initial sample image.

[0156] Optionally, the processor may also execute the program code of the following steps: if the difference between the first prediction result and the second prediction result is less than a first threshold as the training target, the parameters of the initial student model are trained according to the first loss function; if the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than a second threshold as the training target, the parameters of the initial student model are trained according to the second loss function; if the distance between the features of the first generated image and the features of the initial sample image is less than a third threshold as the training target, the parameters of the initial student model are trained according to the third loss function until the preset conditions are met to obtain the target image generation model.

[0157] Optionally, the processor may also execute the program code of the following steps: before obtaining the training sample image set, obtaining an initial sample image, a first sample image and a first mask image, wherein the first sample image is obtained by processing the initial sample image according to the sample mask image corresponding to the initial sample image, and the first mask image is obtained by processing the sample mask image; mapping the initial sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the initial sample image, mapping the first sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the first sample image; randomly adding Gaussian noise to the latent space feature map of the initial sample image to obtain a noise feature map of the initial sample image; splicing the noise feature map of the initial sample image, the latent space feature map of the first sample image and the first mask image to obtain a spliced ​​feature map, and generating a training sample image set based on multiple spliced ​​feature maps.

[0158] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 8 The structure of the electronic device is not limited. Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations shown.

[0159] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0160] Example 7

[0161] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the training method of the image generation model provided in the first embodiment.

[0162] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0163] Example 8

[0164] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the training method of the image generation model provided in the first embodiment is implemented.

[0165] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0166] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0171] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A training method for an image generation model, characterized in that: include: Acquire a training sample image set, wherein the training sample image set includes a plurality of spliced ​​feature maps; Inputting the multiple spliced ​​feature maps into an initial student model to obtain a first prediction result output by the initial student model, and inputting the multiple spliced ​​feature maps into a teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is initialized based on the parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generative model; A target loss function is constructed based on the first prediction result and the second prediction result, and based on the target loss function, the parameters of the initial student model are trained to obtain a target image generation model.

2. The method according to claim 1, characterized in that: Inputting the multiple spliced ​​feature maps into an initial student model to obtain a first prediction result output by the initial student model, and inputting the multiple spliced ​​feature maps into a teacher model to obtain a second prediction result output by the teacher model comprises: Performing a single-step denoising operation on the plurality of concatenated feature maps by using the initial student model to obtain the first prediction result; The second prediction result is obtained by performing a multi-step iterative denoising operation on the multiple spliced ​​feature maps through the teacher model.

3. The method according to claim 1, characterized in that The target loss function includes at least one of the following: a first loss function, a second loss function, and a third loss function, wherein the first loss function, the second loss function, and the third loss function correspond to different training objectives, and constructing the target loss function based on the first prediction result and the second prediction result includes: Constructing the first loss function according to the first prediction result and the second prediction result; Constructing the second loss function according to the first prediction result and a latent space feature map of an initial sample image, wherein the latent space feature map of the initial sample image is used to generate the splicing feature map; The first prediction result is mapped from the latent space to the pixel space through a preset decoder to obtain a first generated image, and the third loss function is constructed based on the first generated image and the initial sample image.

4. The method according to claim 3, characterized in that Based on the target loss function, the parameters of the initial student model are trained to obtain a target image generation model including: If the difference between the first prediction result and the second prediction result is less than a first threshold as a training target, training the parameters of the initial student model according to the first loss function; If the similarity between the feature distribution of the first prediction result and the feature distribution of the latent space feature map of the initial sample image is less than a second threshold as a training target, training the parameters of the initial student model according to the second loss function; If the distance between the features of the first generated image and the features of the initial sample image is less than a third threshold as the training target, the parameters of the initial student model are trained according to the third loss function until the preset conditions are met to obtain the target image generation model.

5. The method according to claim 1, characterized in that Before acquiring the training sample image set, the method further includes: Acquire an initial sample image, a first sample image, and a first mask image, wherein the first sample image is obtained by processing the initial sample image according to a sample mask image corresponding to the initial sample image, and the first mask image is obtained by processing the sample mask image; Mapping the initial sample image from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the initial sample image, and mapping the first sample image from the pixel space to the latent space through the preset encoder to obtain a latent space feature map of the first sample image; Randomly adding Gaussian noise to the latent space feature map of the initial sample image to obtain a noise feature map of the initial sample image; The noise feature map of the initial sample image, the latent space feature map of the first sample image and the first mask image are spliced ​​to obtain a spliced ​​feature map, and the training sample image set is generated based on a plurality of spliced ​​feature maps.

6. An image processing method, characterized in that: include: Acquire an image to be processed, a first image and a target mask image, wherein the image to be processed is obtained by performing pixel expansion processing on an initial image to be processed according to target size information, the first image is obtained by processing the image to be processed according to a mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image; Mapping the image to be processed from the pixel space to the latent space through a preset encoder to obtain a latent space feature map of the image to be processed, and mapping the first image from the pixel space to the latent space through the preset encoder to obtain a latent space feature map of the first image; The latent space feature map of the image to be processed, the latent space feature map of the first image and the target mask image are input into a target image generation model to obtain a target image, wherein the target image generation model is trained using the image generation model training method described in any one of claims 1 to 5.

7. The method according to claim 6, characterized in that Inputting the latent space feature map of the image to be processed, the latent space feature map of the first image, and the target mask image into a target image generation model to obtain a target image includes: Randomly adding Gaussian noise to the latent space feature map of the image to be processed to obtain a noise feature map of the image to be processed; splicing the noise feature map of the image to be processed, the latent space feature map of the first image, and the target mask image to obtain a target splicing feature map; The target splicing feature map is input into the target image generation model to obtain the target image.

8. The method according to claim 7, characterized in that Inputting the target splicing feature map into the target image generation model to obtain the target image includes: Repeating the process of randomly adding a first Gaussian noise to the target splicing feature map according to the time step of the current denoising step through the target image generation model to obtain a first noise feature map; Predicting a second Gaussian noise, and determining a second noise characteristic graph according to the second Gaussian noise and the first noise characteristic graph; Determining a first target noise characteristic graph and a second target noise characteristic graph according to the second noise characteristic graph, wherein the first target noise characteristic graph and the second target noise characteristic graph contain different Gaussian noises; Performing weighted fusion on the first target noise feature map and the second target noise feature map to obtain a noise feature map of the next denoising step of the current denoising step, until the current denoising step reaches a preset value, to obtain a latent space feature map of the target image; The latent space feature map of the target image is mapped from the latent space to the pixel space through a preset decoder to obtain the target image.

9. The method according to claim 8, characterized in that Determining a first target noise characteristic graph and a second target noise characteristic graph according to the second noise characteristic graph includes: Randomly adding a third Gaussian noise to the second noise characteristic graph according to a time step next to the time step of the current denoising step to obtain a third noise characteristic graph; Determining the first target noise characteristic graph according to the second noise characteristic graph and the third noise characteristic graph; Determine a fourth noise characteristic graph according to the second noise characteristic graph and the second Gaussian noise, and determine a fifth noise characteristic graph according to the fourth noise characteristic graph and the third Gaussian noise; The second target noise characteristic graph is determined according to the fourth noise characteristic graph and the fifth noise characteristic graph.

10. An image processing method, characterized in that: include: Acquire an image to be processed, a first image and a target mask image uploaded by a client, wherein the image to be processed is obtained by performing pixel expansion processing on an initial image to be processed according to target size information, the first image is obtained by processing the image to be processed according to a mask image corresponding to the image to be processed, and the target mask image is obtained by processing the mask image; In the cloud server, the image to be processed is mapped from the pixel space to the latent space by a preset encoder to obtain a latent space feature map of the image to be processed, and the first image is mapped from the pixel space to the latent space by the preset encoder to obtain a latent space feature map of the first image; the latent space feature map of the image to be processed, the latent space feature map of the first image and the target mask image are input into a target image generation model to obtain a target image, wherein the target image generation model is trained using the training method of the image generation model described in any one of claims 1 to 5; The target image is fed back to the client.

11. A training device for an image generation model, characterized in that: include: A first acquisition unit is used to acquire a training sample image set, wherein the training sample image set includes a plurality of splicing feature maps; A first processing unit is configured to input the plurality of spliced ​​feature maps into an initial student model to obtain a first prediction result output by the initial student model, and input the plurality of spliced ​​feature maps into a teacher model to obtain a second prediction result output by the teacher model, wherein the initial student model is initialized based on parameters of the teacher model, and the teacher model is a pre-trained Vincent graph generative model; The second processing unit is used to construct a target loss function based on the first prediction result and the second prediction result, and train the parameters of the initial student model based on the target loss function to obtain a target image generation model.

12. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein the program, when running, executes the training method of the image generation model of any one of claims 1 to 5.

13. A computer program product, characterized in that It comprises a computer program or instructions, which, when executed by a processor, implements the training method of the image generation model according to any one of claims 1 to 5.

Images