Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment, and the method comprises the steps: obtaining a first model for converting a text into an image, a to-be-trained text-to-image model and a first text, the model step number of the first model is M * N, the model step number of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0; processing the first text according to the first model, determining a first prediction image generated by the first model in each step, and obtaining M * N first prediction images; processing the first text according to the text-to-image model, determining a second prediction image generated by the text-to-image model in each step, and obtaining N second prediction images; and according to the M * N first prediction images and the N second prediction images, updating model parameters of the text-to-image model. And the text-to-image conversion efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] Multimodal generation technology means that the model can generate data in multiple different forms. For example, the model can convert text into images, and the model can also convert speech into images, etc.

[0003] Currently, electronic devices can train multimodal models based on training samples. For example, electronic devices can train a text-to-image model based on sample text and the image corresponding to the sample text. At the end of the training, the text-to-image model can convert text into an image. However, the model generated by multimodality has a large number of steps, and the model takes a long time to reason, which leads to low efficiency in model processing. Summary of the invention

[0004] The present disclosure provides an image processing method, device and electronic device, which are used to solve one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides an image processing method, the method comprising:

[0006] Obtain a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the number of model steps of the first model is M*N, the number of model steps of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0;

[0007] Processing the first text according to the first model, determining a first predicted image generated by each step of the first model, and obtaining M*N first predicted images;

[0008] Processing the first text according to the text-to-image model, determining a second predicted image generated at each step of the text-to-image model, and obtaining N second predicted images;

[0009] The model parameters of the text-to-image model are updated according to the M*N first predicted images and the N second predicted images.

[0010] In a second aspect, the present disclosure provides an image processing device, the image processing device comprising an acquisition module, a first determination module, a second determination module and an update module, wherein:

[0011] The acquisition module is used to acquire a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the number of model steps of the first model is M*N, the number of model steps of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0;

[0012] The first determination module is used to process the first text according to the first model, determine the first prediction image generated by each step of the first model, and obtain M*N first prediction images;

[0013] The second determination module is used to process the first text according to the text-to-image model, determine the second predicted image generated by each step of the text-to-image model, and obtain N second predicted images;

[0014] The updating module is used to update the model parameters of the text-to-image model according to the M*N first predicted images and the N second predicted images.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image processing method as described in the first aspect and various possible aspects of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0019] The present disclosure provides an image processing method, device and electronic device, wherein the electronic device can obtain a first model for converting text into an image, a text-to-image model to be trained and a first text, wherein the model step number of the first model is M*N, the model step number of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0, the first text is processed according to the first model, the first prediction image generated by each step of the first model is determined, and M*N first prediction images are obtained, the first text is processed according to the text-to-image model, the second prediction image generated by each step of the text-to-image model is determined, and N second prediction images are obtained, and the model parameters of the text-to-image model are updated according to the M*N first prediction images and the N second prediction images. In the above method, since the model step number of the text-to-image model is less than the model step number of the first model, and the electronic device can supervise the output of each step of the text-to-image model based on the output results of multiple steps of the first model, the text-to-image model can learn the ability of the first model, and the number of steps of the text-to-image model is small, and the time required for model inference is small, thereby improving the processing efficiency of the text-to-image model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0022] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of model steps provided in an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of a process for generating a first predicted image provided by an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a process for generating a second predicted image provided by an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of a process for determining a target prediction image provided by an embodiment of the present disclosure;

[0027] Figure 7A schematic diagram of a training process of a text-to-image model provided in an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of a method for obtaining a first model provided in an embodiment of the present disclosure;

[0029] Fig. 9 A schematic diagram of a model relationship provided for an embodiment of the present disclosure;

[0030] Fig.10 A schematic diagram of a distillation training process of an initial model provided in an embodiment of the present disclosure;

[0031] Fig.11 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0032] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0035] Electronic device: a device with wireless transceiver function. Electronic devices can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted; they can also be deployed on the water (such as ships, etc.). The electronic device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a vehicle-mounted electronic device, a wireless terminal in self-driving, a wireless electronic device in remote medical, a wireless electronic device in smart grid, a wireless electronic device in transportation safety, a wireless electronic device in smart city, a wireless electronic device in smart home, a wearable electronic device, etc. The electronic device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access electronic device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote electronic device, a mobile device, a UE electronic device, a wireless communication device, a UE agent or a UE device, etc. Electronic devices may also be fixed or mobile.

[0036] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0037] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including an electronic device. The display page of the electronic device includes a text input box and a jump control. The user can input the text "draw an image of the sun" into the text input box and click the jump control. The electronic device can generate and display an image based on the text content input by the user, wherein the image may include an image of the sun. In this way, the electronic device can generate an image corresponding to the text based on any text input by the user, thereby improving the user experience.

[0038] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0039] In the related art, an electronic device can process text based on a text-to-image model to obtain an image corresponding to the text. The electronic device can train the text-to-image model based on multiple sets of samples, and after the text-to-image model training is completed, the function of generating an image based on the text can be realized. For example, each set of samples can include a text and an image corresponding to the text. The text-to-image model can be trained based on multiple sets of samples. In order to improve the accuracy of text-to-image conversion, the model steps of the text-to-image model are relatively large, so that the ability of text-to-image conversion can be accurately learned during the training of the text-to-image model. However, the model steps of the text-to-image model are relatively large, and the time required for the text-to-image model in the reasoning process is relatively long. For example, if the model steps of the text-to-image model are 50 steps, after the text-to-image model receives a text, it processes the text 50 times to obtain the image corresponding to the text. In this way, the processing efficiency of the text-to-image model is relatively low.

[0040] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides an image processing method, an electronic device can obtain a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the model step number of the first model is M*N, and the model step number of the text-to-image model is N. The electronic device can process the first text according to the first model to determine the first prediction image generated by each step of the first model, and process the first text according to the text-to-image model to determine the second prediction image generated by each step of the text-to-image model. The electronic device can determine N target prediction images from the M*N first prediction images, and update the model parameters of the text-to-image model according to the N target prediction images and the N second prediction images. In this way, since the model step number of the text-to-image model is less than the model step number of the first model, and the electronic device can supervise the output of the text-to-image model N steps based on the N results outputted in the first model N steps, after the training of the text-to-image model is completed, the model step number is small, and the time required for the text-to-image model to perform model inference is small, thereby improving the processing efficiency of the text-to-image model.

[0041] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0042] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0043] S201, obtaining a first model for converting text into an image, a text-to-image model to be trained, and a first text.

[0044] The execution subject of the embodiments of the present disclosure may be an electronic device, or an image processing device in the electronic device. The image processing device may be implemented according to software, or may be implemented according to a combination of software and hardware, which is not limited in the embodiments of the present disclosure. Optionally, the electronic device may be any device with end-to-end computing capabilities, which is not limited in the embodiments of the present disclosure.

[0045] The first model may be a trained text-to-image model. For example, the first model may be a text-to-image model that has been trained. Therefore, the first model may generate an image corresponding to any text based on the text.

[0046] Optionally, the model step number of the first model may be M*N, where M is an integer greater than 1 and N is an integer greater than 0. Optionally, the model step number may be the model training step number step. For example, if the first model processes the text 100 times to obtain an image corresponding to the text, the model step number of the first model may be 100 steps; if the first model processes the text 1000 times to obtain an image of the text image, the model step number of the first model may be 1000 steps.

[0047] Optionally, the text-to-image model may be a model to be trained. The number of model steps of the text-to-image model may be N. For example, if the number of model steps of the first model is 1000, the number of model steps of the text-to-image model may be 50; if the number of model steps of the first model is 40, the number of model steps of the text-to-image model may be 20.

[0048] It should be noted that the model step number of the text-to-image model is smaller than the model step number of the first model. For example, the model step number of the first model may be 2 times, 3 times, etc., of the text-to-image model step number. This is not limited in the embodiments of the present disclosure.

[0049] It should be noted that the initial model parameters in the text-to-image model may be initialized parameters or model parameters determined by any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0050] Next, combine Figure 3 , explain the model steps.

[0051] Figure 3 A schematic diagram of the number of model steps provided by the embodiment of the present disclosure. Figure 3, including: model. The model includes convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4 and convolution layer 5. Since the model can perform 5 convolution processes on the text to obtain the image corresponding to the text (that is, convolution layer 5 can output the image corresponding to the text) when processing the text, the model step number of the model can be 5 steps.

[0052] Optionally, the first text may be text describing an image. For example, the first text may be text “draw a sun”, text “draw a flower”, text “draw an image, the image may include the sun, grass and flowers”, etc., which is not limited in the embodiments of the present disclosure.

[0053] Optionally, the electronic device may obtain the first model based on a distillation training method. For example, the electronic device may perform distillation training on an initial model of the first model based on a trained text-to-image model to obtain the first model.

[0054] Optionally, the electronic device may also train the initial model of the first model based on multiple sets of training samples, and obtain the first model at the end of the training. For example, a set of training samples may include sample text and sample images corresponding to the sample text, and the initial model of the first model may be trained based on multiple sets of training samples to obtain the first model. For example, for any set of training samples, the initial model of the first model processes the sample text to obtain a predicted image corresponding to the sample text, and trains the initial model of the first model based on the loss between the predicted image corresponding to the sample text and the sample image corresponding to the sample text.

[0055] Optionally, the electronic device may also obtain the first model based on any other feasible implementation method. For example, the first model is a trained model, and the electronic device directly obtains it from a database, etc. This is not limited to the embodiments of the present disclosure.

[0056] S202: Process the first text according to the first model, determine the first predicted image generated by each step of the first model, and obtain M*N first predicted images.

[0057] The first predicted image may be an image generated by each step of the first model. For example, if the number of model steps of the first model is 100, the first model processes the first text to generate an image corresponding to the first text and generates 100 images. If the number of model steps of the first model is 1000, the first model processes the first text to generate an image corresponding to the first text and generates 1000 images.

[0058] It should be noted that, in the actual application process, after the first model processes the first text, it will eventually output an image corresponding to the first text, and the other images are images generated by the intermediate process of generating the image. For example, if the number of model steps of the first model is 100, then after the first model processes the first text, it can generate 100 images, the first 99 images are all intermediate images, and the last image is the image corresponding to the first text.

[0059] Next, combine Figure 4 , the process of generating M*N first prediction images by the first model is explained.

[0060] Figure 4 A schematic diagram of a process for generating a first predicted image provided by an embodiment of the present disclosure. Figure 4 , including a first model and a first text. The first model includes a convolution layer A, a convolution layer B and a convolution layer C, and the number of model steps of the first model is 3. Electronic device ( Figure 4 (not shown) The first text can be input into the convolution layer A of the first model, the convolution layer A can process the first text to obtain the first predicted image 1, and the first predicted image 1 can be input into the convolution layer B, the convolution layer B can process the first predicted image 1 to obtain the first predicted image 2, and the first predicted image 2 can be input into the convolution layer C, the convolution layer C can process the first predicted image 2 to obtain the first predicted image 3, and then three first predicted images can be obtained.

[0061] Among them, Figure 4 In the illustrated embodiment, the first predicted image 3 may be an image corresponding to the first text, and the first predicted image 1 and the first predicted image 2 may both be intermediate images for generating the first predicted image 3, that is, the first predicted image 1 and the first predicted image 2 may be intermediate products of internal processing of the first model, and the first predicted image 3 may be a result output by the first model.

[0062] S203: Process the first text according to the text-to-image model, determine the second predicted image generated by each step of the text-to-image model, and obtain N second predicted images.

[0063] The second predicted image may be an image generated at each step of the text-to-image model. For example, if the number of model steps of the text-to-image model is 10, the text-to-image model processes the first text to generate an image corresponding to the first text and can generate 10 second predicted images. If the number of model steps of the text-to-image model is 100, the text-to-image model processes the first text to generate an image corresponding to the first text and can generate 1,000 second predicted images.

[0064] It should be noted that in the actual application process, after the text-to-image model processes the first text, it will eventually output an image corresponding to the first text, and the other images are images generated by the intermediate process of generating the image. For example, if the number of model steps of the text-to-image model is 10, then after the text-to-image model processes the first text, it can generate 10 images, the first 9 images are all intermediate images, and the last image is the image corresponding to the first text.

[0065] Next, combine Figure 5 , the process of generating N second prediction images by the text-to-image model is described.

[0066] Figure 5 A schematic diagram of a process for generating a second predicted image provided by an embodiment of the present disclosure. Figure 5 , including a text-to-image model and a first text. The text-to-image model includes a convolutional layer A and a convolutional layer B, and the model step number of the text-to-image model is 2. Electronic device ( Figure 5 (not shown) The first text can be input into the convolution layer A of the text-to-image model, and the convolution layer A can process the first text to obtain the second predicted image 1, and the second predicted image 1 can be input into the convolution layer B, and the convolution layer B can process the second predicted image 1 to obtain the second predicted image 2.

[0067] Among them, Figure 5 In the illustrated embodiment, the second predicted image 2 may be an image corresponding to the first text, and the second predicted image 1 may be an intermediate image for generating the second predicted image 2, that is, the second predicted image 1 may be an intermediate product of internal processing of the text-to-image model, and the second predicted image 2 may be a result output by the text-to-image model.

[0068] S204: Update model parameters of the text-to-image model according to the M*N first prediction images and the N second prediction images.

[0069] Among them, the electronic device can update the model parameters of the text-to-image model according to the following feasible implementation method: determine N target prediction images among M*N first prediction images, and update the model parameters of the text-to-image model according to the N target prediction images and N second prediction images.

[0070] Among them, since the number of model steps of the first model is greater than the number of model steps of the text-to-image model, when the first model performs supervised training on the text-to-image model, the electronic device can determine N target images from the M*N first prediction images generated by the first model, and perform supervised training on the output results of the N steps of the text-to-image model based on the N target images. In this way, the text-to-image model can quickly learn the image-to-text conversion capability of the first model, thereby improving the processing efficiency and accuracy of the text-to-image model.

[0071] Optionally, the electronic device determines N target prediction images from M*N first prediction images. Specifically, it may be: among the M first prediction images, sequentially obtain the first prediction images output by the M*Lth step of the first model to obtain N first prediction images, and determine the N first prediction images as N target prediction images.

[0072] Wherein, L is 1, 2, ..., N in sequence. For example, if N is 3 and M is 2, six first prediction images can be generated in the process of the first model processing the first text, and the electronic device can determine the first prediction image generated by the first model in step 2, the first prediction image generated in step 4, and the first prediction image generated in step 6 as three target prediction images. Since the number of model steps of the text-to-image model is 3, the electronic device can supervise the results of the three-step output of the text-to-image model based on the three target prediction images, thereby improving the training efficiency and training accuracy of the text-to-image model.

[0073] It should be noted that the electronic device may also determine N target prediction images from M first prediction images based on any other feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0074] Next, combine Figure 6 , the process of determining the target prediction image among M*N first prediction images is explained.

[0075] Figure 6 A schematic diagram of a process for determining a target prediction image provided by an embodiment of the present disclosure. Figure 6 , including a first predicted image set. The images in the first predicted image set may be images generated at each step of the first model, and the first predicted image set may include images of step 1, images of step 2, ..., images of step 2N, that is, the number of model steps of the first model is 2N, and the first predicted image set may include 2N images.

[0076] See also Figure 6, the electronic device can obtain a target prediction image from the first prediction image set every 1 step to obtain a target prediction image set. The target prediction image set includes the image of step 2, the image of step 4, ..., the image of step 2N, that is, the target prediction image set can include N images.

[0077] It should be noted that Figure 6 The first prediction image set and the target prediction image set in the illustrated embodiment are only for the convenience of explaining the process of determining the target prediction image. In actual application, the electronic device can determine the target prediction image from the first prediction image output at each step of the first model.

[0078] Among them, the electronic device updates the model parameters of the text-to-image model according to the N target prediction images and the N second prediction images. Specifically, it can be: according to the i-th target prediction image and the second prediction image outputted in the i-th step of the text-to-image model, the i-th target loss is determined to obtain N target losses, and according to the N target losses, the model parameters of the text-to-image model are updated.

[0079] Among them, i is 1, 2, ..., N in sequence. For example, since the number of target prediction images is N and the number of second prediction images is N, i is 1, 2, ..., N in sequence. Therefore, the electronic device can obtain N target losses according to the target prediction image and the second prediction image. For example, for the i-th target prediction image, the electronic device can determine the difference between the i-th target prediction image and the second prediction image outputted in the i-th step of the text-to-image model as the target loss.

[0080] Optionally, the electronic device updates the model parameters of the text-to-image model according to N target losses. Specifically, for any target loss, a target number of steps is determined in the text-to-image model, and the model parameters corresponding to the target number of steps are updated according to the target loss.

[0081] The second predicted image outputted by the target number of steps can determine the target loss. For example, for any target loss, if the target loss is determined based on the second predicted image outputted at step 3 of the text-to-image model, then the target number of steps corresponding to the target loss is step 3 of the text-to-image model; if the target loss is determined based on the second predicted image outputted at step 10 of the text-to-image model, then the target number of steps corresponding to the target loss is step 10 of the text-to-image model.

[0082] Optionally, the electronic device may update the model parameters of the target number of steps corresponding to the target loss according to the target loss. For example, the number of model steps of the first model is 40 steps, and the number of model steps of the text-to-image model is 20 steps. The electronic device can determine the first target loss based on the first predicted image output at the second step of the first model and the second predicted image output at the first step of the text-to-image model, and update the model parameters of the first step of the text-to-image model based on the first target loss. The electronic device can determine the second target loss based on the first predicted image output at the fourth step of the first model and the second predicted image output at the second step of the text-to-image model, and update the model parameters of the second step of the text-to-image model based on the second target loss. ... The electronic device can determine the 20th target loss based on the first predicted image output at the 40th step of the first model and the second predicted image output at the 20th step of the text-to-image model, and update the model parameters of the 20th step of the text-to-image model based on the 20th target loss. In this way, through repeated training, the text-to-image model can achieve the effect of 40 steps of the first model through 20 steps, thereby improving the processing efficiency of the text-to-image model.

[0083] Next, combine Figure 7 , the training process of the text-to-image model is explained.

[0084] Figure 7 A diagram of the training process of a text-to-image model provided in an embodiment of the present disclosure. Figure 7 , including a first model, a text-to-image model and a first text. The first model may include convolutional layer a, convolutional layer b, convolutional layer c, convolutional layer d, convolutional layer e and convolutional layer f, and the text-to-image model may include convolutional layer A, convolutional layer B and convolutional layer C. Electronic device ( Figure 7 (not shown) a first text may be input to the first model and the text-to-image model.

[0085] See also Figure 7 After the convolution layer a in the first model performs convolution processing on the first text, it can send the convolution result to the convolution layer b. The convolution layer b performs convolution processing on the convolution result to obtain the first predicted image 2, and sends the first predicted image 2 to the convolution layer c. After the convolution layer c receives the first predicted image 2, it can perform convolution processing on the first predicted image 2 and send the convolution result to the convolution layer d. After the convolution layer d receives the convolution result, it can perform convolution processing on the convolution result to obtain the first predicted image 4 and send the first predicted image 4 to the convolution layer e. After the convolution layer e performs convolution processing on the first predicted image 4, it obtains the convolution result and sends the convolution result to the convolution layer f. After the convolution layer f processes the convolution result, it can obtain the first predicted image 6 corresponding to the first text.

[0086] See also Figure 7 After the convolution layer A of the text-to-image model performs convolution processing on the first text, the second predicted image 1 can be obtained. The convolution layer A can send the second predicted image 1 to the convolution layer B. After the convolution layer B receives the second predicted image 1, it can perform convolution processing on the second predicted image 1 to obtain the second predicted image 2, and send the second predicted image 2 to the convolution layer C. After the convolution layer C receives the second predicted image 2, it can perform convolution processing on the second predicted image 2 to obtain the second predicted image 3 corresponding to the first text.

[0087] See also Figure 7 The electronic device may determine a first loss according to a difference between the first predicted image 2 and the second predicted image 1, determine a second loss according to a difference between the first predicted image 4 and the second predicted image 2, and determine a third loss according to a difference between the first predicted image 6 and the second predicted image 3. The electronic device may update the parameters of the convolutional layer A (the target number of steps corresponding to the first loss) in the text-to-image model according to the first loss, update the parameters of the convolutional layer B (the target number of steps corresponding to the second loss) in the text-to-image model according to the second loss, and update the parameters of the convolutional layer C (the target number of steps corresponding to the third loss) in the text-to-image model according to the third loss.

[0088] In this way, after multiple iterative training, the convolutional layer A in the text-to-image model can have the function of the combination of the convolutional layer a and the convolutional layer b in the first model, the convolutional layer B in the text-to-image model can have the function of the combination of the convolutional layer c and the convolutional layer d in the first model, and the convolutional layer C in the text-to-image model can have the function of the combination of the convolutional layer e and the convolutional layer f in the first model. The text-to-image model can realize the function of converting text to image with a smaller number of model steps, thereby improving the efficiency of converting text to image.

[0089] An embodiment of the present disclosure provides an image processing method. An electronic device can obtain a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the number of model steps of the first model is M*N, and the number of model steps of the text-to-image model is N. The electronic device can process the first text according to the first model to determine a first predicted image generated by each step of the first model, and process the first text according to the text-to-image model to determine a second predicted image generated by each step of the text-to-image model. The electronic device can sequentially obtain the first predicted image output by the M*Lth (L is 1, 2, ..., N) step of the first model from the M*N first predicted images to obtain N first predicted images, and update the model parameters of the text-to-image model according to the N first predicted images and the N second predicted images. In this way, since the number of model steps of the text-to-image model is smaller than the number of model steps of the first model, and the electronic device can perform supervised training on the output of the N-step text-to-image model based on the N results of the N-step output in the first model, the text-to-image model can realize the function of text-to-image conversion based on a smaller number of model steps, reduce the time required for the text-to-image model to perform inference, and thereby improve the processing efficiency of the text-to-image model.

[0090] exist Figure 2 Based on the embodiment shown below, combined with Figure 8 , a method for obtaining a first model for converting text into an image in the above-mentioned image processing method is introduced in detail.

[0091] Figure 8 A schematic diagram of a method for obtaining a first model provided by an embodiment of the present disclosure. Figure 8 The method flow may include:

[0092] S801: Acquire a second model, an initial model of the first model, and a plurality of second texts.

[0093] The second model is used to convert text into an image, and the number of model steps of the second model is M*N. For example, the second model may be a trained model that can generate an image corresponding to the text based on the text, and the number of model steps of the second model is the same as the number of model steps of the first model.

[0094] Optionally, the number of channels of the second model is greater than the number of channels of the first model. For example, the model parameter amount of the second model is greater than the model parameter amount of the first model. Optionally, the electronic device may obtain an initialized first model based on the second model. For example, the electronic device may halve the number of channels of the second model to obtain the initialized first model, wherein, since the number of channels of the initialized first model is less than the number of channels of the second model, the model parameter amount of the initialized first model is less than the model parameter amount of the second model.

[0095] Next, combine Fig. 9 , the relationship between the first model, the second model, and the text-to-image model is explained.

[0096] Fig. 9 A schematic diagram of a model relationship provided in an embodiment of the present disclosure. Fig. 9 , including a first model, a second model and a text-to-image model. Among them, the second model may include convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5 and convolutional layer 6. The first model may include convolutional layer a, convolutional layer b, convolutional layer c, convolutional layer d, convolutional layer e and convolutional layer f. The text-to-image model may include convolutional layer A, convolutional layer B and convolutional layer C.

[0097] See also Fig. 9 Although the number of convolutional layers between the first model and the second model is the same, the number of channels of each convolutional layer in the first model is less than that of each convolutional layer in the second model, that is, the model parameter amount of the first model is less than that of the second model. The number of channels of each convolutional layer in the text-to-image model is the same as that of each convolutional layer in the first model, but the number of convolutional layers of the text-to-image model is less than that of the first model, so that the processing efficiency of the text-to-image model is higher.

[0098] Optionally, the second text can be any text describing an image. For example, the second text can be "draw a sun", "draw a moon", etc., which is not limited in the embodiments of the present disclosure. It should be noted that the first text and the second text in the embodiments of the present disclosure can be obtained from a large-scale image and text dataset, which includes a large number of text-image pairs. In the process of model training, not only the first text and the second text can be obtained, but also the image corresponding to the first text (which can be used as a true value label) and the image corresponding to the second text (which can be used as a true value label) can be added to further improve the training accuracy of the model.

[0099] Optionally, the electronic device can train the second model based on multiple groups of training samples. For example, when generating the second model, the electronic device can obtain multiple groups of training samples and train the initial second model. After the initial second model training is completed, the second model can be obtained. For example, each group of training samples can include sample text and sample images corresponding to the sample text (which can be obtained in a large-scale image and text data set), and the sample text is processed based on the initial second model to obtain a sample prediction image, and a loss function is constructed based on the difference between the sample prediction image and the sample image corresponding to the sample text, and the initial second model is trained based on the loss function. After multiple iterations, a second model can be obtained, and the second model can convert text into an image.

[0100] S802: Train the initial model according to the second model and the plurality of second texts to obtain a first model.

[0101] Among them, the electronic device trains the initial model according to the second model and multiple second texts, which can be specifically: for any second text, the second text is processed according to the second model to obtain the first target image and the second target image generated by the second model in the P step, the second text is processed according to the initial model to obtain the third target image and the fourth target image generated by the initial model in the P step, and the initial model is distilled and trained according to the first target image, the second target image, the third target image and the fourth target image.

[0102] Wherein, P can be 1, 2, ..., M*N. For example, if M*N is 50, the second target image can be the image generated by the second model in step 2, the image generated in step 10, or the image generated in step 50, etc., which is not limited in the embodiments of the present disclosure.

[0103] Optionally, the first target image is an image corresponding to the second text output by the second model. For example, since the second model is a trained model, the second model can obtain an image corresponding to the second text after processing the second text. For example, if the second text is "draw a sun", the first target image can be an image of the sun.

[0104] Optionally, the second target image may be an image generated by the second model in step P. For example, if P is 2, the second model image may be an image generated by the second model in step 2, wherein, in the field of text-to-image conversion, if the number of steps of the second model is 1000, the image generated by the second model in step 2 has a lower definition, and the image has a lower matching degree with the second text, and the image generated by the model in step 900 has a higher definition, and the image has a higher matching degree with the second text.

[0105] Optionally, the third target image may be an image corresponding to the second text output by the initial model. For example, since the initial model is a model to be trained, the accuracy of the third target image obtained after the initial model processes the second text is low. The accuracy of the initial model can be improved by training the initial model.

[0106] Optionally, the fourth target image may be an image generated by the initial model at step P. For example, since the second model may guide the initial model to be trained, if the second target image is an image generated by the second model at step 10, the fourth target image may be an image generated by the initial model at step 10; if the second target image is an image generated by the second model at step 100, the fourth target image may be an image generated by the initial model at step 100.

[0107] Optionally, the electronic device performs distillation training on the initial model according to the first target image, the second target image, the third target image and the fourth target image. Specifically, the first loss may be determined according to the first target image and the third target image, the second loss may be determined according to the second target image and the fourth target image, and the initial model may be distilled according to the first loss and the second loss.

[0108] For example, based on the first target image and the third target image, the electronic device can determine the difference between the third target image finally output by the initial model and the true value (the first target image output by the second model can be the true value). Based on the second target image and the fourth target image, the electronic device can determine the difference between the fourth target image output by the initial model in the Pth step and the true value (the second target image output by the second model in the Pth step can be true). In this way, the initial model can learn the text-to-image capability of the second model during the distillation training process, thereby reducing the size of the first model.

[0109] It should be noted that, in the process of performing distillation training on the initial model of the first model, the true value may also be the image corresponding to the second text, and the embodiments of the present disclosure do not limit this. For example, when the electronic device obtains the second text, it may also obtain the image corresponding to the second text. In the process of distillation training, a loss is determined based on the result output by the initial model and the image corresponding to the second text, and another loss is determined based on the result output by the second model in the Pth step and the result output by the initial model in the Pth step, and then the initial model is distilled and trained according to the two losses.

[0110] It should be noted that P can be selected randomly or in sequence (e.g., 1, 2, ..., M*N are selected in sequence). The embodiments of the present disclosure are not limited to this. In actual application, the model parameters of each step of the first model can be trained through a large number of training samples.

[0111] Next, combine Fig.10 , the process of distillation training the initial model by the electronic device is explained.

[0112] Fig.10 A schematic diagram of the distillation training process of an initial model provided in an embodiment of the present disclosure. Fig.10 , including: a second model, an initial model of the first model, and a second text. The second model may include convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, and convolution layer 6, and the initial model of the first model may include convolution layer a, convolution layer b, convolution layer c, convolution layer d, convolution layer e, and convolution layer f, wherein the initial model of the first model and the second model have the same number of model steps, but the number of channels of the initial model of the first model is less than the number of channels of the second model, that is, the model parameter amount of the initial model of the first model is less than the model parameter amount of the second model.

[0113] See also Fig.10 , electronic equipment ( Fig.10 (not shown) a second text can be input into the second model. After convolution layer 1 receives the second text, it can perform convolution processing on the second text and send the convolution result to convolution layer 2. Convolution layer 2 performs convolution processing on the convolution result to obtain the convolution result of convolution layer 2 and sends the convolution result of convolution layer 2 to convolution layer 3. Convolution layer 3 can process the convolution result of convolution layer 2 to obtain a second target image and send the second target image to convolution layer 4. Convolution layer 4 performs convolution processing on the second target image and sends the convolution result of convolution layer 4 to convolution layer 5. Convolution layer 5 processes the convolution result of convolution layer 4 to obtain the convolution result of convolution layer 5 and sends the convolution result of convolution layer 5 to convolution layer 6. Convolution layer 6 can process the convolution result of convolution layer 5 to obtain the first target image.

[0114] See also Fig.10 The electronic device can input a second text into the initial model. After receiving the second text, the convolution layer a can perform convolution processing on the second text and send the convolution result to the convolution layer b. The convolution layer b performs convolution processing on the convolution result to obtain the convolution result of the convolution layer b and sends the convolution result of the convolution layer b to the convolution layer c. The convolution layer c can process the convolution result of the convolution layer b to obtain a fourth target image and send the fourth target image to the convolution layer d. The convolution layer d performs convolution processing on the fourth target image and sends the convolution result of the convolution layer d to the convolution layer e. The convolution layer e processes the convolution result of the convolution layer d to obtain the convolution result of the convolution layer e and sends the convolution result of the convolution layer e to the convolution layer f. The convolution layer f can process the convolution result of the convolution layer e to obtain a third target image.

[0115] See also Fig.10 The electronic device can determine the first loss according to the difference between the first target image and the third target image, determine the second loss according to the difference between the second target image and the fourth target image, and then update the model parameters of the initial model according to the first loss, and update the model parameters of the convolution layer c in the initial model according to the second loss. In this way, through multiple iterative training, each convolution layer in the first model can learn the ability of the convolution layer in the second model, and since the first model has a small number of channels, it can not only ensure the text-to-image capability of the first model, but also reduce the volume of the first model.

[0116] The disclosed embodiment provides a method for obtaining a first model, where the electronic device can obtain a second model, an initial model of the first model, and multiple second texts, and for any second text, the second text is processed according to the second model to obtain a first target image and a second target image generated by the second model in step P, and the second text is processed according to the initial model to obtain a third target image and a fourth target image generated by the initial model in step P, and the initial model is distilled and trained according to the first target image, the second target image, the third target image, and the fourth target image. In this way, through iterative training of the first model with multiple second texts, the convolution layer in the first model can learn the function of the convolution layer of the second model, and then realize the function of converting text to image through a smaller first model.

[0117] Fig.11 This is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. Fig.11 The image processing device 110 includes an acquisition module 111, a first determination module 112, a second determination module 113 and an update module 114, wherein:

[0118] The acquisition module 111 is used to acquire a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the model step number of the first model is M*N, the model step number of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0;

[0119] The first determination module 112 is used to process the first text according to the first model, determine the first prediction image generated by each step of the first model, and obtain M*N first prediction images;

[0120] The second determination module 113 is used to process the first text according to the text-to-image model, determine the second predicted image generated by each step of the text-to-image model, and obtain N second predicted images;

[0121] The updating module 114 is used to update the model parameters of the text-to-image model according to the M*N first predicted images and the N second predicted images.

[0122] According to one or more embodiments of the present disclosure, the update module 114 is specifically used to:

[0123] Determining N target prediction images from the M*N first prediction images;

[0124] The model parameters of the text-to-image model are updated according to the N target predicted images and the N second predicted images.

[0125] According to one or more embodiments of the present disclosure, the update module 114 is specifically used to:

[0126] Sequentially obtain the first prediction images output by the M*Lth step of the first model from the M*N first prediction images to obtain N first prediction images, where L is 1, 2, ..., N in sequence;

[0127] The N first prediction images are determined as the N target prediction images.

[0128] According to one or more embodiments of the present disclosure, the update module 114 is specifically used to:

[0129] Determine the i-th target loss according to the i-th target prediction image and the second prediction image outputted in the i-th step of the text-to-image model, and obtain N target losses, where i is 1, 2, ..., N in sequence;

[0130] According to the N target losses, model parameters of the text-to-image model are updated.

[0131] According to one or more embodiments of the present disclosure, the update module 114 is specifically used to:

[0132] For any target loss, determining a target number of steps in the text-to-image model, wherein the target loss is determined based on a second predicted image outputted by the target number of steps;

[0133] According to the target loss, the model parameters corresponding to the target number of steps are updated.

[0134] According to one or more embodiments of the present disclosure, the acquisition module 111 is specifically used for:

[0135] Acquire a second model, an initial model of the first model, and a plurality of second texts, wherein the second model is used to convert the texts into images, the number of model steps of the second model is M*N, and the number of channels of the second model is greater than the number of channels of the first model;

[0136] The initial model is trained according to the second model and the plurality of second texts to obtain the first model.

[0137] According to one or more embodiments of the present disclosure, the acquisition module 111 is specifically used for:

[0138] For any second text, the second text is processed according to the second model to obtain a first target image and a second target image generated by the second model in step P, where P is 1, 2, ..., M*N;

[0139] According to the initial model, the second text is processed to obtain a third target image and a fourth target image generated by the initial model in step P;

[0140] The initial model is subjected to distillation training according to the first target image, the second target image, the third target image and the fourth target image.

[0141] The image processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0142] Fig.12 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.12 , which shows a structural schematic diagram of an electronic device 1200 suitable for implementing an embodiment of the present disclosure. The electronic device 1200 can be any device with end-to-end computing capabilities. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0143] like Fig.12 As shown, the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 to a random access memory (RAM) 1203. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0144] Typically, the following devices may be connected to the I / O interface 1205: input devices 1206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1209. The communication device 1209 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 1200 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0145] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1209, or installed from a storage device 1208, or installed from a ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0146] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0147] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0148] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0149] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0150] An embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and various image processing methods that may be involved in the first aspect.

[0151] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0154] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0155] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0157] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0158] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0159] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as a terminal device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0160] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the terminal device.

[0161] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0162] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0163] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0164] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0165] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0166] In a first aspect, one or more embodiments of the present disclosure provide an image processing method, the method comprising:

[0167] Obtain a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the number of model steps of the first model is M*N, the number of model steps of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0;

[0168] Processing the first text according to the first model, determining a first predicted image generated by each step of the first model, and obtaining M*N first predicted images;

[0169] Processing the first text according to the text-to-image model, determining a second predicted image generated at each step of the text-to-image model, and obtaining N second predicted images;

[0170] The model parameters of the text-to-image model are updated according to the M*N first predicted images and the N second predicted images.

[0171] According to one or more embodiments of the present disclosure, updating the model parameters of the text-to-image model according to the M*N first predicted images and the N second predicted images includes:

[0172] Determining N target prediction images from the M*N first prediction images;

[0173] The model parameters of the text-to-image model are updated according to the N target prediction images and the N second prediction images.

[0174] According to one or more embodiments of the present disclosure, the determining N target prediction images from the M*N first prediction images includes:

[0175] Sequentially obtain the first prediction images output by the M*Lth step of the first model from the M*N first prediction images to obtain N first prediction images, where L is 1, 2, ..., N in sequence;

[0176] The N first prediction images are determined as the N target prediction images.

[0177] According to one or more embodiments of the present disclosure, updating the model parameters of the text-to-image model according to the N target predicted images and the N second predicted images includes:

[0178] Determine the i-th target loss according to the i-th target prediction image and the second prediction image outputted in the i-th step of the text-to-image model, and obtain N target losses, where i is 1, 2, ..., N in sequence;

[0179] The model parameters of the text-to-image model are updated according to the N target losses.

[0180] According to one or more embodiments of the present disclosure, updating the model parameters of the text-to-image model according to the N target losses includes:

[0181] For any target loss, determining a target number of steps in the text-to-image model, wherein the target loss is determined based on a second predicted image outputted by the target number of steps;

[0182] According to the target loss, the model parameters corresponding to the target number of steps are updated.

[0183] According to one or more embodiments of the present disclosure, obtaining a first model for converting text into an image includes:

[0184] Acquire a second model, an initial model of the first model, and a plurality of second texts, wherein the second model is used to convert the texts into images, the number of model steps of the second model is M*N, and the number of channels of the second model is greater than the number of channels of the first model;

[0185] The initial model is trained according to the second model and the plurality of second texts to obtain the first model.

[0186] According to one or more embodiments of the present disclosure, the training of the initial model according to the second model and the plurality of second texts includes:

[0187] For any second text, the second text is processed according to the second model to obtain a first target image and a second target image generated by the second model in step P, where P is 1, 2, ..., M*N;

[0188] According to the initial model, the second text is processed to obtain a third target image and a fourth target image generated by the initial model in step P;

[0189] The initial model is subjected to distillation training according to the first target image, the second target image, the third target image and the fourth target image.

[0190] In a second aspect, one or more embodiments of the present disclosure provide an image processing device, the image processing device comprising an acquisition module, a first determination module, a second determination module and an update module, wherein:

[0191] The acquisition module is used to acquire a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the number of model steps of the first model is M*N, the number of model steps of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0;

[0192] The first determination module is used to process the first text according to the first model, determine the first prediction image generated by each step of the first model, and obtain M*N first prediction images;

[0193] The second determination module is used to process the first text according to the text-to-image model, determine the second predicted image generated by each step of the text-to-image model, and obtain N second predicted images;

[0194] The updating module is used to update the model parameters of the text-to-image model according to the M*N first predicted images and the N second predicted images.

[0195] According to one or more embodiments of the present disclosure, the update module is specifically used to:

[0196] Determining N target prediction images from the M*N first prediction images;

[0197] The model parameters of the text-to-image model are updated according to the N target prediction images and the N second prediction images.

[0198] According to one or more embodiments of the present disclosure, the update module is specifically used to:

[0199] Sequentially obtain the first prediction images output by the M*Lth step of the first model from the M*N first prediction images to obtain N first prediction images, where L is 1, 2, ..., N in sequence;

[0200] The N first prediction images are determined as the N target prediction images.

[0201] According to one or more embodiments of the present disclosure, the update module is specifically used to:

[0202] Determine the i-th target loss according to the i-th target prediction image and the second prediction image outputted in the i-th step of the text-to-image model, and obtain N target losses, where i is 1, 2, ..., N in sequence;

[0203] The model parameters of the text-to-image model are updated according to the N target losses.

[0204] According to one or more embodiments of the present disclosure, the update module is specifically used to:

[0205] For any target loss, determining a target number of steps in the text-to-image model, wherein the target loss is determined based on a second predicted image outputted by the target number of steps;

[0206] According to the target loss, the model parameters corresponding to the target number of steps are updated.

[0207] According to one or more embodiments of the present disclosure, the acquisition module is specifically used to:

[0208] Acquire a second model, an initial model of the first model, and a plurality of second texts, wherein the second model is used to convert the texts into images, the number of model steps of the second model is M*N, and the number of channels of the second model is greater than the number of channels of the first model;

[0209] The initial model is trained according to the second model and the plurality of second texts to obtain the first model.

[0210] According to one or more embodiments of the present disclosure, the acquisition module is specifically used to:

[0211] For any second text, the second text is processed according to the second model to obtain a first target image and a second target image generated by the second model in step P, where P is 1, 2, ..., M*N;

[0212] According to the initial model, the second text is processed to obtain a third target image and a fourth target image generated by the initial model in step P;

[0213] The initial model is subjected to distillation training according to the first target image, the second target image, the third target image and the fourth target image.

[0214] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0215] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned first aspect and various image processing methods that may be involved in the first aspect.

[0216] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0217] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0218] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Obtain a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the number of model steps of the first model is M*N, the number of model steps of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0; Processing the first text according to the first model, determining a first predicted image generated by each step of the first model, and obtaining M*N first predicted images; Processing the first text according to the text-to-image model, determining a second predicted image generated at each step of the text-to-image model, and obtaining N second predicted images; The model parameters of the text-to-image model are updated according to the M*N first predicted images and the N second predicted images.

2. The method according to claim 1, characterized in that The updating of the model parameters of the text-to-image model according to the M*N first predicted images and the N second predicted images includes: Determining N target prediction images from the M*N first prediction images; The model parameters of the text-to-image model are updated according to the N target prediction images and the N second prediction images.

3. The method according to claim 2, characterized in that The determining N target prediction images from the M*N first prediction images comprises: Sequentially obtain the first prediction images output by the M*Lth step of the first model from the M*N first prediction images to obtain N first prediction images, where L is 1, 2, ..., N in sequence; The N first prediction images are determined as the N target prediction images.

4. The method according to claim 3, characterized in that The updating of the model parameters of the text-to-image model according to the N target predicted images and the N second predicted images includes: Determine the i-th target loss according to the i-th target prediction image and the second prediction image outputted in the i-th step of the text-to-image model, and obtain N target losses, where i is 1, 2, ..., N in sequence; According to the N target losses, model parameters of the text-to-image model are updated.

5. The method according to claim 4, characterized in that The updating of the model parameters of the text-to-image model according to the N target losses includes: For any target loss, determining a target number of steps in the text-to-image model, wherein the target loss is determined based on a second predicted image outputted by the target number of steps; According to the target loss, the model parameters corresponding to the target number of steps are updated.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining a first model for converting text into an image comprises: Acquire a second model, an initial model of the first model, and a plurality of second texts, wherein the second model is used to convert the texts into images, the number of model steps of the second model is M*N, and the number of channels of the second model is greater than the number of channels of the first model; The initial model is trained according to the second model and the plurality of second texts to obtain the first model.

7. The method according to claim 6, characterized in that The step of training the initial model according to the second model and the plurality of second texts comprises: For any second text, the second text is processed according to the second model to obtain a first target image and a second target image generated by the second model in step P, where P is 1, 2, ..., M*N; According to the initial model, the second text is processed to obtain a third target image and a fourth target image generated by the initial model in step P; The initial model is subjected to distillation training according to the first target image, the second target image, the third target image and the fourth target image.

8. An image processing model, characterized in that: It includes an acquisition module, a first determination module, a second determination module and an update module, wherein: The acquisition module is used to acquire a first model for converting text into an image, a text-to-image model to be trained, and a first text, wherein the number of model steps of the first model is M*N, the number of model steps of the text-to-image model is N, M is an integer greater than 1, and N is an integer greater than 0; The first determination module is used to process the first text according to the first model, determine the first prediction image generated by each step of the first model, and obtain M*N first prediction images; The second determination module is used to process the first text according to the text-to-image model, determine the second predicted image generated by each step of the text-to-image model, and obtain N second predicted images; The updating module is used to update the model parameters of the text-to-image model according to the M*N first predicted images and the N second predicted images.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the image processing method according to any one of claims 1 to 7 is implemented.

Images