Image generation method, device, equipment and medium

Abstract

The present application relates to an image generation method, device, equipment and medium, and to the field of computer technology. The image generation method includes: determining target synthetic image information based on the acquired image information to be processed and the reference image information corresponding to the image information to be processed; inputting the target synthetic image information into a pre-trained first generator to obtain the initial generated image information output by the first generator; based on the mouth area information in the initial generated image information, generating the mouth residual image information corresponding to the initial generated image information through a pre-trained second generator, wherein the second generator is a generator jointly trained with the first generator; generating the target image information corresponding to the image information to be processed based on the mouth residual image information and the initial generated image information. The present application improves the generation effect of the mouth shape while ensuring the inference speed, thereby improving the overall generation effect of the image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, device, equipment and medium. Background Art

[0002] Digital human is a kind of virtual simulation of human body at different levels of form and function using information science methods. With the rapid development of computer technology, the technology of generating digital human is becoming more and more mature.

[0003] In actual processing, if digital human generation technology is to be applied in business and realize real-time interaction, the digital human generation solution must meet at least two points: good generation effect and fast reasoning speed; among them, good digital human generation effect is a necessary prerequisite, and fast reasoning speed is a business need. A large number of generation technologies that have emerged so far are mainly based on image translation models such as pix2pix, pix2pixHD, Vid2Vid, few shot video2video, NERF, StyleGAN, etc., to realize the generation of digital human videos. Specifically, these existing image translation models mainly input real pictures or sketches of real pictures into the generator to generate digital human pictures corresponding to video and audio through the generator, so that digital human videos can be generated and output based on the digital human pictures generated by the generator.

[0004] However, the area around the mouth is the area with the largest change. The existing image translation model uses a digital human generator to generate digital human images, which has the problem of poor mouth shape generation, such as blurred mouth shape or mouth teeth, poor clarity, etc., which affects the digital human generation effect. Summary of the invention

[0005] In view of this, the present application provides an image generation method, device, equipment and medium to improve the clarity around the mouth while ensuring the inference speed, thereby improving the overall image generation effect.

[0006] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0007] Determining target composite image information based on the acquired image information to be processed and reference image information corresponding to the image information to be processed;

[0008] Inputting the target synthetic graph information into a pre-trained first generator to obtain initial generated graph information output by the first generator;

[0009] According to the mouth area information in the initial generated image information, a pre-trained second generator is used to generate mouth residual image information corresponding to the initial generated image information, wherein the second generator is a generator jointly trained with the first generator;

[0010] The target image information corresponding to the image information to be processed is generated according to the mouth residual image information and the initial generated image information.

[0011] Optionally, determining the target composite image information according to the acquired image information to be processed and reference image information corresponding to the image information to be processed includes:

[0012] Extracting the frame image information to be processed from the video to be processed, and determining the extracted frame image information to be processed as the image information to be processed;

[0013] Determine the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed;

[0014] Image synthesis is performed based on the sketch image information and the reference image information to obtain the target synthesized image information.

[0015] Optionally, the determining the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed includes:

[0016] Extracting facial key point information and contour line information from the image information to be processed, and generating sketch information of the image information to be processed based on the extracted facial key point information and contour line information

[0017] If the image information to be processed is the first frame of video image information, creating all-zero image information based on the image to be processed, and determining the all-zero image information as reference image information corresponding to the image information to be processed;

[0018] If the image information to be processed is not the first frame image information of the video, the reference frame target image information is determined as the reference image information corresponding to the image information to be processed, and the reference frame target image information is the target image information corresponding to the previous frame image of the image information to be processed.

[0019] Optionally, the generating, based on the mouth region information in the initial generated graph information, the mouth residual graph information corresponding to the initial generated graph information by a pre-trained second generator includes:

[0020] Extracting mouth area information from the initial generated image information to serve as initial generated image mouth information;

[0021] Based on the initial generated image mouth information and the image mouth information to be processed, synthesis is performed to obtain target mouth synthesis image information corresponding to the initial generated image information, wherein the image mouth information to be processed is the mouth area information in the image information to be processed.

[0022] The target mouth synthesis map information is input into a pre-trained second generator to obtain the mouth residual map information output by the second generator.

[0023] Optionally, generating target image information corresponding to the image information to be processed according to the mouth residual image information and the initial generated image information includes:

[0024] Based on the mouth residual image information, supplementing the mouth area information in the initial generated image information to obtain supplemented image information;

[0025] The image information after the supplementary processing is determined as the target image information.

[0026] Optionally, before inputting the target synthetic graph information into the pre-trained first generator, the method further includes:

[0027] Obtaining video image information to be trained from a preset training data set;

[0028] Model training is performed based on the video image information to be trained to obtain an image processing model including the first generator and the second generator.

[0029] Optionally, performing model training according to the video image information to be trained includes:

[0030] Determine the sketch image information to be trained based on the video image information to be trained, the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information;

[0031] Generate first generated graph information corresponding to the first to-be-trained sketch graph information by using a pre-built first generator;

[0032] Generate, by the second generator, first mouth residual map information corresponding to the first generated map information according to the mouth region information in the first generated map information;

[0033] generating first synthetic image information corresponding to the first video image information according to the first mouth residual image information and the first generated image information;

[0034] Merging the first synthetic image information and the second sketch image information to be trained to obtain second merged image information corresponding to the second sketch image information to be trained;

[0035] Inputting the second merged graph information into the first generator to obtain second generated graph information output by the first generator;

[0036] According to the mouth area information in the second generated image information, generating second mouth residual image information corresponding to the second generated image information by the second generator;

[0037] generating second synthetic image information corresponding to the second video image information according to the second mouth residual image information and the second generated image information;

[0038] Determine a model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual image information, and the second mouth residual image information;

[0039] If the model loss value does not meet the preset model convergence conditions, the parameters of the first generator and / or the parameters of the second generator are updated based on the model loss value, and the first generator and the second generator with updated parameters are iteratively trained until the model loss value meets the preset model convergence conditions.

[0040] Optionally, the generating, by using a pre-built first generator, first generated graph information corresponding to the first to-be-trained sketch graph information includes:

[0041] Obtaining all-zero image information corresponding to the first sketch image information to be trained;

[0042] Merging the all-zero image information and the first sketch image information to be trained to obtain first merged image information corresponding to the first sketch image information to be trained;

[0043] The first merged graph information is input into the first generator to obtain the first generated graph information output by the first generator.

[0044] Optionally, generating, by the second generator, first mouth residual map information corresponding to the first generated map information based on the mouth region information in the first generated map information, includes: merging the mouth region information in the first generated map information with the mouth region information in the first to-be-trained sketch map information, and inputting the first merged image information obtained after the merging into the second generator to obtain the first mouth residual map information;

[0045] The method of generating second mouth residual map information corresponding to the second generated map information based on the mouth area information in the second generated map information through the second generator includes: merging the mouth area information in the second generated map information and the mouth area information in the second sketch map information to be trained, and inputting the second merged image information obtained after the merging into the second generator to obtain the second mouth residual map information.

[0046] Optionally, determining the model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual image information, and the second mouth residual image information includes:

[0047] Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information into a discriminator of the image processing model to obtain first loss information output by the discriminator;

[0048] Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information into the discriminator to obtain second loss information output by the discriminator;

[0049] Inputting the mouth region information in the first sketch image information to be trained, the mouth region information in the second sketch image information to be trained, the first mouth residual image information, and the second mouth residual image information into the discriminator to obtain third loss information output by the discriminator;

[0050] Inputting the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information into the discriminator to obtain fourth loss information output by the discriminator;

[0051] Determine a target discriminator loss value and a target generator loss value based on the first loss information and the second loss information;

[0052] Determine a mouth discriminator loss value and a mouth residual generator loss value based on the third loss information and the fourth loss information;

[0053] The model loss value is determined based on the target discriminator loss value, the target generator loss value, the mouth discriminator loss value and the mouth residual generator loss value.

[0054] In a second aspect, an embodiment of the present application provides an image generating device, including:

[0055] A target composite image determination module, used to determine target composite image information based on the acquired image information to be processed and reference image information corresponding to the image information to be processed;

[0056] An initial generated graph module, used for inputting the target synthetic graph information into a pre-trained first generator to obtain the initial generated graph information output by the first generator;

[0057] A mouth residual map generation module, used to generate mouth residual map information corresponding to the initial generated map information through a pre-trained second generator according to the mouth area information in the initial generated map information, wherein the second generator is a generator jointly trained with the first generator;

[0058] The target image generation module is used to generate target image information corresponding to the image information to be processed based on the mouth residual image information and the initial generated image information.

[0059] In a third aspect, an embodiment of the present application provides an image generating device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the image generating method as described in any one of the first aspects when executing the program stored in the memory.

[0060] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image generation method as described in any one of the first aspects are implemented.

[0061] The embodiment of the present application determines the target synthetic image information based on the acquired image information to be processed and the reference image information corresponding to the image information to be processed, and obtains the initial generated image information by inputting the target synthetic image information into a pre-trained first generator, and then generates the mouth residual image information based on the mouth area information in the initial generated image information through a pre-trained second generator, so that the target image information corresponding to the image information to be processed can be generated based on the mouth residual image information and the initial generated image information, that is, the mouth area information in the initial generated image information is supplemented by the mouth residual image information, so as to improve the clarity of the mouth area in the target image information finally generated, improve the mouth shape generation effect, and further improve the overall image generation effect under the premise of ensuring the inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0064] Figure 1is a flowchart of a method for generating an image provided by an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a sketch of an image to be processed in an example of the present application;

[0066] Figure 3 A flowchart of a method for generating an image according to an optional embodiment of the present application;

[0067] Figure 4 A flowchart of a method for generating an image provided in an optional embodiment of the present application;

[0068] Figure 5 A structural block diagram of an image generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0070] Reference Figure 1 , shows a flowchart of the steps of an image generation method provided by an embodiment of the present application. Specifically, the image generation method provided by the present application can be applied to image generation processing, such as the case of generating a virtual image of a video based on a real picture; wherein the virtual image can be a digital human image in a video, and can be specifically used to represent a digital human in a digital human video. Figure 1 As shown, the image generation method in the embodiment of the present application may specifically include the following steps:

[0071] Step 110: determining target composite image information according to the acquired image information to be processed and reference image information corresponding to the image information to be processed.

[0072] Among them, the image information to be processed can be used to represent the image to be processed, such as a real picture to be processed, or a sketch of the real picture to be processed, and the embodiment of the present application does not impose specific restrictions on this. Specifically, when performing image processing, the embodiment of the present application can obtain the real picture that needs to be processed at present, such as obtaining the real picture Rt of the tth frame to be processed in the user's recorded video as the image information to be processed, and can obtain the corresponding reference image information for the currently acquired image information to be processed, so as to perform image synthesis based on the reference image information corresponding to the image information to be processed, and obtain the target synthesis image information corresponding to the image information to be processed. The reference image information corresponding to the image information to be processed can represent the reference image corresponding to the currently acquired image to be processed. It should be noted that the real picture Rt can refer to the real picture R of the tth frame to be processed in the user's recorded video, t is an integer, the real picture Rt can include the face of the target object, and the target object can refer to the object that needs to be processed by the image processing, such as a person object used to represent a person, or an animal object used to represent an animal, etc., and the embodiment of the present application does not impose specific restrictions on this.

[0073] Furthermore, after acquiring the image information to be processed, the embodiment of the present application can determine whether the image information to be processed is sketch information, so as to determine whether the real picture to be processed is a sketch. Thus, when the real picture to be processed is not a sketch, facial key point information and edge (Canny) line information can be extracted from the image information to be processed, so as to construct the sketch information of the image to be processed based on the extracted facial key point information and Canny line information. When the real picture to be processed is a sketch, the image information to be processed can be directly determined as the sketch information of the image to be processed, so as to improve the efficiency of information acquisition. The sketch information can be used to represent the sketch St of the image to be processed, such as Figure 2 It should be noted that the extracted facial key point information can be used to represent facial key points, such as the coordinate values ​​of 68 facial key points; the Canny line information can represent the Canny line in the figure, and can be specifically used to represent the extracted edge line features, such as the extracted hair and clothing features.

[0074] Step 120, input the target synthetic graph information into a pre-trained first generator to obtain initial generated graph information output by the first generator.

[0075] Specifically, after obtaining the target synthetic image information corresponding to the image information to be processed, the embodiment of the present application can input the target synthetic image information into a pre-trained first generator G1 to generate an initial image of the target object by inference through the first generator G1, and then the initial image generated by the first generator G1 can be output as the initial generated image information.

[0076] Step 130, based on the mouth area information in the initial generated image information, generate the mouth residual image information corresponding to the initial generated image information through a pre-trained second generator, wherein the second generator is a generator jointly trained with the first generator.

[0077] Specifically, after obtaining the initial generated graph information, the embodiment of the present application can extract the mouth area information from the initial generated graph information, and can use the extracted mouth area information as the initial generated mouth area information, so as to generate the mouth residual graph information of the target object through the pre-trained second generator G2 to learn the residual around the mouth based on the initial generated mouth area information, as the mouth residual graph information corresponding to the initial generated graph information. The mouth residual graph information can represent the residual image of the generated mouth, and can be specifically used to improve the clarity of the mouth generation.

[0078] Step 140: Generate target image information corresponding to the image information to be processed based on the mouth residual image information and the initial generated image information.

[0079] Specifically, after the embodiment of the present application generates the mouth residual map information corresponding to the initial generated map information, the mouth residual map information can be used to supplement the mouth area information in the initial generated map information to generate the final picture information as the target image information corresponding to the image information to be processed. The mouth area information in the initial generated map information can represent the mouth area of ​​the target object in the initial image generated by the first generator G1; the target image information can be used to represent the virtual object image generated by supplementing the initial generated map information output by the first generator with the mouth residual map information generated by the second generator G2, such as the digital human image in the digital human video, etc., and the present application does not impose specific restrictions on this.

[0080] It can be seen that after determining that the target synthetic image information corresponds to the image information to be processed, the embodiment of the present application obtains the initial generated image information by inputting the target synthetic image information into the pre-trained first generator, and generates the mouth residual image information based on the mouth area information in the initial generated image information through the pre-trained second generator, and then generates the target image information corresponding to the image information to be processed based on the mouth residual image information and the initial generated image information, so as to supplement the mouth area information in the initial generated image information with the mouth residual map information, thereby improving the clarity of the mouth area in the target image information finally generated, improving the mouth shape generation effect, and thereby improving the overall image generation effect while ensuring the reasoning speed.

[0081] In a specific implementation, the embodiment of the present application can pre-use the image information to be trained in the training data set for model training to train an image processing model including the above-mentioned first generator and the second generator. Among them, the first generator can be used to generate the overall image of the target object, such as the overall structure diagram of the digital human; the second generator can be used to learn the mouth area and generate the mouth residual map information, such as the residual around the mouth can be learned to generate the mouth residual picture to make up for the problem that the first generator generates the mouth poorly; the training data set can refer to a collection of data used for model training, which can specifically include one or more image data, such as a continuous multi-frame video image data for training in a video; the image information to be trained can refer to the image data to be trained, such as the video image information to be trained, and the video image information to be trained can refer to the video image to be trained. Further, based on the above embodiment, the image generation method provided in the embodiment of the present application also includes: obtaining the video image information to be trained from the preset training data set before inputting the target synthetic image information into the pre-trained first generator; model training is performed based on the video image information to be trained to obtain the image processing model including the first generator and the second generator.

[0082] In actual processing, one or more image data can be obtained from the training data set for model training. For example, N consecutive frames of video image data of a video in the training data set can be obtained as the image information to be trained for model training, so that N consecutive frames of image data in the video can be used for model training. For example, two consecutive frames of video images in the video can be obtained from the training data set for model training, where N is an integer greater than 1. Optionally, an embodiment of the present application performs model training based on the video image information to be trained, which may specifically include: determining the sketch image information to be trained based on the video image information to be trained, the video image information to be trained comprising first video image information and second video image information, the sketch image information to be trained comprising first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information; generating first generated image information corresponding to the first sketch image information to be trained through a pre-constructed first generator; generating first mouth residual image information corresponding to the first generated image information through the second generator based on mouth area information in the first generated image information; generating first composite image information corresponding to the first video image information based on the first mouth residual image information and the first generated image information; merging the first composite image information and the second sketch image information to be trained to obtain a second merged image corresponding to the second sketch image information to be trained information; input the second merged graph information into the first generator to obtain the second generated graph information output by the first generator; based on the mouth area information in the second generated graph information, generate the second mouth residual graph information corresponding to the second generated graph information through the second generator; based on the second mouth residual graph information and the second generated graph information, generate the second synthetic image information corresponding to the second video image information; based on the first video image information, the second video image information, the first sketch graph information to be trained, the second sketch graph information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual graph information and the second mouth residual graph information, determine the model loss value; if the model loss value does not meet the preset model convergence condition, update the parameters of the first generator and / or the parameters of the second generator based on the model loss value, and iteratively train the first generator and the second generator after the updated parameters until the model loss value meets the preset model convergence condition. Among them, the first video image information can be used to represent the first frame of video image in two consecutive frames of video images extracted from the training data set; the second video image data can represent the second frame of video image in two consecutive frames of video images extracted from the training data set, that is, the second frame of video image is the previous frame of video image of the first frame of video image.

[0083] Reference Figure 3, shows a flowchart of the steps of an image generation method provided by an optional embodiment of the present application. Specifically, the image processing method provided by the embodiment of the present application may include the following steps in the model training stage:

[0084] Step 301: Obtain video image information to be trained from a preset training data set.

[0085] Step 302: determine the sketch image information to be trained based on the video image information to be trained, wherein the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information.

[0086] The first sketch information to be trained may refer to the sketch information corresponding to the first video image information, which can be used to represent the sketch of the first video image; the second sketch information to be trained may refer to the sketch information corresponding to the second video image information, which can be used to represent the sketch of the second video image. Specifically, in the model training stage, this embodiment can extract one or more frames of video image information from the training data set as training video image information; and can extract facial key point information and Canny line information for each frame of video image information extracted, that is, extract facial key point information and Canny line information from each frame of training video image information obtained, so as to generate corresponding sketch information to be trained based on the extracted facial key point information and Canny line information.

[0087] For example, after extracting the real picture Rt of the tth frame in the video in the training data set, the open source key point extraction method can be used to extract 68 key points and Canny lines of the real picture of the tth frame, that is, the facial key point information and Canny line information are extracted from the video image information to be trained, and then the extracted facial key point information and Canny line information can be used to generate the corresponding sketch image information to be trained, that is, the 68 key points are used to draw the lines of the face to combine the lines of the face with the Canny lines to generate the corresponding sketch image St; for another example, after extracting the real picture R(t+1) of the (t+1)th frame in the video in the training data set, the open source key point extraction method can be used to extract 68 key points and Canny lines of the real picture of the (t+1)th frame, and then the 68 key points can be used to draw the lines of the face to combine the lines of the face with the Canny lines to generate the corresponding sketch image S(t+1).

[0088] Step 303: Generate first generated graph information corresponding to the first sketch graph information to be trained by a pre-built first generator.

[0089] Specifically, in the model training stage, the embodiment of the present application can pre-construct a generator, such as directly using the structure of the pix2pix generator to construct a generator as the first generator G1. The first generator G1 can be used to generate an overall structural image of the target object, and the input of the first generator G1 can be the synthetic image information corresponding to the video image information to be trained, and the output of the first generator G1 can be the initial generated image information corresponding to the sketch image information to be trained. Among them, the synthetic image information can refer to the initial synthetic image obtained by performing image synthesis based on the sketch image information to be trained and the reference image information corresponding to the sketch image information to be trained. Therefore, after determining the first sketch image information to be trained, the embodiment of the present application can obtain the corresponding reference image information for the first sketch image information to be trained, and then merge based on the first sketch image information to be trained and the reference image information corresponding to the first sketch image information to be trained, and input the merged image information into the pre-constructed first generator to generate the corresponding first generated image information through the first generator.

[0090] In actual processing, an all-zero image of the same size as the real image can be created based on the real image to be used as the reference image information corresponding to the first sketch image to be trained for model training. Furthermore, the embodiment of the present application generates the first generated image information corresponding to the first sketch image information to be trained through a pre-constructed first generator, which can specifically include: obtaining the all-zero image information corresponding to the first sketch image information to be trained; merging the all-zero image information and the first sketch image information to be trained to obtain the first merged image information corresponding to the first sketch image information to be trained; inputting the first merged image information into the first generator to obtain the first generated image information output by the first generator.

[0091] As an example of the present application, in the model training stage, a generator can be first constructed to serve as the above-mentioned first generator G1. After obtaining the sketch St of the video image of the t-th frame of the video, the sketch St of the video image of the t-th frame of the video can be determined as the first sketch image information to be trained, and the corresponding all-zero image information Zt can be obtained for the sketch St of the t-th frame of the video image, so as to perform image merging according to the all-zero image information Zt, merge the sketch St of the t-th frame of the video image and the all-zero image to obtain the merged initial composite image It, and the initial composite image It can be used as the first merged image information corresponding to the first sketch image information to be trained, and input to the first generator G1, so that the first generator G1 generates the picture Ft of the t-th frame according to the initial composite image It and outputs it as the first generated image information corresponding to the first sketch image information to be trained.

[0092] Step 304: Generate first mouth residual map information corresponding to the first generated map information through the second generator according to the mouth area information in the first generated map information.

[0093] Specifically, after obtaining the first generated image information, the embodiment of the present application can extract the mouth area information from the first generated image information, so as to generate the mouth residual image information through the second generator based on the extracted mouth area information, and the mouth residual image information generated by the second generator can be determined as the first mouth residual image information corresponding to the first generated image information, so that the first mouth residual image information can be used to supplement the mouth area information in the first generated image information later, thereby improving the clarity of the mouth shape area. Further, the embodiment of the present application generates the first mouth residual image information corresponding to the first generated image information through the second generator based on the mouth area information in the first generated image information, which can specifically include: merging the mouth area information in the first generated image information and the mouth area information in the first sketch image information to be trained, and inputting the first merged image information obtained after the merger into the second generator to obtain the first mouth residual image information.

[0094] For example, in combination with the above example, when the first generated image information is a picture Ft of the t-th frame generated by the first generator G1, the mouth area information FMt can be extracted from the generated picture Ft of the t-th frame to be merged with the mouth area information SMt in the first sketch image information to be trained. For example, the mouth area information FMt can be extracted from the picture Ft according to a preset method for obtaining mouth area information, and the extracted mouth area information FMt and the mouth area information SMt in the first sketch image information to be trained can be combined. Subsequently, the first merged image information generated after the combination can be input into the second generator G2 to generate a mouth residual picture Mt of the t-th frame, and the mouth residual picture Mt of the t-th frame generated by the second generator G2 can be output as the first mouth residual image information.

[0095] It should be noted that the preset method for obtaining mouth area information can be mainly used to extract mouth area information from the extracted facial key points. Specifically, after extracting the facial key point Lt of the tth frame, the mouth area information Lt[48:68,:] can be obtained, and then based on the obtained mouth area information Lt[48:68,:], the minimum x-coordinate value Xmin and the minimum y-coordinate value Ymin of the mouth, as well as the maximum x-coordinate value Xmax and the minimum y-coordinate value Ymax of the mouth can be extracted, such as Xmin, Ymin=Min(Lt[48:68,:]), Xmax, Ymax=Max(Lt[48:68,:]); then the minimum x-coordinate value Xmin and the minimum y-coordinate value Ymin of the mouth, as well as the maximum x-coordinate value Xmax and the minimum y-coordinate value Ymax of the mouth can be used. The x-coordinate value Xcenter of the middle point of the mouth and the y-coordinate value of the middle point of the mouth are determined by the y-coordinate value Ymax, that is, Xcenter=(Xmin+Xmax) / 2, Ycenter=(Ymin+Ymax) / 2; thus, the x-coordinate value Xcenter of the middle point of the mouth and the y-coordinate value of the middle point of the mouth can be used to determine the area of ​​the mouth. For example, when the image area is 128 points, the x-coordinate value Xcenter of the middle point of the mouth and the y-coordinate value of the middle point of the mouth and the number of points in the mouth area can be used to determine the regional mouth area information, wherein the number of points in the mouth area can be half of the number of points in the image area, that is, according to the formula Xmin, Xmax, Ymin, Ymax=

[0096] Xcenter-64, Xcenter+64, Ycenter-64, Ycenter+64 are calculated to obtain the final mouth area information. For example, the mouth area information RMt of the real picture of the tth frame is

[0097] Rt[Ymin:Ymax,Xmin:Xmax,:], that is, RMt=Rt[Ymin:Ymax,Xmin:Xmax,:]; for example, the mouth area information SMt of the t-th frame sketch picture is St[Ymin:Ymax,Xmin:Xmax,:], that is, SMt=St[Ymin:Ymax,Xmin:Xmax,:]; similarly, the mouth area information IMt of the t-th frame synthetic picture is It[Ymin:Ymax,Xmin:Xmax,:], that is, IMt=It[Ymin:Ymax,Xmin:Xmax,:], etc.

[0098] Step 305: Generate first synthetic image information corresponding to the first video image information according to the first mouth residual image information and the first generated image information.

[0099] Specifically, after obtaining the first mouth residual map information, the embodiment of the present application can synthesize the first mouth residual map information with the first generated map information, so as to use the first mouth residual map information to supplement the mouth area information in the first generated map information, and update the mouth area information in the first generated map information, thereby improving the clarity of the mouth shape area, and obtaining an updated synthesized picture as the first synthesized image information corresponding to the first video image information. For example, combined with the above example, after the mouth residual picture Mt of the t-th frame is generated by the second generator G2, data can be combined according to the formula FFt[:,:,Ymin:Ymax,Xmin:Xmax]=Ft[:,:,Ymin:Ymax,Xmin:Xmax]+Mt, and the mouth residual picture Mt of the t-th frame can be used to supplement the mouth area information Ft[:,:,Ymin:Ymax,Xmin:Xmax] in the picture Ft of the t-th frame generated by the first generator G1, so as to use the residual around the mouth shape learned by the second generator G2 to improve the clarity around the mouth, and the obtained composite picture FFt of the t-th frame can be used as the first composite image information.

[0100] Step 306: Merge the first synthetic image information and the second sketch image information to be trained to obtain second merged image information corresponding to the second sketch image information to be trained.

[0101] Step 307: input the second merged graph information into the first generator to obtain the second generated graph information output by the first generator.

[0102] Specifically, after obtaining the first synthetic image information, the embodiment of the present application may merge the first synthetic image information with the second sketch image information to be trained to obtain the second merged image information corresponding to the second sketch image information to be trained, and then the second merged image information may be input into the first generator, so that the first generator may generate and output the second generated image information corresponding to the second sketch image information to be trained based on the input second merged image information, thereby obtaining the second generated image information output by the first generator. For example, after obtaining the synthetic image FFt of the t-th frame of the video, the synthetic image FFt of the t-th frame may be used as the first synthetic image information and combined with the sketch image S(t+1) of the (t+1)-th frame of the video image to form the second merged image information I(t+1), and then the second merged image information I(t+1) may be input into the first generator G1, so that the first generator G1 generates and outputs the image F(t+1) of the (t+1)-th frame based on the input second merged image information I(t+1) as the second generated image information corresponding to the second sketch image information to be trained.

[0103] Step 308: Generate second mouth residual map information corresponding to the second generated map information through the second generator according to the mouth area information in the second generated map information.

[0104] Specifically, after obtaining the second generated image information, the embodiment of the present application can extract the mouth area information from the second generated image information, so as to generate the mouth residual image information through the second generator based on the extracted mouth area information, and the mouth residual image information can be determined as the second mouth residual image information corresponding to the second generated image information, so that the second mouth residual image information can be used to supplement the mouth area information in the second generated image information later to improve the clarity of the mouth shape area. Optionally, the embodiment of the present application generates the second mouth residual image information corresponding to the second generated image information through the second generator based on the mouth area information in the second generated image information, which can specifically include: merging the mouth area information in the second generated image information and the mouth area information in the second sketch image information to be trained, and inputting the second merged image information obtained after merging into the second generator to obtain the second mouth residual image information.

[0105] For example, in combination with the above example, after the first generator G1 generates and outputs the picture F(t+1) of the (t+1)th frame, the mouth area information FM(t+1) can be extracted from the picture F(t+1) as the mouth area information in the second generated image information, and then the extracted mouth area information FM(t+1) can be merged with the mouth area information SM(t+1) in the second sketch image information to be trained. For example, the mouth area information FM(t+1) can be extracted from the picture F(t+1) according to a preset method for obtaining mouth area information, and then the extracted mouth area information FM(t+1) and the mouth area information SM(t+1) in the second sketch image information to be trained are combined, and the second merged image information generated after the combination is input into the second generator G2 to generate the mouth residual picture M(t+1) of the (t+1)th frame, so that the mouth residual picture M(t+1) of the (t+1)th frame generated by the second generator G2 can be output as the second mouth residual image information.

[0106] Step 309: Generate second synthetic image information corresponding to the second video image information according to the second mouth residual image information and the second generated image information.

[0107] Specifically, after obtaining the second mouth residual map information, the embodiment of the present application can synthesize the second mouth residual map information with the second generated map information, so as to use the second mouth residual map information to supplement the mouth area information in the second generated map information, and update the mouth area information in the second generated map information, thereby improving the clarity of the mouth shape area, and obtaining an updated synthesized picture as the second synthesized image information corresponding to the second video image information. For example, in combination with the above example, after the mouth residual picture M(t+1) of the (t+1)th frame is generated by the second generator G2, data combination can be performed according to the formula FF(t+1)[:, :, Ymin:Ymax, Xmin:Xmax] = F(t+1)[:, :, Ymin:Ymax, Xmin:Xmax] + M(t+1), and the mouth residual picture M(t+1) of the (t+1)th frame is used to supplement the mouth area information F(t+1)[:, :, Ymin:Ymax, Xmin:Xmax] in the picture F(t+1) of the (t+1)th frame generated by the first generator G1, so as to use the residual around the mouth shape learned by the second generator G2 to improve the clarity around the mouth, and the obtained composite picture FF(t+1) of the (t+1)th frame is used as the second composite image information.

[0108] Step 310, determining a model loss value based on the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual image information and the second mouth residual image information.

[0109] Specifically, after obtaining the second synthetic image information corresponding to the second video image information, the embodiment of the present application can combine the first sketch information to be trained, the second sketch information to be trained, the first synthetic image information and the second synthetic image information, and input them into the discriminator to obtain N values, and then use these N values ​​to determine the loss value of the overall generator and the loss value of the overall discriminator; and can combine the mouth area information of the first sketch information to be trained, the mouth area information of the second sketch information to be trained, the first mouth residual map information and the second mouth residual map information, and input them into the discriminator to obtain K values ​​output by the discriminator, and then use these K values ​​to determine the mouth discriminator loss value and the mouth residual generator loss value; thereby, the loss value of the overall generator, the loss value of the overall discriminator, the mouth discriminator loss value and the mouth residual generator loss value can be used for calculation to obtain the model loss value. Among them, N and K can both be integers approximately 1, and N and K can be the same integer, such as N and K can both be 2 or 3, etc., and this implementation does not impose any specific restrictions on this; the model loss value can refer to the loss value of the training model, which can be specifically used to determine whether the training model has converged, and then when the training model has converged, the converged model can be determined as the image processing model, so that the subsequent application reasoning stage can use the image processing model to generate a virtual object image corresponding to the image information to be processed.

[0110] Of course, the embodiments of the present application may also adopt other methods to determine the mouth residual generator loss value, such as respectively calculating the perceptual loss between the first mouth residual map information and the mouth area information in the first video image information, and the perceptual loss between the first mouth residual map information and the mouth area information in the second video image information, so as to determine the mouth residual generator loss value by the perceptual loss between the first mouth residual map information and the mouth area information in the first video image information, and the perceptual loss between the first mouth residual map information and the mouth area information in the second video image information. The embodiments of the present application are not limited to this. In addition, the embodiments of the present application may also use other methods to determine the loss value of the overall generator and the loss value of the overall discriminator. For example, the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information may be combined and input into the discriminator to determine the loss value of the overall discriminator by calculating the distance between the N values ​​output by the discriminator and 1; and / or, the perceptual loss (Perceptual Loss) between the first synthetic image information and the first video image information, and the Perceptual Loss between the second synthetic image information and the second video image information may be respectively determined, and then these two Perceptual Losses may be added to obtain the loss value of the overall generator, and the embodiments of the present application are not limited to this.

[0111] Furthermore, the embodiment of the present application determines the model loss value based on the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual image information and the second mouth residual image information. Specifically, it may include: inputting the first sketch image information to be trained, the second sketch image information to be trained, the first video image information and the second video image information into the discriminator of the image processing model to obtain the first loss information output by the discriminator; inputting the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information and the second synthetic image information into the discriminator to obtain the second loss information output by the discriminator; inputting the mouth area information in the first sketch image information to be trained, the second The mouth area information in the sketch image information to be trained, the first mouth residual image information and the second mouth residual image information are input into the discriminator to obtain the third loss information output by the discriminator; the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the mouth area information in the first video image information and the mouth area information in the second video image information are input into the discriminator to obtain the fourth loss information output by the discriminator; based on the first loss information and the second loss information, the target discriminator loss value and the target generator loss value are determined; based on the third loss information and the fourth loss information, the mouth discriminator loss value and the mouth residual generator loss value are determined; according to the target discriminator loss value, the target generator loss value, the mouth discriminator loss value and the mouth residual generator loss value, the model loss value is determined.

[0112] It should be noted that the target discriminator loss value in the embodiment of the present application may refer to the loss value of the overall discriminator of the target object, such as the loss value of the overall discriminator of the digital human; the target generator loss value refers to the loss value of the overall generator of the target object, such as the loss value of the overall generator of the digital human; the mouth discriminator loss value may refer to the loss value of the discriminator of the target object's mouth shape, such as the loss value of the digital human mouth discriminator; the mouth residual generator loss value may refer to the loss value of the target object's mouth shape residual generator, such as the loss value of the digital human mouth residual generator.

[0113] As an example of the present application, when the image processing model obtained after training is used to infer and generate digital human images, the model loss value Loss can mainly include four parts, namely, the loss value loss1 of the digital human overall discriminator, the loss value loss2 of the digital human overall generator, the loss value loss3 of the digital human mouth residual generator and the loss value loss4 of the digital human mouth discriminator.

[0114] Among them, the loss value loss1 of the digital human overall discriminator can be determined by calculating the loss of the probability of a false image and / or the loss of the probability of a true image. For example, the first sketch information to be trained, the second sketch information to be trained, the first synthetic image information and the second synthetic image information can be merged to merge the sketch St of the t-th frame video image, the sketch S(t+1) of the (t+1)-th frame video image, the digital human image FFt generated in the t-th frame and the virtual object image FF(t+1) generated in the (t+1)-th frame into one image in a channel synthesis manner, and input it into the discriminator D to obtain N values, and then the preset loss function L1 can be used. loss calculates the distance between these N values ​​and 0, so as to determine the calculated distance value as the loss of the probability of a false image. The smaller the calculated distance value is, the better the discriminator can judge the false image. For example, the first sketch information to be trained, the second sketch information to be trained, the first video image information and the second video image information can be merged to merge the sketch St of the t-th frame video image, the sketch S(t+1) of the (t+1)-th frame video image, the t-th frame real image Rt and the (t+1)-th frame real image R(t+1) into one picture in a channel synthesis manner, and input it into the discriminator D to obtain N values ​​as the first loss information, and then L1 can be used. loss calculates the distance between these N values ​​and 1 to determine the calculated distance value as the loss of the probability of a true image. The smaller the calculated distance value is, the more capable the discriminator is of judging a true image. After determining the loss of the probability of a false image and the loss of the probability of a true image, the loss of the probability of a false image and the loss of the probability of a true image can be accumulated to determine the accumulated result as the loss value loss1 of the overall discriminator of the digital human.

[0115] The loss value loss2 of the digital human overall generator can be specifically determined by the first loss information and / or the second loss information output by the discriminator. For example, the sketch image St of the t-th frame video image, the sketch image S(t+1) of the (t+1)-th frame video image, the digital human image FFt generated by the t-th frame, and the virtual object image FF(t+1) generated by the (t+1)-th frame can be combined into one picture in a channel synthesis manner, and input into the discriminator D to obtain N values, and then these N values ​​can be used as the second loss information to calculate the distance between these N values ​​and 1 using the preset loss function L1 loss, so that the calculated distance value can be used as the first loss function value of the digital human overall generator, so that the loss value loss2 of the digital human overall generator can be determined based on the first loss function value, so that the loss value loss2 of the digital human overall generator can include the first loss function value calculated using the loss function L1 loss.

[0116] Of course, the loss value loss2 of the digital human overall generator may include other loss function values ​​in addition to the first loss function value calculated using the loss function L1loss, such as the second loss function value, feature matching loss value (Feature Matching Loss), etc. This example does not impose specific restrictions on this. Among them, the second loss function value can be determined by calculating the perceptual loss (PerceptualLoss) between the generated virtual object image and the real picture. For example, the digital human image FFt generated at the t-th frame and the real picture Rt at the t-th frame can be input into the VGG network model to calculate the feature values ​​of the digital human image FFt and the real picture Rt at different layers of the VGG network model to obtain the distance value, so that the distance value can be used as the perceptual loss PL1 between the digital human image FFt and the real picture Rt; similarly, the digital human image FFt generated at the (t+1)-th frame can be used. The virtual object image FF(t+1) and the (t+1)th frame real image R(t+1) are input into the VGG network model to obtain the perceptual loss PL2 between the virtual object image FF(t+1) generated at the (t+1)th frame and the real image R(t+1) at the (t+1)th frame by calculating the feature values ​​of the virtual object image FF(t+1) and the real image R(t+1) at different layers of the VGG network model. Then, the perceptual loss PL1 and the perceptual loss PL2 are added to obtain the entropy result as the second loss function value of the digital human overall generator. In addition, the feature matching loss value of the digital human overall generator can be obtained by inputting the sketch image St of the tth frame video image, the sketch image S(t+1) of the (t+1)th frame video image, the digital human image FFt generated at the tth frame, and the virtual object image FF(t+1) generated at the (t+1)th frame into the discriminator for calculation.

[0117] The loss value loss4 of the digital mouth discriminator can be determined by calculating the loss of the probability of a fake mouth image and the loss of the probability of a real mouth image. For example, the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the first mouth residual image information, and the second mouth residual image information can be merged to combine the mouth area information SMt of the sketch image of the t-th frame video image, the mouth area information SM(t+1) of the sketch image of the (t+1)-th frame video image, the first mouth residual image information Mt generated and output by the second generator G2 of the t-th frame, and the second mouth residual image information Mt(t+1) generated and output by the (t+1)-th frame into one picture in a channel synthesis manner, and input it into the discriminator D to obtain N values ​​as the third loss information, and then the preset loss function L1 can be used. loss calculates the distance between these N values ​​and 0, so as to determine the calculated distance value as the loss of the probability of a fake mouth picture. The smaller the calculated distance value is, the better the discriminator can judge the fake mouth picture. For example, the mouth area information in the first sketch information to be trained, the mouth area information in the second sketch information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information can be merged to merge the mouth area information SMt of the sketch of the t-th frame video image, the mouth area information SM(t+1) of the sketch of the (t+1)-th frame video image, the mouth area information RMt of the t-th frame real picture, and the mouth area information RM(t+1) of the (t+1)-th frame real picture into one picture in a channel synthesis manner, which is input into the discriminator D to obtain N values ​​as the fourth loss information, and then L1 can be used. loss calculates the distance between the N value and 1 to determine the calculated distance value as the loss of the probability of a real mouth picture. The smaller the calculated distance value is, the more capable the digital mouth discriminator is in identifying a real mouth picture. After determining the loss of the probability of a fake mouth picture and the loss of the probability of a real mouth picture, the loss of the probability of a fake mouth picture and the loss of the probability of a real mouth picture can be added to determine the accumulated result as the loss value loss4 of the digital mouth discriminator.

[0118] The loss value loss3 of the digital human mouth residual generator can be specifically determined by the third loss information and the fourth loss information, and can specifically include two loss function values, which can be the third loss function value and the fourth loss function value respectively. Among them, the third loss function value can be determined by the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the first mouth residual image information and the second mouth residual image information. For example, the mouth area information SMt of the sketch image of the t-th frame video image, the mouth area information SM(t+1) of the sketch image of the (t+1)-th frame video image, the first mouth residual image information Mt generated and output by the second generator G2 of the t-th frame, and the second mouth residual image information Mt(t+1) generated and output by the (t+1)-th frame are combined and input into the discriminator D to obtain N values ​​as the third loss information, and then the preset loss function L1 can be used. loss calculates the distance between these N values ​​and 1, so as to determine the calculated distance value as the third loss function value; the fourth loss function value can be the perceptual loss, such as the perceptual loss PL3 between the first mouth residual map information Mt and the mouth area information RMt of the real picture, and the perceptual loss PL4 between the second mouth residual map information M(t+1) and the mouth area information RM(t+1) of the real picture can be calculated respectively, and then the perceptual loss PL3 and the perceptual loss PL4 can be added to obtain the entropy, and the entropy result of the fourth loss function value as the digital human mouth residual generator can be obtained, so that the loss value loss4 of the digital human mouth discriminator can be determined according to the fourth loss function value and the third loss function value, and then the loss value loss4 of the digital human mouth discriminator, the loss value loss1 of the digital human overall discriminator, the loss value loss2 of the digital human overall generator and the loss value loss3 of the digital human mouth residual generator can be accumulated to obtain the current model loss value Loss.

[0119] Step 311: If the model loss value does not meet the preset model convergence conditions, the parameters of the first generator and / or the parameters of the second generator are updated based on the model loss value, and the first generator and the second generator with updated parameters are iteratively trained until the model loss value meets the preset model convergence conditions.

[0120] Specifically, after obtaining the model loss value, the embodiment of the present application can determine whether the model obtained by the current training has converged by judging whether the model loss value meets the preset model convergence condition, so that the trained model can be determined as the image processing model when the training model converges, so that the image processing model can be used in the subsequent application reasoning stage to generate a virtual object image corresponding to the image to be processed. For example, when the model loss value is greater than the convergence loss threshold in the model convergence condition, it can be determined that the current model loss value does not meet the preset model convergence condition, and it is necessary to continue model training. Subsequently, the back propagation algorithm can be used to update the parameters of the first generator and / or the parameters of the second generator based on the model loss value, so as to iteratively train the first generator and the second generator after updating the parameters until the model loss value is equal to or less than the convergence loss threshold in the model convergence bar.

[0121] After the image processing model is trained, the embodiment of the present application can use the image processing model to generate virtual images such as digital humans using two-frame image reasoning, thereby improving the mouth shape generation effect of the virtual image.

[0122] In actual processing, the image frames in the video can be used as the image information to be processed in the embodiment of the present application, and the previous frame of the image frame currently being processed can be used as the reference image information to generate a virtual image using the two frames of image information, thereby improving the stability of the video generation frame and the mouth shape generation effect while ensuring the reasoning speed. Further, on the basis of the above-mentioned embodiment, the embodiment of the present application determines the target synthetic image information based on the acquired image information to be processed and the reference image information corresponding to the image information to be processed, which can specifically include: extracting the image information of the frame to be processed from the video to be processed, and determining the extracted image information of the frame to be processed as the image information to be processed; determining the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed; performing image synthesis based on the sketch image information and the reference image information to obtain the target synthetic image information.

[0123] Reference Figure 4 , shows a flowchart of the steps of an image generation method provided by an optional embodiment of the present application. Specifically, the image generation method provided by the embodiment of the present application is applied in the model application reasoning stage, and may specifically include the following steps:

[0124] Step 410: extracting frame image information to be processed from the video to be processed, and determining the extracted frame image information to be processed as the image information to be processed.

[0125] Step 420, determining the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed.

[0126] For example, when the current image to be processed is the first frame image in the video, an image Zt with all zeros of the same size as the first frame image and 3 channels can be created, and the image Zt can be used as the reference image information of the first frame image, so that the reference image information can be used to generate a virtual image corresponding to the first frame image, thereby achieving the purpose of generating a virtual image using two frames of images; when the current image to be processed is not the first frame image in the video, such as when the current image to be processed is the second frame image or the third frame image in the video, the previous frame image of the image to be processed can be determined as the reference image information, so that the previous frame image and the image information to be processed can be used to generate a virtual image of the current frame, thereby improving the mouth shape generation effect of the virtual image and improving the stability of the video frame.

[0127] Furthermore, the embodiment of the present application determines the sketch information of the image information to be processed and the reference image information corresponding to the image information to be processed, which may specifically include: extracting facial key point information and contour line information from the image information to be processed, and generating the sketch information of the image information to be processed based on the extracted facial key point information and contour line information; if the image information to be processed is the first frame image information of the video, creating all-zero image information based on the image to be processed, and determining the all-zero image information as the reference image information corresponding to the image information to be processed; if the image information to be processed is not the first frame image information of the video, determining the reference frame target image information as the reference image information corresponding to the image information to be processed, and the reference frame target image information is the target image information corresponding to the previous frame image of the image information to be processed.

[0128] Step 430, performing image synthesis based on the sketch image information and the reference image information to obtain the target synthesized image information.

[0129] Step 440: input the target synthetic graph information into a pre-trained first generator to obtain initial generated graph information output by the first generator.

[0130] Step 450: Based on the mouth area information in the initial generated image information, a pre-trained second generator is used to generate mouth residual image information corresponding to the initial generated image information.

[0131] Among them, the second generator is a generator jointly trained with the first generator. Specifically, after obtaining the initial generated graph information, the embodiment of the present application can merge the mouth area information of the initial generated graph information and the mouth area information of the image information to be processed to obtain the target mouth synthesis graph information generated after the merger, and then the target mouth synthesis graph information generated after the merger can be input into the pre-trained second generator, so that the second generator can generate the mouth residual graph information corresponding to the initial generated graph information based on the target mouth synthesis graph information, thereby obtaining the mouth residual graph information output by the second generator.

[0132] Optionally, the embodiment of the present application generates mouth residual map information corresponding to the initial generated map information by a pre-trained second generator based on the mouth area information in the initial generated map information, which may specifically include the following sub-steps:

[0133] Sub-step 4501, extracting the mouth area information in the initial generated image information to serve as the initial generated image mouth information;

[0134] Sub-step 4502, synthesizing the mouth information of the initially generated image and the mouth information of the image to be processed to obtain target mouth synthesis image information corresponding to the initially generated image information, wherein the mouth information of the image to be processed is the mouth area information in the image information to be processed;

[0135] Sub-step 4503, inputting the target mouth synthesis map information into a pre-trained second generator to obtain the mouth residual map information output by the second generator.

[0136] Step 460: Generate target image information corresponding to the image information to be processed based on the mouth residual image information and the initial generated image information.

[0137] Furthermore, the embodiment of the present application generates target image information corresponding to the image information to be processed based on the mouth residual map information and the initial generated map information, which may specifically include: based on the mouth residual map information, supplementing the mouth area information in the initial generated map information to obtain the supplemented image information; and determining the supplemented image information as the target image information.

[0138] As an example of the present application, in a digital human video generation scenario, after extracting the frame image information Rt to be processed from the video to be processed, the currently extracted frame image information Rt to be processed can be determined as the image information to be processed, and then the facial key point information and Canny line information can be extracted from the frame image information Rt to be processed, so as to generate the sketch image information St of the image information Rt to be processed based on the extracted facial key point information and Canny line information, and the corresponding reference image information can be obtained for the image information Rt to be processed, so as to perform image synthesis based on the reference image information and the sketch image information St, and obtain the target synthesis image information input to the first generator G1, so that the initial generation image information can be generated by the first generator G1 in the image processing model, and the mouth residual image information can be generated by the second generator G2, so as to use the mouth residual image information to supplement the mouth area information in the initial generation image information, and obtain the final digital human image as the target image information, so that the corresponding digital human video can be generated and output using multiple frames of digital human images in the future, thereby improving the overall generation effect of the digital human video.

[0139] Specifically, when the sketch image information St of the image to be processed is the sketch image information of the first frame image in the video, the pre-created all-zero image information Zt can be used as the reference image information, so that the sketch image information St of the image to be processed and the all-zero image information Zt can be combined into the target synthetic image information It of the first frame of the video, and input into the first generator G1, so that the first generator G1 generates the initial generated image information Ft according to the target synthetic image information It as the initial generated image of the first frame, and then the mouth area information can be extracted from the initial generated image information Ft, so as to combine the mouth area information FMt of the initial generated image of the first frame with the mouth area information SMt of the sketch image information of the first frame image. The rows are merged and input into the second generator G2, that is, the mouth area information SMt of the sketch image information of the first frame and the mouth area information FMt of the initial generated image of the first frame are combined and input into the second generator G2 to generate a residual picture of the mouth through the second generator G2 as the mouth residual image information Mt of the first frame. Subsequently, the mouth residual image Mt of the first frame can be used to supplement the mouth area information Ft[:,:,Ymin:Ymax,Xmin:Xmax] in the initial generated image information image Ft generated by the first frame, so as to use the residual around the mouth shape learned by the second generator G2 to improve the clarity around the mouth, and obtain the digital human image FFt as the target image information of the first frame.

[0140] After generating the digital human image FFt as the target image information, the digital human image FFt can be used as the reference image information corresponding to the next frame of the image information to be processed R(t+1), that is, the digital human image FFt of the first frame can be used as the reference image information corresponding to the image information to be processed of the second frame, and merged with the sketch image information S(t+1) of the second frame to be processed to obtain the target synthetic image information I(t+1) of the second frame of the video, and the target synthetic image information I(t+1) of the second frame can be input to the first generator G1, so that the first generator G1 generates the initial generated image information F(t+1) of the second frame according to the target synthetic image information I(t+1) of the second frame as the initial generated image of the second frame, and then the mouth area information can be extracted from the initial generated image information F(t+1) of the second frame to combine the mouth area information FM(t+1) of the initial generated image of the second frame and the mouth area information SM(t+1) of the sketch image information of the second frame image. The mouth area information SM(t+1) of the sketch image information of the second frame and the mouth area information FM(t+1) of the initial generated image of the second frame are combined and input into the second generator G2, so as to generate a residual image of the mouth through the second generator G2 as the mouth residual image information M(t+1) of the second frame. Subsequently, the mouth residual image Mt of the second frame can be used to supplement the mouth area information F(t+1)[:,:,Ymin:Ymax,Xmin:Xmax] in the initial generated image information image F(t+1) generated by the second frame, so as to improve the clarity around the mouth by using the residual around the mouth shape learned by the second generator G2, and obtain the digital human image FF(t+1) as the target image information of the second frame… And so on, the digital human images of other frames of the video frame can be generated, and then the digital human video can be generated based on the multi-frame digital human images generated by inference, so as to realize the generation of digital human video.

[0141] In summary, the embodiment of the present application adds an additional second generator based on the first generator, so that the residual around the mouth shape can be learned by the second generator, so that the mouth area information in the initial generation image information generated by the first generator can be supplemented by the mouth residual map information learned by the second generator, so as to improve the clarity around the mouth, thereby improving the clarity of the mouth area in the final generated target image information, such as improving the clarity of the mouth area in the final generated digital human image, that is, improving the mouth shape generation effect, and then the overall generation effect of the digital human image can be improved while ensuring the inference speed, and the stability of the video generation frame can be improved to meet the generation requirements of the digital human video application.

[0142] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.

[0143] The present application also provides an image generation device. Figure 5 As shown, the image generating device 500 provided in the embodiment of the present application may include the following modules:.

[0144] A target composite image determination module 510 is used to determine target composite image information based on the acquired image information to be processed and reference image information corresponding to the image information to be processed;

[0145] An initial generated graph module 520, configured to input the target synthetic graph information into a pre-trained first generator to obtain initial generated graph information output by the first generator;

[0146] A mouth residual map generation module 530, configured to generate mouth residual map information corresponding to the initial generated map information by a pre-trained second generator according to the mouth region information in the initial generated map information, wherein the second generator is a generator jointly trained with the first generator;

[0147] The target image generation module 540 is used to generate target image information corresponding to the image information to be processed according to the mouth residual image information and the initial generated image information.

[0148] Optionally, the target composite image determination module 510 includes:

[0149] An extraction submodule, used for extracting the frame image information to be processed from the video to be processed, and determining the extracted frame image information to be processed as the image information to be processed;

[0150] A determination submodule, used to determine the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed;

[0151] The synthesis submodule is used to perform image synthesis based on the sketch image information and the reference image information to obtain the target synthetic image information.

[0152] Optionally, the determining submodule includes:

[0153] An extraction unit, used to extract facial key point information and contour line information from the image information to be processed;

[0154] A generating unit, for generating the sketch information of the image information to be processed according to the extracted facial key point information and contour line information

[0155] A first determining unit is used to create all-zero image information based on the image to be processed when the image information to be processed is the first frame image information of the video, and determine the all-zero image information as the reference image information corresponding to the image information to be processed;

[0156] The second determination unit is used to determine the reference frame target image information as the reference image information corresponding to the image information to be processed when the image information to be processed is not the first frame image information of the video, and the reference frame target image information is the target image information corresponding to the previous frame image of the image information to be processed.

[0157] Optionally, the mouth residual map generation module 530 includes:

[0158] A mouth region extraction submodule, used to extract the mouth region information in the initial generated image information as the initial generated image mouth information;

[0159] The initial synthesis submodule is used to synthesize based on the initial generated image mouth information and the image mouth information to be processed to obtain the target mouth synthesis image information corresponding to the initial generated image information, wherein the image mouth information to be processed is the mouth area information in the image information to be processed.

[0160] The mouth residual map submodule is used to input the target mouth synthesis map information into a pre-trained second generator to obtain the mouth residual map information output by the second generator.

[0161] Optionally, the target image generation module includes:

[0162] A supplementary processing submodule, used for supplementing the mouth region information in the initial generated image information based on the mouth residual image information to obtain image information after supplementary processing;

[0163] The target image determination submodule is used to determine the image information after the supplementary processing as the target image information.

[0164] Optionally, the image generating device further includes:

[0165] A training video image acquisition module is used to obtain the video image information to be trained from a preset training data set;

[0166] A model training module is used to perform model training based on the video image information to be trained to obtain an image processing model including the first generator and the second generator.

[0167] Optionally, the model training module includes:

[0168] A sketch image to be trained determination submodule, used to determine the sketch image information to be trained based on the video image information to be trained, wherein the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information;

[0169] A first generated graph submodule, used for generating first generated graph information corresponding to the first to-be-trained sketch graph information by using a pre-built first generator;

[0170] A first mouth residual map submodule, configured to generate, according to the mouth region information in the first generated map information, first mouth residual map information corresponding to the first generated map information through the second generator;

[0171] A first synthetic image submodule, configured to generate first synthetic image information corresponding to the first video image information according to the first mouth residual image information and the first generated image information;

[0172] A second merged image submodule, configured to merge the first synthetic image information and the second sketch image information to be trained to obtain second merged image information corresponding to the second sketch image information to be trained;

[0173] A second generated graph submodule, used for inputting the second merged graph information into the first generator to obtain the second generated graph information output by the first generator;

[0174] A second mouth residual map submodule, used to generate second mouth residual map information corresponding to the second generated map information through the second generator according to the mouth area information in the second generated map information;

[0175] A second synthetic image submodule, used for generating second synthetic image information corresponding to the second video image information according to the second mouth residual image information and the second generated image information;

[0176] A model loss value determination submodule, used to determine a model loss value based on the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual image information, and the second mouth residual image information;

[0177] The parameter update training submodule is used to update the parameters of the first generator and / or the parameters of the second generator based on the model loss value when the model loss value does not meet the preset model convergence condition, and iteratively train the first generator and the second generator after the updated parameters until the model loss value meets the preset model convergence condition.

[0178] Optionally, the first generated graph submodule is specifically used to obtain the all-zero image information corresponding to the first sketch graph information to be trained; and to merge the all-zero image information and the first sketch graph information to be trained to obtain the first merged graph information corresponding to the first sketch graph information to be trained; and, to input the first merged graph information into the first generator to obtain the first generated graph information output by the first generator.

[0179] Optionally, the first mouth residual map submodule is specifically used to merge the mouth area information in the first generated image information and the mouth area information in the first sketch image information to be trained, and input the first merged image information obtained after the merger into the second generator to obtain the first mouth residual map information.

[0180] Optionally, the second mouth residual map submodule is specifically used to merge the mouth area information in the second generated image information and the mouth area information in the second sketch image information to be trained, and input the second merged image information obtained after the merger into the second generator to obtain the second mouth residual map information.

[0181] Optionally, the model loss value determination submodule includes:

[0182] a first loss information determining unit, configured to input the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information into a discriminator of the image processing model, and obtain first loss information output by the discriminator;

[0183] A second loss information determining unit, configured to input the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information into the discriminator to obtain second loss information output by the discriminator;

[0184] a third loss information determining unit, configured to input the mouth region information in the first sketch image information to be trained, the mouth region information in the second sketch image information to be trained, the first mouth residual image information, and the second mouth residual image information into the discriminator, to obtain third loss information output by the discriminator;

[0185] a fourth loss information determining unit, configured to input the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information into the discriminator, and obtain fourth loss information output by the discriminator;

[0186] A first loss value determining unit, configured to determine a target discriminator loss value and a target generator loss value based on the first loss information and the second loss information;

[0187] A second loss value determining unit, configured to determine a mouth discriminator loss value and a mouth residual generator loss value based on the third loss information and the fourth loss information;

[0188] A model loss value unit is used to determine the model loss value based on the target discriminator loss value, the target generator loss value, the mouth discriminator loss value and the mouth residual generator loss value.

[0189] It should be noted that the image generating device provided above can execute the image generating method provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0190] In a specific implementation, the above-mentioned image generation device can be applied to electronic devices such as personal computers and servers, so that the electronic device can be used as an image generation device to generate images based on the acquired sketch information of the image to be processed, thereby improving the generation effect of the mouth shape and the overall generation effect of the image while ensuring the reasoning speed. Furthermore, an embodiment of the present application also provides an image generation device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to execute the program stored in the memory, and implement the steps of the image generation method described in any of the above method embodiments.

[0191] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image generating method described in any one of the above method embodiments are implemented.

[0192] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device, equipment, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0193] In this article, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0194] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. An image generation method, characterized in that: include: Determine target composite image information according to the acquired image information to be processed and reference image information corresponding to the image information to be processed, wherein the reference image information is all-zero image information or target image information corresponding to a previous frame image of the image information to be processed; Inputting the target synthetic graph information into a pre-trained first generator to obtain initial generated graph information output by the first generator; According to the mouth area information in the initial generated image information, a pre-trained second generator is used to generate mouth residual image information corresponding to the initial generated image information, wherein the second generator is a generator jointly trained with the first generator; Generating target image information corresponding to the image information to be processed according to the mouth residual image information and the initial generated image information; The method of generating the mouth residual map information corresponding to the initial generated map information by a pre-trained second generator based on the mouth region information in the initial generated map information includes: Extracting mouth area information from the initial generated image information to serve as initial generated image mouth information; Based on the initial generated image mouth information and the image mouth information to be processed, synthesizing is performed to obtain target mouth synthesis image information corresponding to the initial generated image information, wherein the image mouth information to be processed is the mouth area information in the image information to be processed; Inputting the target mouth synthesis map information into a pre-trained second generator to obtain the mouth residual map information output by the second generator; The step of determining target composite image information based on the acquired image information to be processed and reference image information corresponding to the image information to be processed includes: Extracting the frame image information to be processed from the video to be processed, and determining the extracted frame image information to be processed as the image information to be processed; Determine the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed; Performing image synthesis based on the sketch image information and the reference image information to obtain the target synthetic image information; The step of determining the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed includes: Extracting facial key point information and contour line information from the image information to be processed, and generating sketch information of the image information to be processed based on the extracted facial key point information and contour line information; If the image information to be processed is the first frame of video image information, creating all-zero image information based on the image to be processed, and determining the all-zero image information as reference image information corresponding to the image information to be processed; If the image information to be processed is not the first frame image information of the video, the reference frame target image information is determined as the reference image information corresponding to the image information to be processed, and the reference frame target image information is the target image information corresponding to the previous frame image of the image information to be processed.

2. The image generation method according to claim 1, characterized in that: Generating target image information corresponding to the image information to be processed according to the mouth residual image information and the initial generated image information, including: Based on the mouth residual image information, supplementing the mouth area information in the initial generated image information to obtain supplemented image information; The image information after the supplementary processing is determined as the target image information.

3. The image generation method according to claim 1, characterized in that: Before inputting the target synthetic graph information into the pre-trained first generator, the method further includes: Obtaining video image information to be trained from a preset training data set; Model training is performed based on the video image information to be trained to obtain an image processing model including the first generator and the second generator.

4. The image generation method according to claim 3, characterized in that: The performing model training according to the video image information to be trained includes: Determine the sketch image information to be trained based on the video image information to be trained, the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information; Generate first generated graph information corresponding to the first to-be-trained sketch graph information by using a pre-built first generator; Generate, by the second generator, first mouth residual map information corresponding to the first generated map information according to the mouth region information in the first generated map information; generating first synthetic image information corresponding to the first video image information according to the first mouth residual image information and the first generated image information; Merging the first synthetic image information and the second sketch image information to be trained to obtain second merged image information corresponding to the second sketch image information to be trained; Inputting the second merged graph information into the first generator to obtain second generated graph information output by the first generator; According to the mouth area information in the second generated image information, generating second mouth residual image information corresponding to the second generated image information by the second generator; generating second synthetic image information corresponding to the second video image information according to the second mouth residual image information and the second generated image information; Determine a model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual image information, and the second mouth residual image information; If the model loss value does not meet the preset model convergence conditions, the parameters of the first generator and / or the parameters of the second generator are updated based on the model loss value, and the first generator and the second generator with updated parameters are iteratively trained until the model loss value meets the preset model convergence conditions.

5. The image generation method according to claim 4, characterized in that: The step of generating first generated graph information corresponding to the first to-be-trained sketch graph information by using a pre-built first generator includes: Obtaining all-zero image information corresponding to the first sketch image information to be trained; Merging the all-zero image information and the first sketch image information to be trained to obtain first merged image information corresponding to the first sketch image information to be trained; The first merged graph information is input into the first generator to obtain the first generated graph information output by the first generator.

6. The image generation method according to claim 5, characterized in that: The method of generating first mouth residual image information corresponding to the first generated image information by the second generator according to the mouth area information in the first generated image information includes: merging the mouth area information in the first generated image information with the mouth area information in the first to-be-trained sketch image information, and inputting the first merged image information obtained after merging into the second generator to obtain the first mouth residual image information; The method of generating second mouth residual map information corresponding to the second generated map information based on the mouth area information in the second generated map information through the second generator includes: merging the mouth area information in the second generated map information and the mouth area information in the second sketch map information to be trained, and inputting the second merged image information obtained after the merging into the second generator to obtain the second mouth residual map information.

7. The image generation method according to claim 4, characterized in that: The determining of the model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, the second synthetic image information, the first mouth residual image information and the second mouth residual image information includes: Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information into a discriminator of the image processing model to obtain first loss information output by the discriminator; Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information into the discriminator to obtain second loss information output by the discriminator; Inputting the mouth region information in the first sketch image information to be trained, the mouth region information in the second sketch image information to be trained, the first mouth residual image information, and the second mouth residual image information into the discriminator to obtain third loss information output by the discriminator; Inputting the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information into the discriminator to obtain fourth loss information output by the discriminator; Determine a target discriminator loss value and a target generator loss value based on the first loss information and the second loss information; Determine a mouth discriminator loss value and a mouth residual generator loss value based on the third loss information and the fourth loss information; The model loss value is determined based on the target discriminator loss value, the target generator loss value, the mouth discriminator loss value and the mouth residual generator loss value.

8. An image generating device, characterized in that: include: A target composite image determination module, used to determine the target composite image information according to the acquired image information to be processed and the reference image information corresponding to the image information to be processed, wherein the reference image information is all-zero image information or the target image information corresponding to the previous frame image of the image information to be processed; An initial generated graph module, used for inputting the target synthetic graph information into a pre-trained first generator to obtain the initial generated graph information output by the first generator; A mouth residual map generation module, used to generate mouth residual map information corresponding to the initial generated map information through a pre-trained second generator according to the mouth area information in the initial generated map information, wherein the second generator is a generator jointly trained with the first generator; A target image generation module, used for generating target image information corresponding to the image information to be processed according to the mouth residual image information and the initial generated image information; The method of generating the mouth residual map information corresponding to the initial generated map information by a pre-trained second generator based on the mouth region information in the initial generated map information includes: Extracting mouth area information from the initial generated image information to serve as initial generated image mouth information; Based on the initial generated image mouth information and the image mouth information to be processed, synthesizing is performed to obtain target mouth synthesis image information corresponding to the initial generated image information, wherein the image mouth information to be processed is the mouth area information in the image information to be processed; Inputting the target mouth synthesis map information into a pre-trained second generator to obtain the mouth residual map information output by the second generator; The step of determining target composite image information based on the acquired image information to be processed and reference image information corresponding to the image information to be processed includes: Extracting the frame image information to be processed from the video to be processed, and determining the extracted frame image information to be processed as the image information to be processed; Determine the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed; Performing image synthesis based on the sketch image information and the reference image information to obtain the target synthetic image information; The step of determining the sketch image information of the image information to be processed and the reference image information corresponding to the image information to be processed includes: Extracting facial key point information and contour line information from the image information to be processed, and generating sketch information of the image information to be processed based on the extracted facial key point information and contour line information; If the image information to be processed is the first frame of video image information, creating all-zero image information based on the image to be processed, and determining the all-zero image information as reference image information corresponding to the image information to be processed; If the image information to be processed is not the first frame image information of the video, the reference frame target image information is determined as the reference image information corresponding to the image information to be processed, and the reference frame target image information is the target image information corresponding to the previous frame image of the image information to be processed.

9. An image generating device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps of the image generation method described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image generating method according to any one of claims 1 to 7 are implemented.

Images