Image processing method, device, equipment and medium

Abstract

The present application relates to an image processing method, device, equipment and medium, and to the field of computer technology. The image processing method includes: obtaining sketch information of an image to be processed; inputting the sketch information into a pre-trained first generator to obtain target mouth area information output by the first generator; determining target composite image information based on the target mouth area information, the sketch information and reference image information corresponding to the image to be processed; inputting the target composite image information into a pre-trained second generator, and determining the image information output by the second generator as the target image information corresponding to the image to be processed, wherein the second generator is a generator jointly trained with the first generator. The present application improves the generation effect of the mouth shape while ensuring the inference speed, thereby improving the overall generation effect of the image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, equipment and medium. Background Art

[0002] Digital human is a kind of virtual simulation of human body at different levels of form and function using information science methods. With the rapid development of computer technology, the technology of generating digital human is becoming more and more mature.

[0003] In actual processing, if digital human generation technology is to be applied in business and realize real-time interaction, the digital human generation solution must meet at least two points: good generation effect and fast reasoning speed; among them, good digital human generation effect is a necessary prerequisite, and fast reasoning speed is a business need. A large number of generation technologies that have emerged so far are mainly based on image translation models such as pix2pix, pix2pixHD, Vid2Vid, few shot video2video, NERF, StyleGAN, etc., to realize the generation of digital human videos. Specifically, these existing image translation models mainly input real pictures or sketches of real pictures into the generator to generate digital human pictures corresponding to video and audio through the generator, so that digital human videos can be generated and output based on the digital human pictures generated by the generator.

[0004] However, the existing digital human generation schemes directly convert real pictures or sketches of real pictures into digital human pictures through the generator, which has the problem of poor technical effect of generation, such as the generated digital human pictures have defects and large jitters between video frames; or the inference speed cannot meet the requirements of real-time interaction, affecting the user experience. Summary of the invention

[0005] In view of this, the present application provides an image processing method, device, equipment and medium to improve the mouth shape generation effect while ensuring the inference speed, thereby improving the image generation effect.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Get the sketch information of the image to be processed;

[0008] Inputting the sketch image information into a pre-trained first generator to obtain target mouth area information output by the first generator;

[0009] Determining target composite image information according to the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed;

[0010] The target synthetic image information is input into a pre-trained second generator, and the image information output by the second generator is determined as the target image information corresponding to the image to be processed, wherein the second generator is a generator jointly trained with the first generator.

[0011] Optionally, determining target composite image information according to the target mouth area information, the sketch image information and reference image information corresponding to the image to be processed includes:

[0012] Based on the target mouth area information, respectively updating the mouth area information in the reference image information and the mouth area information in the sketch information to obtain updated first reference image information and first sketch information;

[0013] The target synthesized image information is obtained by synthesizing the first reference image information and the first sketch image information.

[0014] Optionally, before determining the target composite image information based on the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed, the image processing method also includes: if the image to be processed is the first frame image of the video, creating all-zero image information based on the image to be processed, and determining the all-zero image information as the reference image information corresponding to the image to be processed; if the image to be processed is not the first frame image of the video, determining the reference frame target image information as the reference image information corresponding to the image to be processed, and the reference frame target image information is the target image information corresponding to the previous frame image of the image to be processed.

[0015] Optionally, before obtaining the sketch information of the image to be processed, the process further includes:

[0016] Obtaining video image information to be trained from a preset training data set;

[0017] Model training is performed based on the video image information to be trained to obtain an image processing model including the first generator and the second generator.

[0018] Optionally, performing model training according to the video image information to be trained includes:

[0019] Determine the sketch image information to be trained based on the video image information to be trained, the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information;

[0020] Generate first mouth region information corresponding to the first sketch image information to be trained and second mouth region information corresponding to the second sketch image information to be trained respectively through a pre-constructed first generator;

[0021] Performing data synthesis based on the first mouth region information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained to obtain initial synthesized image information;

[0022] Inputting the initial composite image information into the second generator to obtain first composite image information corresponding to the first video image information;

[0023] Performing data synthesis based on the first synthetic image information, the second mouth region information and the second sketch image information to be trained to obtain intermediate synthetic image information;

[0024] Inputting the intermediate synthetic image information into the second generator to obtain second synthetic image information corresponding to the second video image information;

[0025] Determine a model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information;

[0026] If the model loss value does not meet the preset model convergence conditions, the parameters of the first generator and / or the parameters of the second generator are updated based on the model loss value, and the first generator and the second generator with updated parameters are iteratively trained until the model loss value meets the preset model convergence conditions.

[0027] Optionally, the generating, by respectively using a pre-constructed first generator, first mouth region information corresponding to the first sketch image information to be trained and second mouth region information corresponding to the second sketch image information to be trained comprises:

[0028] Inputting the first sketch image information to be trained into the pre-built first generator to obtain the first mouth region information;

[0029] The second sketch image information to be trained is input into the pre-built first generator to obtain the second mouth area information.

[0030] Optionally, performing data synthesis based on the first mouth region information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained to obtain initial synthesized image information includes:

[0031] Based on the first mouth region information, the mouth region information in the first to-be-trained sketch image information is updated to obtain updated target training sketch image information;

[0032] Based on the first mouth region information, the mouth region information in the all-zero image information is updated to obtain target training initial image information;

[0033] The target training sketch image information and the target training initial image information are synthesized to obtain the initial synthesized image information.

[0034] Optionally, determining the model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information includes:

[0035] Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information into a discriminator of the image processing model to obtain first loss information output by the discriminator;

[0036] Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information into the discriminator to obtain second loss information output by the discriminator;

[0037] Inputting the mouth region information in the first sketch image information to be trained, the mouth region information in the second sketch image information to be trained, the first mouth region information, and the second mouth region information into the discriminator to obtain third loss information output by the discriminator;

[0038] Inputting the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information into the discriminator to obtain fourth loss information output by the discriminator;

[0039] Based on the first loss information and the second loss information, determining a loss value of the discriminator and a loss value of the second generator;

[0040] Determining a loss value of the first generator based on the third loss information and the fourth loss information;

[0041] The model loss value is determined according to the loss value of the discriminator, the loss value of the second generator, and the loss value of the first generator.

[0042] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0043] A contour image acquisition module is used to obtain the sketch image information of the image to be processed;

[0044] A target mouth region determination module, used for inputting the sketch image information into a pre-trained first generator to obtain target mouth region information output by the first generator;

[0045] A target composite image determination module, used to determine target composite image information according to the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed;

[0046] A target image determination module is used to input the target composite image information into a pre-trained second generator, and determine the image information output by the second generator as the target image information corresponding to the image to be processed, wherein the second generator is a generator jointly trained with the first generator.

[0047] In a third aspect, an embodiment of the present application provides an image processing device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the image processing method as described in any one of the first aspects when executing the program stored in the memory.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image processing method as described in any one of the first aspects are implemented.

[0049] The embodiment of the present application obtains the target mouth area information output by the first generator by inputting the acquired sketch information of the image to be processed into the first generator, and determines the target composite image information according to the target mouth area information, the sketch information and the reference image information corresponding to the image to be processed, and then inputs the target composite image information into the pre-trained second generator, so that the second generator can generate area information other than the mouth while retaining the mouth area information generated by the first generator, thereby improving the generation effect of the mouth shape while ensuring the reasoning speed, and improving the overall generation effect of the image while ensuring the reasoning speed, thereby being able to meet the generation needs of the applied digital human. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 is a flowchart of the steps of an image processing method provided by an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a sketch of an image to be processed in an example of the present application;

[0054] Figure 3 A flowchart of an image processing method provided for an optional embodiment of the present application;

[0055] Figure 4 A flowchart of an image processing method provided for an optional embodiment of the present application;

[0056] Figure 5 A structural block diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] Reference Figure 1 , shows a flowchart of the steps of an image processing method provided by an embodiment of the present application. Specifically, the image processing method provided by the present application can be applied to video image processing, such as generating a virtual image of a video based on a real image; wherein the virtual image can be a digital human image in a video, and can be specifically used to represent a digital human in a digital human video. Figure 1 As shown, the image processing method in the embodiment of the present application may specifically include the following steps:

[0059] Step 110, obtaining sketch information of the image to be processed.

[0060] Among them, the image to be processed may refer to a real picture to be processed, such as the real picture Rt of the tth frame to be processed in the user's recorded video, where t is an integer; the real picture Rt contains the face of the target object, and the target object may refer to the object to be processed by image processing, such as a person object used to represent a person, or an animal object used to represent an animal, etc. The embodiment of the present application does not impose any specific restrictions on this.

[0061] Specifically, when performing image processing, the embodiment of the present application can use the real picture that needs to be processed at present as the image to be processed, and can determine whether the image to be processed is a sketch. Therefore, when the real picture that needs to be processed at present is not a sketch, facial key point information and edge (Canny) line information can be extracted from the image to be processed, so as to form the sketch information of the image to be processed based on the extracted facial key point information and Canny line information; and when the real picture that needs to be processed at present is a sketch, the information of the image to be processed can be directly determined as the sketch information of the image to be processed, thereby improving the efficiency of information acquisition. Among them, the sketch information can be used to represent the sketch St of the image to be processed, such as Figure 2 It should be noted that the extracted facial key point information can be used to represent facial key points, such as the coordinate values ​​of 68 facial key points; the Canny line information can represent the Canny line in the figure, and can be specifically used to represent the extracted edge line features, such as the extracted hair and clothing features.

[0062] Step 120: input the sketch image information into a pre-trained first generator to obtain the target mouth area information output by the first generator.

[0063] The target mouth area information may represent the mouth shape area image Mt generated by the first generator according to the input sketch information, and may be specifically used to determine the mouth shape generated by the first generator, such as being used to represent the digital human mouth generated by the first generator. Specifically, after obtaining the sketch information of the image to be processed, the embodiment of the present application may input the sketch information into the pre-trained first generator G1 to generate the mouth area information of the target object through the first generator G1, and then the mouth area information generated by the first generator G1 may be output as the target mouth area information.

[0064] Step 130, determining target composite image information based on the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed.

[0065] Specifically, after obtaining the target mouth area information, the embodiment of the present application can perform data combination based on the target mouth area information to combine the target mouth area information into the reference image information and the sketch information, respectively, so that the mouth area information in the reference image information and the mouth area information in the sketch information can be updated to the target mouth area information, and the updated first reference image information and the first sketch information can be obtained, and then the updated first reference image information and the first sketch information can be synthesized into the target synthetic image information.

[0066] Further, the embodiment of the present application determines the target composite image information based on the target mouth area information, the sketch information, and the reference image information corresponding to the image to be processed, which may specifically include: based on the target mouth area information, respectively updating the mouth area information in the reference image information and the mouth area information in the sketch information to obtain updated first reference image information and first sketch information; synthesizing based on the first reference image information and the first sketch information to obtain the target composite image information. Among them, the first reference image information may represent the reference image obtained after the mouth area information in the reference image information is updated and replaced with the target mouth area information; the first sketch information may represent the sketch obtained after the mouth area information in the sketch information is updated and replaced with the target mouth area information; the target composite image information may represent the composite image It generated by merging the updated reference image and the updated sketch.

[0067] For example, in the case where the first generator G1 is mainly used to generate a digital human mouth, after the acquired sketch information of the image to be processed is input into the first generator G1, that is, after the sketch St of the real image Rt to be processed is input into the first generator G1, the human mouth area image Mt output by the first generator G1 can be determined as the target mouth area information, so that the mouth area information Zt[:,:,Ymin:Ymax,Xmin:Xmax] in the reference image information and the mouth area information St[:,:,Ymin:Ymax,Xmin:Xmax] in the sketch information can be updated respectively based on the human mouth area image Mt output by the first generator G1 to obtain the updated reference image information (that is, the first reference image information) and the updated sketch information (that is, the first sketch information). Specifically, the mouth area information Zt[:,:,Ymin:Ymax,Xmin:Xmax] in the reference image information can be updated to the target mouth area information by digital combination, that is, Zt[:,:,Ymin:Ymax,Xmin:Xmax]=Mt, to obtain the updated reference image information (i.e., the first reference image information), and the mouth area information St[:,:,Ymin:Ymax,Xmin:Xmax] in the tracing information can be updated to the target mouth area information Mt, that is, St[:,:,Ymin:Ymax,Xmin:Xmax]=Mt, to obtain the updated sketch information (i.e., the first sketch information). Subsequently, the updated reference image information can be merged with the first sketch information to merge the updated reference image and the updated sketch to obtain the target composite image information.

[0068] Step 140: input the target composite image information into a pre-trained second generator, and determine the image information output by the second generator as the target image information corresponding to the image to be processed.

[0069] Among them, the second generator is a generator jointly trained with the first generator. Specifically, after obtaining the target synthetic image information, the embodiment of the present application can input the target synthetic image information into the pre-trained second generator G2, so as to generate area information other than the mouth through the second generator G2 while retaining the mouth area information generated by the first generator G1, and obtain the image information output by the second generator G2, and then the image information output by the second generator can be determined as the target image information corresponding to the image to be processed. The target image information can be used to represent the virtual object image generated by the second generator G2 while retaining the mouth information generated by the first generator G1, such as the digital human image generated by the second generator G2 while retaining the mouth information generated by the first generator G1.

[0070] It can be seen that after obtaining the sketch information of the image to be processed, the embodiment of the present application obtains the target mouth area information output by the first generator by inputting the sketch information into the pre-trained first generator, and determines the target composite image information based on the target mouth area information, the sketch information and the reference image information corresponding to the image to be processed, and then inputs the target composite image information into the pre-trained second generator, so that the second generator can generate area information other than the mouth while retaining the mouth area information generated by the first generator, thereby improving the generation effect of the mouth shape while ensuring the reasoning speed, and improving the overall generation effect of the image while ensuring the reasoning speed, thereby improving the user experience.

[0071] In a specific implementation, the embodiment of the present application can pre-use the image information to be trained in the training data set for model training to train an image processing model including a first generator and a second generator. Among them, the first generator can be used to generate mouth area information, such as being used to generate a digital human mouth picture; the second generator can be used to generate information of areas outside the mouth while retaining the information of the mouth area generated by the first generator; the training data set can refer to a collection of data used for model training, and can specifically include one or more image data, such as can include a continuous multi-frame video image data for training in a video; the image information to be trained can refer to the image data to be trained, such as the video image information to be trained, and the video image information to be trained can refer to the video image to be trained. Further, based on the above-mentioned embodiment, the image processing method provided in the embodiment of the present application can also include: obtaining the video image information to be trained from a preset training data set before obtaining the sketch image information of the image to be processed; performing model training based on the video image information to be trained to obtain an image processing model including the first generator and the second generator.

[0072] In actual processing, one or more image data can be obtained from the training data set for model training. For example, N consecutive frames of video image data of a video in the training data set can be obtained as the image information to be trained for model training, so that N consecutive frames of video image data in the video can be used for model training. For example, two consecutive frames of video images in the video can be obtained from the training data set for model training, where N is an integer greater than 1. Optionally, the embodiment of the present application performs model training according to the video image information to be trained, which may specifically include: determining the sketch image information to be trained based on the video image information to be trained, the video image information to be trained comprising first video image information and second video image information, the sketch image information to be trained comprising first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information; generating first mouth area information corresponding to the first sketch image information to be trained and second mouth area information corresponding to the second sketch image information to be trained respectively by a pre-constructed first generator; performing data synthesis according to the first mouth area information, the first sketch image information to be trained and the all-zero image information corresponding to the sketch image information to be trained to obtain initial synthesized image information; inputting the initial synthesized image information into the second generator to obtain to the first synthetic image information corresponding to the first video image information; perform data synthesis based on the first synthetic image information, the second mouth area information and the second sketch image information to be trained to obtain intermediate synthetic image information; input the intermediate synthetic image information into the second generator to obtain the second synthetic image information corresponding to the second video image information; determine the model loss value based on the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information and the second synthetic image information; if the model loss value does not meet the preset model convergence condition, update the parameters of the first generator and / or the parameters of the second generator based on the model loss value, and iteratively train the first generator and the second generator after the updated parameters until the model loss value meets the preset model convergence condition. Wherein, the first video image information can be used to represent the first frame of video image in two consecutive frames of video images extracted from the training data set; the second video image data can represent the second frame of video image in two consecutive frames of video images extracted from the training data set, that is, the second frame of video image is the previous frame of video image of the first frame of video image.

[0073] Reference Figure 3 , shows a flowchart of the steps of an image processing method provided by an optional embodiment of the present application. Specifically, the image processing method provided by the embodiment of the present application may include the following steps in the model training stage:

[0074] Step 310: Acquire video image information to be trained from a preset training data set.

[0075] Step 320: Determine the sketch image information to be trained based on the video image information to be trained, wherein the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained and second sketch image information to be trained.

[0076] Among them, the first sketch information to be trained may refer to the sketch information corresponding to the first video image information, which can be specifically used to represent the sketch of the first video image; the second sketch information to be trained may refer to the sketch information corresponding to the second video image information, which can be specifically used to represent the sketch of the second video image. Specifically, in the model training stage, this embodiment can extract one or more frames of video image information from the training data set as training video image information; and for each frame of video image information extracted, facial key point information and Canny line information can be extracted, that is, facial key point information and Canny line information are extracted from each frame of training video image information obtained, so as to generate corresponding sketch information to be trained based on the extracted facial key point information and Canny line information.

[0077] For example, after extracting the real picture Rt of the tth frame in the video from the training data set, the open source key point extraction method can be used to extract 68 key points and Canny lines of the real picture of the tth frame, that is, extract the facial key point information and Canny line information, and then the extracted facial key point information and Canny line information can be used to generate the corresponding sketch image information to be trained, that is, use these 68 key points to draw facial lines to combine the facial lines with the Canny lines to generate the corresponding sketch image St; for another example, after extracting the real picture R(t+1) of the (t+1)th frame in the video from the training data set, the open source key point extraction method can be used to extract 68 key points and Canny lines of the real picture of the (t+1)th frame, and then the 68 key points can be used to draw facial lines to combine the facial lines with the Canny lines to generate the corresponding sketch image S(t+1).

[0078] Step 330 , generating first mouth region information corresponding to the first sketch image information to be trained and second mouth region information corresponding to the second sketch image information to be trained respectively through a pre-constructed first generator.

[0079] In actual processing, the embodiment of the present application can input the to-be-trained sketch image information corresponding to each real image into the pre-constructed first generator to generate the corresponding mouth area information through the first generator. Further, the embodiment of the present application generates the first mouth area information corresponding to the first to-be-trained sketch image information and the second mouth area information corresponding to the second to-be-trained sketch image information respectively through the pre-constructed first generator, which can specifically include inputting the first to-be-trained sketch image information into the pre-constructed first generator to obtain the first mouth area information; and inputting the second to-be-trained sketch image information into the pre-constructed first generator to obtain the second mouth area information.

[0080] As an example of the present application, in the model training stage, a generator may be first constructed, such as the architecture of mobileNetV2, to serve as the first generator G1 for generating mouth region information. The input of the first generator G1 may be sketch information, and the output of the first generator G1 may be mouth region information. For example, after obtaining the sketch St of the t-th frame video image of the video, the sketch St of the t-th frame video image of the video can be used as the first sketch information to be trained and input into the first generator G1, so that the first generator G1 generates the mouth area information Mt of the t-th frame based on the sketch St of the t-th frame video image and outputs it as the first mouth area information corresponding to the first sketch information to be trained; similarly, after obtaining the sketch of the next frame of video image after the t-th frame video image of the video, that is, after obtaining the sketch S(t+1) of the (t+1)-th frame video image of the video, the sketch S(t+1) of the (t+1)-th frame video image of the video can be used as the second sketch information to be trained and input into the first generator G1, so that the first generator G1 generates the mouth area information M(t+1) of the (t+1)-th frame based on the sketch S(t+1) of the (t+1)-th frame video image and outputs it as the second mouth area information corresponding to the second sketch information to be trained.

[0081] Step 340 , performing data synthesis based on the first mouth region information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained, to obtain initial synthesized image information.

[0082] In actual processing, the embodiment of the present application can create an all-zero image of the same size as the real image based on the real image, as the all-zero image information corresponding to the sketch image to be trained for model training. Further, after obtaining the first mouth area information output by the first generator G1, the embodiment of the present application performs data synthesis based on the first mouth area information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained to obtain initial synthesized image information, which can specifically include: based on the first mouth area information, updating the mouth area information in the first sketch image information to be trained to obtain updated target training sketch image information; based on the first mouth area information, updating the mouth area information in the all-zero image information to obtain target training initial image information; synthesizing the target training sketch image information and the target training initial image information to obtain the initial synthesized image information.

[0083] For example, in combination with the above example, when the image size of the real image Rt corresponding to the first sketch image information to be trained is 512*512*3, a picture Zt with an all-zero image of the same size as the t-th frame image and a channel number of 3 can be created as the all-zero image information corresponding to the first sketch image information to be trained. After the first generator G1 generates and outputs the mouth area information Mt of the t-th frame, that is, after the first generator G1 generates and outputs the first mouth area information, the mouth area information St[:,:,Ymin:Ymax,Xmin:Xmax] in the sketch image St of the t-th frame video image can be updated based on the mouth area information Mt generated and output by the first generator G1, that is, the mouth area information in the first sketch image information to be trained is updated based on the first mouth area information, so as to update the mouth area information St[:,:,Ymin:Ymax,Xmin:Xmax] in the sketch image St of the t-th frame video image to the mouth area information Mt generated and output by the first generator G1, that is, St[:,:,Ymin:Ymax,Xmin:Xmax]=Mt, so that the updated first sketch image information to be trained can be obtained, and the updated first sketch image information to be trained can be used as the target training sketch image information; At the same time, based on the mouth area information Mt generated and output by the first generator G1, the mouth area information Zt[:,:,Ymin:Ymax,Xmin:Xmax] in the all-zero image information created in advance according to the real image Rt can be updated to update the mouth area information Zt[:,:,Ymin:Ymax,Xmin:Xmax] in the all-zero image information to the mouth area information Mt generated and output by the first generator G1, that is, Zt[:,:,Ymin:Ymax,Xmin:Xmax]=Mt, so that the updated initial image information can be obtained, and the updated initial image information can be used as the target training initial image information, so as to merge the target training initial image information with the target training sketch image information. For example, the target training initial image information and the target training sketch image information can be merged in a channel merging manner to obtain a merged image It as the initial synthetic image information. Among them, the size of the merged image It can be 512*512*4, and the merged image It can be used as the initial synthetic image information and input into the second generator G2, so that the second generator G2 can generate the area outside the mouth based on the merged image It while retaining the information of the mouth generated by the first generator.

[0084] Step 350: input the initial composite image information into a second generator to obtain first composite image information corresponding to the first video image information.

[0085] Specifically, in the training phase, the embodiment of the present application may input the initial synthetic image information synthesized by the target training sketch image information and the target training initial image information into the second generator G2, so as to generate the first synthetic image information corresponding to the first video image information through the second generator G2. The first synthetic image information may represent the virtual object image generated by the second generator G2 for the sketch image of the first video image information.

[0086] Step 360, performing data synthesis based on the first synthetic image information, the second mouth area information and the second sketch image information to be trained to obtain intermediate synthetic image information.

[0087] Specifically, after obtaining the first composite image information corresponding to the first video image information output by the second generator G and the second mouth area information output by the second generator G1, the embodiment of the present application can update the mouth area information of the first composite image information and the mouth area information in the second sketch image information to be trained based on the second mouth area information, so as to update the mouth area information of the first composite image information and the mouth area information in the second sketch image information to be trained to the second mouth area information, respectively, to obtain the updated first composite image information and the updated second sketch image information to be trained, and can merge the updated first composite image information and the updated second sketch image information to be trained to obtain the intermediate composite image information. The intermediate composite image information can represent the image generated by merging the updated first composite image information and the updated second sketch image information to be trained.

[0088] Step 370: input the intermediate synthetic image information into the second generator to obtain second synthetic image information corresponding to the second video image information.

[0089] Specifically, the embodiment of the present application can input the intermediate synthetic image information generated after merging into the second generator G2, so that the second generator G2 can generate the second synthetic image information according to the intermediate synthetic image information. The second synthetic image information can represent the virtual object image generated by the second generator G2 for the sketch of the second video image information, that is, the second synthetic image information can be used as the synthetic image information corresponding to the second video image information.

[0090] For example, in combination with the above example, after the merged image It as the initial synthetic image information is input into the second generator G2, the virtual object image Ft generated and output by the second generator G2 according to the merged image It can be obtained, and the virtual object image Ft can be used as the first synthetic image information corresponding to the first video image information, so as to merge the virtual object image Ft of the tth frame generated by the second generator G2 with the mouth area information M(t+1) of the (t+1)th frame generated by the first generator G1, so as to use the mouth area information M(t+1) of the (t+1)th frame generated by the first generator G1 to update the mouth area information Ft[:,:,Ymin:Ymax,Xmin:Xmax] in the virtual object image Ft of the tth frame, so that the mouth area information Ft[:,:,Ymin:Ymax,Xmin:Xmax] in the virtual object image Ft of the tth frame can be updated to the second synthetic image information generated and output by the first generator G1. Mouth area information M(t+1), that is, Ft[:,:,Ymin:Ymax,Xmin:Xmax]=M(t+1), obtains the updated first synthetic image information; at the same time, the first generator G1 can be used to generate and output the mouth area information M(t+1) of the (t+1)th frame to update the mouth area information S(t+1)[:,:,Ymin:Ymax,Xmin:Xmax] in the sketch image S(t+1) of the (t+1)th frame video image, so as to update the mouth area information S(t+1)[:,:,Ymin:Ymax,Xmin:Xmax] in the sketch image S(t+1) of the (t+1)th frame video image to the second mouth area information M(t+1) generated and output by the first generator G1, that is, S(t+1)[:,:,Ymin:Ymax,Xmin:Xmax]=M(t+1), thereby obtaining the updated second sketch image information to be trained. Subsequently, the updated second sketch image information to be trained can be merged with the updated first synthetic image information, so that the merged image information can be input into the second generator G2 as the intermediate synthetic image information I(t+1), so that the second generator G2 generates and outputs the virtual object image F(t+1) according to the intermediate synthetic image information I(t+1) as the second synthetic image information corresponding to the second video image information.

[0091] Step 380, determining a model loss value based on the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information and the second synthetic image information.

[0092] Specifically, after obtaining the second synthetic image information corresponding to the second video image information, the embodiment of the present application can combine the first sketch information to be trained, the second sketch information to be trained, the first synthetic image information and the second synthetic image information, and input them into the discriminator to obtain N values, and then use these N values ​​to determine the loss value of the second generator and the loss value of the overall discriminator; and can combine the mouth area information of the first sketch information to be trained, the mouth area information of the second sketch information to be trained, the first mouth area information and the second mouth area information, and input them into the discriminator to obtain K values ​​output by the discriminator, and then use these K values ​​to determine the loss value of the first generator; thereby, the loss value of the first generator, the loss value of the second generator and the loss value of the overall discriminator can be used for calculation to obtain the model loss value. Among them, N and K can both be integers approximately 1, and N and K can be the same integer, such as N and K can both be 2 or 3, etc., and this implementation does not impose any specific restrictions on this; the model loss value can refer to the loss value of the training model, which can be specifically used to determine whether the training model has converged, and then when the training model has converged, the converged model can be determined as the image processing model, so that the subsequent application reasoning stage can use the image processing model to generate a virtual object image corresponding to the image to be processed.

[0093] Of course, the embodiments of the present application may also adopt other methods to determine the loss value of the first generator, such as respectively calculating the perceptual loss between the mouth area information of the first sketch information to be trained and the first mouth area information, and the perceptual loss between the mouth area information of the second sketch information to be trained and the second mouth area information, so as to determine the loss value of the first generator by the perceptual loss between the mouth area information of the first sketch information to be trained and the first mouth area information, and the perceptual loss between the mouth area information of the second sketch information to be trained and the second mouth area information. The embodiments of the present application are not limited to this. In addition, the embodiments of the present application may also use other methods to determine the loss value of the second generator or the loss value of the overall discriminator. For example, the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information may be combined and input into the discriminator to determine the loss value of the overall discriminator by calculating the distance between the N values ​​output by the discriminator and 1; and / or, the perceptual loss (Perceptual Loss) between the first synthetic image information and the first video image information, and the Perceptual Loss between the second synthetic image information and the second video image information may be respectively determined, and then these two Perceptual Losses may be added to obtain the loss value of the overall generator, and the embodiments of the present application are not limited to this.

[0094] Furthermore, the embodiment of the present application determines the model loss value based on the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information and the second synthetic image information, which may specifically include: inputting the first sketch image information to be trained, the second sketch image information to be trained, the first video image information and the second video image information into the discriminator of the image processing model to obtain the first loss information output by the discriminator; inputting the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information and the second synthetic image information into the discriminator to obtain the second loss information output by the discriminator; inputting the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, and the second synthetic image information into the discriminator to obtain the second loss information output by the discriminator; inputting the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, and the second synthetic image information in the second synthetic image information into the discriminator to obtain the second loss information output by the discriminator; The mouth area information in the first to-be-trained sketch information, the first mouth area information and the second mouth area information are input into the discriminator to obtain the third loss information output by the discriminator; the mouth area information in the first to-be-trained sketch information, the mouth area information in the second to-be-trained sketch information, the mouth area information in the first video image information and the mouth area information in the second video image information are input into the discriminator to obtain the fourth loss information output by the discriminator; based on the first loss information and the second loss information, the loss value of the discriminator and the loss value of the second generator are determined; based on the third loss information and the fourth loss information, the loss value of the first generator is determined; according to the loss value of the discriminator, the loss value of the second generator and the loss value of the first generator, the model loss value is determined.

[0095] As an example of the present application, when the image processing model obtained after training is used to infer and generate digital human images, the model loss value Loss can mainly include four parts, namely, the loss value loss1 of the digital human overall discriminator, the loss value loss2 of the digital human overall generator, the loss value loss3 of the digital human mouth area generator and the loss value loss4 of the digital human mouth area discriminator.

[0096] Among them, the loss value loss1 of the digital human overall discriminator can be determined by calculating the loss of the probability of a false image and / or the loss of the probability of a true image. For example, the first sketch information to be trained, the second sketch information to be trained, the first synthetic image information and the second synthetic image information can be merged to merge the sketch St of the t-th frame video image, the sketch S(t+1) of the (t+1)-th frame video image, the digital human image Ft generated in the t-th frame and the virtual object image F(t+1) generated in the (t+1)-th frame into one image in a channel synthesis manner, and input it into the discriminator D to obtain N values, and then the preset loss function L1 can be used. loss calculates the distance between these N values ​​and 0, so as to determine the calculated distance value as the loss of the probability of a false image. The smaller the calculated distance value is, the better the discriminator can judge the false image. For example, the first sketch information to be trained, the second sketch information to be trained, the first video image information and the second video image information can be merged to merge the sketch St of the t-th frame video image, the sketch S(t+1) of the (t+1)-th frame video image, the t-th frame real image Rt and the (t+1)-th frame real image R(t+1) into one picture in a channel synthesis manner, and input it into the discriminator D to obtain N values ​​as the first loss information, and then L1 can be used. loss calculates the distance between the N value and 1 to determine the calculated distance value as the loss of the probability of a true image. The smaller the calculated distance value is, the more capable the discriminator is of judging a true image. After determining the loss of the probability of a false image and the loss of the probability of a true image, the loss of the probability of a false image and the loss of the probability of a true image can be accumulated to determine the accumulated result as the loss value loss1 of the overall discriminator of the digital human.

[0097] The loss value loss2 of the digital human overall generator is used as the loss value of the second generator, which can be specifically determined by the second loss information output by the discriminator. For example, in a channel synthesis manner, the sketch image St of the t-th frame video image, the sketch image S(t+1) of the (t+1)-th frame video image, the digital human image Ft generated by the t-th frame, and the virtual object image F(t+1) generated by the (t+1)-th frame are combined into one picture, and input into the discriminator D to obtain N values, which can then be used as the second loss information to calculate the distance between the N values ​​and 1 using the preset loss function L1 loss, so that the calculated distance value can be used as the first loss function value of the digital human overall generator, so that the loss value loss2 of the digital human overall generator can be determined based on the first loss function value, so that the loss value loss2 of the digital human overall generator can include the first loss function value calculated using the loss function L1 loss.

[0098] Of course, the loss value loss2 of the digital human overall generator may include other loss function values ​​in addition to the first loss function value calculated using the loss function L1loss, such as the second loss function value, feature matching loss value (Feature Matching Loss), etc. This example does not impose specific restrictions on this. Among them, the second loss function value can be determined by calculating the perceptual loss (PerceptualLoss) between the generated virtual object image and the real image. For example, the digital human image Ft generated at the tth frame and the real image Rt at the tth frame can be input into the VGG network model to calculate the feature values ​​of the digital human image Ft and the real image Rt at different layers of the VGG network model to obtain the distance value, so that the distance value can be used as the perceptual loss PL1 between the digital human image Ft and the real image Rt; similarly, the digital human image Ft generated at the (t+1)th frame can be used. The virtual object image F(t+1) and the (t+1)th frame real image R(t+1) are input into the VGG network model to obtain the perceptual loss PL2 between the virtual object image F(t+1) generated at the (t+1)th frame and the real image R(t+1) at the (t+1)th frame by calculating the feature values ​​of the virtual object image F(t+1) and the real image R(t+1) at different layers of the VGG network model. Then, the perceptual loss PL1 and the perceptual loss PL2 are added to obtain the entropy result as the second loss function value of the digital human overall generator. In addition, the feature matching loss value of the digital human overall generator can be obtained by inputting the sketch image St of the tth frame video image, the sketch image S(t+1) of the (t+1)th frame video image, the digital human image Ft generated at the tth frame, and the virtual object image F(t+1) generated at the (t+1)th frame into the discriminator for calculation.

[0099] The loss value loss4 of the digital human mouth area discriminator can be determined by calculating the loss of the probability of a fake mouth image and the loss of the probability of a real mouth image. For example, the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the first mouth area information and the second mouth area information can be merged to combine the mouth area information SMt of the sketch image of the t-th frame video image, the mouth area information SM(t+1) of the sketch image of the (t+1)-th frame video image, the first mouth area information Mt generated and output by the first generator G1 of the t-th frame, and the second mouth area information Mt(t+1) generated and output by the (t+1)-th frame into one picture in a channel synthesis manner, and input it into the discriminator D to obtain N values ​​as the third loss information, and then the preset loss function L1 can be used. loss calculates the distance between these N values ​​and 0, so as to determine the calculated distance value as the loss of the probability of a fake mouth picture. The smaller the calculated distance value is, the better the discriminator can judge the fake mouth picture. For example, the mouth area information in the first sketch information to be trained, the mouth area information in the second sketch information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information can be merged to merge the mouth area information SMt of the sketch of the t-th frame video image, the mouth area information SM(t+1) of the sketch of the (t+1)-th frame video image, the mouth area information RMt of the t-th frame real picture, and the mouth area information RM(t+1) of the (t+1)-th frame real picture into one picture in a channel synthesis manner, which is input into the discriminator D to obtain N values ​​as the fourth loss information, and then L1 can be used. loss calculates the distance between the N value and 1 to determine the calculated distance value as the loss of the probability of a real mouth picture. The smaller the calculated distance value is, the more capable the digital human mouth area discriminator is in identifying a real mouth picture. After determining the loss of the probability of a fake mouth picture and the loss of the probability of a real mouth picture, the loss of the probability of a fake mouth picture and the loss of the probability of a real mouth picture can be accumulated to determine the accumulated result as the loss value loss4 of the digital human mouth area discriminator.

[0100] The loss value loss3 of the digital human mouth area generator is used as the loss value of the first generator, which can be specifically determined by the third loss information and the fourth loss information. For example, in the case of constructing a digital human mouth area generator based on a residual algorithm, the loss value loss3 of the digital human mouth area generator can be the loss of the digital human mouth residual generator, which can specifically include two loss function values, which can be the third loss function value and the fourth loss function value respectively. Among them, the third loss function value can be determined by the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the first mouth area information and the second mouth area information. For example, the mouth area information SMt of the sketch image of the t-th frame video image, the mouth area information SM(t+1) of the sketch image of the (t+1)-th frame video image, the first mouth area information Mt generated and output by the first generator G1 of the t-th frame, and the second mouth area information Mt(t+1) generated and output by the (t+1)-th frame can be combined and input into the discriminator D to obtain N values ​​as the third loss information, and then the preset loss function L1 can be used. loss calculates the distance between these N values ​​and 1, so as to determine the calculated distance value as the third loss function value; the fourth loss function value can be the perceptual loss, such as the perceptual loss PL3 between the first mouth area information Mt and the mouth area information RMt of the real picture, and the perceptual loss PL4 between the second mouth area information M(t+1) and the mouth area information RM(t+1) of the real picture can be calculated respectively, and then the perceptual loss PL3 and the perceptual loss PL4 can be added to obtain the entropy, and the entropy result of the fourth loss function value as the digital human mouth area generator can be obtained, so that the loss value loss4 of the digital human mouth area discriminator can be determined according to the fourth loss function value and the third loss function value, and then the loss value loss4 of the digital human mouth area discriminator, the loss value loss1 of the digital human overall discriminator, the loss value loss2 of the digital human overall generator and the loss value loss3 of the digital human mouth area generator can be accumulated to obtain the current model loss value Loss.

[0101] Step 390: If the model loss value does not meet the preset model convergence conditions, the parameters of the first generator and / or the parameters of the second generator are updated based on the model loss value, and the first generator and the second generator with updated parameters are iteratively trained until the model loss value meets the preset model convergence conditions.

[0102] Specifically, after obtaining the model loss value, the embodiment of the present application can determine whether the model obtained by the current training has converged by judging whether the model loss value meets the preset model convergence condition, so that the trained model can be determined as the image processing model when the training model converges, so that the subsequent application reasoning stage can use the image processing model to generate a virtual object image corresponding to the image to be processed. For example, when the model loss value is greater than the convergence loss threshold in the model convergence condition, it can be determined that the current model loss value does not meet the preset model convergence condition, and it is necessary to continue model training. Subsequently, the back propagation algorithm can be used to update the parameters of the first generator and / or the parameters of the second generator based on the model loss value, so as to iteratively train the first generator and the second generator after updating the parameters until the model loss value is equal to or less than the convergence loss threshold in the model convergence bar.

[0103] After the image processing model is trained, the embodiment of the present application can use the image processing model to generate virtual images such as digital humans using two-frame image reasoning, thereby improving the mouth shape generation effect of the virtual image.

[0104] In actual processing, the image frame in the video can be used as the image to be processed in the embodiment of the present application, and the previous frame of the image frame currently being processed can be used as reference image information to generate a virtual image using two frames of image information, thereby improving the stability of the video generation frame and the mouth shape generation effect under the premise of ensuring the reasoning speed. For example, when the current image to be processed is the first frame of the video, a picture Zt with an all-zero picture of the same size as the first frame and a channel number of 3 can be created, and the picture Zt can be used as the reference image information of the first frame, so that the reference image information can be used to generate a virtual image corresponding to the first frame of the image, so as to achieve the purpose of generating a virtual image using two frames of images; when the current image to be processed is not the first frame of the video, such as when the current image to be processed is the second or third frame of the video, the previous frame of the image to be processed can be determined as the reference image information, so that the previous frame of the image and the image to be processed can be used to generate a virtual image of the current frame, thereby improving the mouth shape generation effect of the virtual image and improving the stability of the video frame.

[0105] Furthermore, before determining the target composite image information based on the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed, the image processing method in the embodiment of the present application may also include: if the image to be processed is the first frame image of the video, creating all-zero image information based on the image to be processed, and determining the all-zero image information as the reference image information corresponding to the image to be processed; if the image to be processed is not the first frame image of the video, determining the reference frame target image information as the reference image information corresponding to the image to be processed, and the reference frame target image information is the target image information corresponding to the previous frame image of the image to be processed.

[0106] Reference Figure 4 , shows a flowchart of the steps of an image processing method provided by an optional embodiment of the present application. Specifically, the image processing method provided by the embodiment of the present application may include the following steps in the model application reasoning stage:

[0107] Step 410, obtaining sketch information of the image to be processed.

[0108] Step 420: input the sketch image information into a pre-trained first generator to obtain the target mouth area information output by the first generator.

[0109] Step 430, based on the target mouth area information, respectively update the mouth area information in the reference image information and the mouth area information in the sketch information to obtain updated first reference image information and first sketch information.

[0110] Step 440, synthesizing the first reference image information and the first sketch image information to obtain the target synthesized image information.

[0111] Step 450: input the target composite image information into a pre-trained second generator, and determine the image information output by the second generator as the target image information corresponding to the image to be processed.

[0112] Among them, the second generator is a generator trained jointly with the first generator.

[0113] As an example of the present application, in a digital human video generation scenario, after the sketch information St of the image to be processed in the current frame is input into the first generator G1 and the target mouth area information Mt output by the first generator G1 is obtained, new data can be combined based on the target mouth area information Mt to update the mouth area information in the reference image information and the mouth area information in the sketch information based on the target mouth area information Mt, respectively. The mouth area information in the reference image information is updated to the target mouth area information to obtain the updated first reference image information and the first sketch information, so that the updated first reference image information and the first sketch information can be used to generate the corresponding digital human image through the second generator G2, and then the digital human image can be used as the target image information, so that the corresponding digital human video can be generated and output using multiple frames of digital human images.

[0114] Specifically, when the sketch information St of the image to be processed is the sketch information of the first frame image in the video, the pre-created all-zero image information Zt can be used as the reference image information, so that the mouth area information Zt[:,:,Ymin:Ymax,Xmin:Xmax] in the all-zero image information can be updated based on the target mouth area information Mt, so as to update the mouth area information Zt[:,:,Ymin:Ymax,Xmin:Xmax] in the all-zero image information to the target mouth area information Mt, that is, Zt[:,:,Ymin:Ymax,Xmin:Xmax]=Mt; and the mouth area information St[: ,:,Ymin:Ymax,Xmin:Xmax] is updated to update the mouth area information St[:,:,Ymin:Ymax,Xmin:Xmax] in the tracing information of the first frame image to the target mouth area information Mt, that is, St[:,:,Ymin:Ymax,Xmin:Xmax]=Mt, and then the updated reference image information Zt (that is, the first reference image information) is merged with the updated first sketch image information St to merge the updated reference image and the updated sketch image to obtain the target composite image information It, and then the target composite image information It is input into the second generator G2 to generate the first frame as the target image information of the digital human image Ft.

[0115] After generating the digital human image Ft of the first frame as the target image information, the target mouth area information M(t+1) of the second frame generated by the first generator G1 can be obtained to combine new data based on the target mouth area information M(t+1) of the second frame. Specifically, the mouth area information Ft[:,:,Ymin:Ymax,Xmin:Xmax] in the digital human image of the first frame as the reference image information and the mouth area information S(t+1)[:,:,Ymin:Ymax,Xmin:Xmax] of the sketch information of the second frame are both updated to the target mouth area information M(t+1) and, that is, Ft[:,:,Ymin:Ymax,Xmin:Xmax]=M(t+1), S(t+1)[:,:,Ymin:Ymax,Xmin:Xmax]=M(t+1). Subsequently, the updated digital human image Ft can be merged with the updated sketch information S(t+1) of the second frame to merge the updated reference image and the updated sketch to obtain the target composite image information I(t+1), and the target composite image information I(t+1) can be input into the second generator G2 to generate the digital human image F(t+1) as the target image information of the second frame… and so on, the digital human images of other frames of the video frame can be generated, and then the digital human video can be generated based on the multi-frame digital human images generated by reasoning, thereby realizing the generation of the digital human video.

[0116] In summary, the embodiment of the present application adds an additional first generator on the basis of the second generator, so as to learn to generate the mouth area through the first generator, so that the second generator can generate information of areas other than the mouth while retaining the learning of the mouth area generated by the first generator, thereby improving the accuracy of the mouth area information in the final generated target image information, that is, improving the clarity around the mouth of the virtual object in the image, and further, on the premise of ensuring the generation effect and inference speed of the digital human, it is possible to improve the stability of the video generation frame and improve the mouth shape generation effect of the video frame.

[0117] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.

[0118] The present application also provides an image processing device. Figure 5 As shown, the image processing device 500 provided in the embodiment of the present application may include the following modules:

[0119] A contour image acquisition module 510 is used to acquire the sketch image information of the image to be processed;

[0120] A target mouth region determination module 520, configured to input the sketch image information into a pre-trained first generator to obtain target mouth region information output by the first generator;

[0121] A target composite image determination module 530 is used to determine target composite image information according to the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed;

[0122] The target image determination module 540 is used to input the target composite image information into a pre-trained second generator, and determine the image information output by the second generator as the target image information corresponding to the image to be processed. The second generator is a generator trained jointly with the first generator.

[0123] Optionally, the target composite image determination module 530 in the embodiment of the present application may include:

[0124] An updating submodule, used for updating the mouth area information in the reference image information and the mouth area information in the sketch image information respectively based on the target mouth area information, so as to obtain updated first reference image information and first sketch image information;

[0125] A synthesis submodule is used to perform synthesis based on the first reference image information and the first sketch image information to obtain the target synthesized image information.

[0126] Optionally, the image processing device 500 further includes the following modules:

[0127] An all-zero image information module, used for creating all-zero image information based on the image to be processed when the image to be processed is the first frame of the video, and determining the all-zero image information as reference image information corresponding to the image to be processed;

[0128] The reference image information module is used to determine the reference frame target image information as the reference image information corresponding to the image to be processed when the image to be processed is not the first frame image of the video, and the reference frame target image information is the target image information corresponding to the previous frame image of the image to be processed.

[0129] Optionally, the image processing device 500 further includes the following modules:

[0130] A training video image information acquisition module is used to acquire the video image information to be trained from a preset training data set;

[0131] A model training module is used to perform model training based on the video image information to be trained to obtain an image processing model including the first generator and the second generator.

[0132] Optionally, the model training module may include the following submodules:

[0133] A sketch image information determination submodule to be trained, used to determine the sketch image information to be trained based on the video image information to be trained, the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information;

[0134] A mouth region generating submodule, used to generate first mouth region information corresponding to the first to-be-trained sketch image information and second mouth region information corresponding to the second to-be-trained sketch image information respectively through a pre-constructed first generator;

[0135] An initial synthetic image submodule, used to perform data synthesis based on the first mouth area information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained, to obtain initial synthetic image information;

[0136] A first synthetic image submodule, used for inputting the initial synthetic image information into the second generator to obtain first synthetic image information corresponding to the first video image information;

[0137] An intermediate synthetic image submodule, used for performing data synthesis based on the first synthetic image information, the second mouth region information and the second sketch image information to be trained to obtain intermediate synthetic image information;

[0138] A second synthetic image submodule, used for inputting the intermediate synthetic image information into the second generator to obtain second synthetic image information corresponding to the second video image information;

[0139] A model loss value determination submodule, used to determine a model loss value based on the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information;

[0140] The parameter update training submodule is used to update the parameters of the first generator and / or the parameters of the second generator based on the model loss value when the model loss value does not meet the preset model convergence condition, and iteratively train the first generator and the second generator after the updated parameters until the model loss value meets the preset model convergence condition.

[0141] Optionally, the mouth area generation submodule is specifically used to input the first sketch image information to be trained into the pre-built first generator to obtain the first mouth area information; and input the second sketch image information to be trained into the pre-built first generator to obtain the second mouth area information.

[0142] Optionally, the initial composite graph submodule includes the following units:

[0143] A first updating unit, configured to update the mouth region information in the first to-be-trained sketch image information based on the first mouth region information, to obtain updated target training sketch image information;

[0144] A second updating unit, configured to update the mouth region information in the all-zero image information based on the first mouth region information to obtain target training initial image information;

[0145] The initial image synthesis unit is used to synthesize the target training sketch image information and the target training initial image information to obtain the initial synthesized image information.

[0146] Optionally, the model loss value determination submodule includes the following units:

[0147] a first loss information determining unit, configured to input the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information into a discriminator of the image processing model, and obtain first loss information output by the discriminator;

[0148] A second loss information determining unit, configured to input the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information into the discriminator to obtain second loss information output by the discriminator;

[0149] a third loss information determining unit, configured to input the mouth region information in the first sketch image information to be trained, the mouth region information in the second sketch image information to be trained, the first mouth region information, and the second mouth region information into the discriminator, and obtain third loss information output by the discriminator;

[0150] a fourth loss information determining unit, configured to input the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information into the discriminator, and obtain fourth loss information output by the discriminator;

[0151] A first loss value determining unit, configured to determine a loss value of the discriminator and a loss value of the second generator based on the first loss information and the second loss information;

[0152] A second loss value determining unit, configured to determine a loss value of the first generator based on the third loss information and the fourth loss information;

[0153] A model loss value determining unit is used to determine the model loss value based on the loss value of the discriminator, the loss value of the second generator and the loss value of the first generator.

[0154] It should be noted that the image processing device provided above can execute the image processing method provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0155] In a specific implementation, the above-mentioned image processing device can be applied to electronic devices such as personal computers and servers, so that the electronic device can perform image processing based on the acquired sketch information of the image to be processed as an image processing device, thereby improving the generation effect of the mouth shape and the overall generation effect of the image while ensuring the reasoning speed. Furthermore, an embodiment of the present application also provides an image processing device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to execute the program stored in the memory, and implement the steps of the image processing method described in any of the above method embodiments.

[0156] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method described in any one of the above method embodiments are implemented.

[0157] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device, equipment, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] In this article, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0159] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. An image processing method, characterized in that: include: Get the sketch information of the image to be processed; Inputting the sketch image information into a pre-trained first generator to obtain target mouth area information output by the first generator; Determining target composite image information according to the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed; Inputting the target synthetic image information into a pre-trained second generator, and determining the image information output by the second generator as the target image information corresponding to the image to be processed, wherein the second generator is a generator jointly trained with the first generator; Before obtaining the sketch information of the image to be processed, the method further includes: Obtaining video image information to be trained from a preset training data set; Performing model training according to the video image information to be trained to obtain an image processing model including the first generator and the second generator; The performing model training according to the video image information to be trained includes: Determine the sketch image information to be trained based on the video image information to be trained, the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information; Generate first mouth region information corresponding to the first sketch image information to be trained and second mouth region information corresponding to the second sketch image information to be trained respectively through a pre-constructed first generator; Performing data synthesis based on the first mouth region information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained to obtain initial synthesized image information; Inputting the initial composite image information into the second generator to obtain first composite image information corresponding to the first video image information; Performing data synthesis based on the first synthetic image information, the second mouth region information and the second sketch image information to be trained to obtain intermediate synthetic image information; Inputting the intermediate synthetic image information into the second generator to obtain second synthetic image information corresponding to the second video image information; Determine a model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information; If the model loss value does not meet the preset model convergence conditions, the parameters of the first generator and / or the parameters of the second generator are updated based on the model loss value, and the first generator and the second generator with updated parameters are iteratively trained until the model loss value meets the preset model convergence conditions.

2. The image processing method according to claim 1, characterized in that: The step of determining target composite image information based on the target mouth area information, the sketch image information, and reference image information corresponding to the image to be processed includes: Based on the target mouth area information, respectively updating the mouth area information in the reference image information and the mouth area information in the sketch information to obtain updated first reference image information and first sketch information; The target synthesized image information is obtained by synthesizing the first reference image information and the first sketch image information.

3. The image processing method according to claim 1 or 2, characterized in that: Before determining target composite image information based on the target mouth area information, the sketch image information, and the reference image information corresponding to the image to be processed, the image processing method further includes: If the image to be processed is the first frame of the video, creating all-zero image information based on the image to be processed, and determining the all-zero image information as reference image information corresponding to the image to be processed; If the image to be processed is not the first frame of the video, the reference frame target image information is determined as the reference image information corresponding to the image to be processed, and the reference frame target image information is the target image information corresponding to the previous frame of the image to be processed.

4. The image processing method according to claim 1, characterized in that: The generating, by respectively using a pre-constructed first generator, first mouth region information corresponding to the first to-be-trained sketch image information and second mouth region information corresponding to the second to-be-trained sketch image information comprises: Inputting the first sketch image information to be trained into the pre-built first generator to obtain the first mouth region information; The second sketch image information to be trained is input into the pre-built first generator to obtain the second mouth area information.

5. The image processing method according to claim 4, characterized in that: The step of synthesizing data based on the first mouth region information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained to obtain initial synthesized image information includes: Based on the first mouth region information, the mouth region information in the first to-be-trained sketch image information is updated to obtain updated target training sketch image information; Based on the first mouth region information, the mouth region information in the all-zero image information is updated to obtain target training initial image information; The target training sketch image information and the target training initial image information are synthesized to obtain the initial synthesized image information.

6. The image processing method according to claim 4, characterized in that: The determining of the model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information and the second synthetic image information includes: Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first video image information, and the second video image information into a discriminator of the image processing model to obtain first loss information output by the discriminator; Inputting the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information into the discriminator to obtain second loss information output by the discriminator; Inputting the mouth region information in the first sketch image information to be trained, the mouth region information in the second sketch image information to be trained, the first mouth region information, and the second mouth region information into the discriminator to obtain third loss information output by the discriminator; Inputting the mouth area information in the first sketch image information to be trained, the mouth area information in the second sketch image information to be trained, the mouth area information in the first video image information, and the mouth area information in the second video image information into the discriminator to obtain fourth loss information output by the discriminator; Based on the first loss information and the second loss information, determining a loss value of the discriminator and a loss value of the second generator; Determining a loss value of the first generator based on the third loss information and the fourth loss information; The model loss value is determined according to the loss value of the discriminator, the loss value of the second generator, and the loss value of the first generator.

7. An image processing device, characterized in that: include: A contour image acquisition module is used to obtain the sketch image information of the image to be processed; A target mouth region determination module, used for inputting the sketch image information into a pre-trained first generator to obtain target mouth region information output by the first generator; A target composite image determination module, used to determine target composite image information according to the target mouth area information, the sketch image information and the reference image information corresponding to the image to be processed; a target image determination module, configured to input the target composite image information into a pre-trained second generator, and determine the image information output by the second generator as the target image information corresponding to the image to be processed, wherein the second generator is a generator jointly trained with the first generator; Before obtaining the sketch information of the image to be processed, obtaining the video image information to be trained from a preset training data set; performing model training based on the video image information to be trained to obtain an image processing model including the first generator and the second generator; The performing model training according to the video image information to be trained includes: Determine the sketch image information to be trained based on the video image information to be trained, the video image information to be trained includes first video image information and second video image information, and the sketch image information to be trained includes first sketch image information to be trained corresponding to the first video image information and second sketch image information to be trained corresponding to the second video image information; Generate first mouth region information corresponding to the first sketch image information to be trained and second mouth region information corresponding to the second sketch image information to be trained respectively through a pre-constructed first generator; Performing data synthesis based on the first mouth region information, the first sketch image information to be trained, and the all-zero image information corresponding to the sketch image information to be trained to obtain initial synthesized image information; Inputting the initial composite image information into the second generator to obtain first composite image information corresponding to the first video image information; Performing data synthesis based on the first synthetic image information, the second mouth region information and the second sketch image information to be trained to obtain intermediate synthetic image information; Inputting the intermediate synthetic image information into the second generator to obtain second synthetic image information corresponding to the second video image information; Determine a model loss value according to the first video image information, the second video image information, the first sketch image information to be trained, the second sketch image information to be trained, the first synthetic image information, and the second synthetic image information; If the model loss value does not meet the preset model convergence conditions, the parameters of the first generator and / or the parameters of the second generator are updated based on the model loss value, and the first generator and the second generator with updated parameters are iteratively trained until the model loss value meets the preset model convergence conditions.

8. An image processing device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps of the image processing method described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

Images