Image processing method, device and device for image processing

The dynamic masking technology integrates the face image, which solves the problem that the face image generation in the prior art is not realistic enough, and realizes flexible control of the changes in facial expressions and textures, improving the authenticity and quality of the image.

CN113706430BActive Publication Date: 2024-11-15BEIJING SOGOU TECHNOLOGY DEVELOPMENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110943965.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-17
Publication Date
2024-11-15
Estimated Expiration
2041-08-17

AI Technical Summary

Technical Problem

The existing face image generation methods cannot flexibly control the expression changes and texture changes of faces, and the three-dimensional face reconstruction method relies on multi-view datasets, resulting in the generated face images not being realistic enough and the image quality is poor.

Method used

By obtaining the texture images of the first face image and the second face image, using a dynamic mask for fusion processing, a fused texture image is generated, and based on this, the second face image is rendered to obtain the target face image. This method includes steps such as face skin detection, key point detection, dynamic mask calculation, texture mapping and rendering, ensuring that the fusion area is within an effective range and improving the authenticity and accuracy of the image.

Benefits of technology

It realizes flexible control of face images, improves the authenticity and image quality of generated face images, avoids the accumulation and amplification of defects in long videos, and ensures color stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113706430B_ABST
    Figure CN113706430B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention provide an image processing method, device and device for image processing. The method comprises: obtaining a first texture image corresponding to a first face image and a second texture image corresponding to a second face image; fusing the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate the area to be fused in the first face image; rendering the second face image based on the fused texture image to obtain a target face image. The embodiments of the present invention can limit the fusion area of ​​the image within the effective range, fit the face boundary of the pre-set area to be fused, and improve the accuracy of face fusion; and the embodiments of the present invention can ensure the color stability and authenticity of the generated target face image, and improve the image quality of the generated target face image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image processing method, device and a device for image processing. Background Art

[0002] With the development of computer technology, face generation technology has also been greatly developed. For example, through face generation technology, the face in the template image can be replaced with the face in the target image, or the facial expression and other features in the template image can be changed to obtain a series of dynamically transformed face images.

[0003] There are two main methods for generating facial images: facial image generation methods based on deep learning and facial image generation methods based on 3D facial reconstruction. However, in facial image generation methods based on deep learning, the facial generation process is often driven by feature points and cannot flexibly control facial expression and texture changes; while facial image generation methods based on 3D facial reconstruction are heavily dependent on datasets with multi-view images of the same person, and the current frame will be affected by the previous frames, and defects will be accumulated and magnified when generating longer videos. In short, the facial images generated by the existing facial generation methods are not realistic enough and have poor image quality. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method, an apparatus and an apparatus for image processing, which can flexibly control the texture change of a face image and improve the authenticity of the generated face image.

[0005] In order to solve the above problem, an embodiment of the present invention discloses an image processing method, which includes:

[0006] Acquire a first texture image corresponding to the first face image and a second texture image corresponding to the second face image;

[0007] Performing a fusion process on the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate a to-be-fused area in the first facial image;

[0008] The second facial image is rendered based on the fused texture image to obtain a target facial image.

[0009] Optionally, the fusing the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes:

[0010] Mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space according to the first dynamic mask to obtain a texture mask;

[0011] The first texture image is fused into the second texture image according to the texture mask to obtain a fused texture image.

[0012] Optionally, before fusing the first texture image and the second texture image based on the first dynamic mask to obtain the fused texture image, the method further includes:

[0013] Performing facial skin detection on the first facial image to obtain a skin detection range of the first facial image;

[0014] Performing key point detection on the first facial image to obtain a key point detection range of the first facial image;

[0015] A first dynamic mask is calculated based on the skin detection range and the key point detection range.

[0016] Optionally, the acquiring a first texture image corresponding to the first facial image and a second texture image corresponding to the second facial image includes:

[0017] Inputting the first face image and the second face image into a pre-trained 3D face reconstruction network to perform 3D face reconstruction processing, thereby obtaining 3D model parameters of the first face image and 3D model parameters of the second face image;

[0018] constructing a three-dimensional face model corresponding to the first face image and a three-dimensional face model corresponding to the second face image according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image;

[0019] A first texture image corresponding to the first facial image is drawn based on the three-dimensional facial model corresponding to the first facial image, and a second texture image corresponding to the second facial image is drawn based on the three-dimensional facial model corresponding to the second facial image.

[0020] Optionally, the three-dimensional model parameters include shape parameters, expression parameters and posture parameters, and constructing the three-dimensional face model corresponding to the first face image and the three-dimensional face model corresponding to the second face image according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image includes:

[0021] constructing a three-dimensional face model corresponding to the first face image according to the shape parameters, expression parameters and posture parameters of the first face image;

[0022] A three-dimensional face model corresponding to the second face image is constructed according to the shape parameters of the second face image and the expression parameters and posture parameters of the first face image.

[0023] Optionally, the expression parameters include at least two basis vectors, and constructing a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image, the expression parameters and the posture parameters of the first face image includes:

[0024] Adjusting the coefficients of each basis vector in the expression parameters of the first facial image to obtain adjusted expression parameters;

[0025] A three-dimensional face model corresponding to the second face image is constructed according to the shape parameters of the second face image, the adjusted expression parameters and the posture parameters of the first face image.

[0026] Optionally, rendering the second facial image based on the fused texture image to obtain a target facial image includes:

[0027] Mapping the texture value of each pixel in the fused texture image to the three-dimensional face model corresponding to the second image to obtain a target face model;

[0028] Drawing a third face image based on the target face model, wherein the third face image includes the area to be fused in the first face image;

[0029] The third facial image is rendered into the second facial image according to a second dynamic mask to obtain a target facial image, wherein the second dynamic mask is used to indicate an area to be covered in the second facial image.

[0030] Optionally, the drawing of a first texture image corresponding to the first facial image based on the three-dimensional facial model corresponding to the first facial image, and the drawing of a second texture image corresponding to the second facial image based on the three-dimensional facial model corresponding to the second facial image include:

[0031] Mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to the texture space to obtain a first texture image corresponding to the first face image;

[0032] The texture value of each coordinate point in the three-dimensional face model corresponding to the second face image is mapped to the texture space to obtain a second texture image corresponding to the second face image.

[0033] Optionally, the method further comprises:

[0034] Acquire a training sample set, the training sample set including a face image and two-dimensional key point coordinates corresponding to the face image;

[0035] Based on the training sample set, weakly supervised training is performed on the pre-constructed three-dimensional face reconstruction network, and a loss value of the three-dimensional face reconstruction network is calculated until the loss value meets a convergence condition to obtain a trained three-dimensional face reconstruction network.

[0036] Optionally, calculating the loss value of the 3D face reconstruction network includes:

[0037] In each round of training, determining loss parameters of the three-dimensional face network, wherein the loss parameters include a pixel loss value, a key point distance loss value, an identity loss value, and a regularization loss value;

[0038] The loss parameters are weighted and summed according to preset weights to obtain the loss value of the three-dimensional face reconstruction network.

[0039] Optionally, the number of the first face images is greater than 1, and the fusing process of the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes:

[0040] Determine a first dynamic mask corresponding to each first face image respectively;

[0041] According to the first dynamic mask corresponding to each first facial image, the first texture images corresponding to each first facial image are fused to obtain a third texture image;

[0042] Determining a third dynamic mask corresponding to the third texture image;

[0043] The third texture image and the second texture image are fused based on the third dynamic mask to obtain a fused texture image.

[0044] On the other hand, an embodiment of the present invention discloses an image processing device, the device comprising:

[0045] A texture image acquisition module, used to acquire a first texture image corresponding to the first face image and a second texture image corresponding to the second face image;

[0046] a texture image fusion module, configured to fuse the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate a to-be-fused area in the first facial image;

[0047] The target image generation module is used to render the second face image based on the fused texture image to obtain a target face image.

[0048] Optionally, the texture image fusion module includes:

[0049] a texture mask acquisition submodule, configured to map the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space according to the first dynamic mask, so as to obtain a texture mask;

[0050] The texture image fusion submodule is used to fuse the first texture image into the second texture image according to the texture mask to obtain a fused texture image.

[0051] Optionally, the device further comprises:

[0052] A skin detection module, configured to perform facial skin detection on the first facial image to obtain a skin detection range of the first facial image;

[0053] A key point detection module, used to perform key point detection on the first face image to obtain a key point detection range of the first face image;

[0054] A dynamic mask calculation module is used to calculate a first dynamic mask based on the skin detection range and the key point detection range.

[0055] Optionally, the texture image acquisition module includes:

[0056] A 3D face reconstruction submodule, used to input the first face image and the second face image into a pre-trained 3D face reconstruction network to perform 3D face reconstruction processing, and obtain 3D model parameters of the first face image and 3D model parameters of the second face image;

[0057] a 3D face model construction submodule, configured to construct a 3D face model corresponding to the first face image and a 3D face model corresponding to the second face image according to the 3D model parameters of the first face image and the 3D model parameters of the second face image;

[0058] The texture image drawing submodule is used to draw a first texture image corresponding to the first face image based on the three-dimensional face model corresponding to the first face image, and to draw a second texture image corresponding to the second face image based on the three-dimensional face model corresponding to the second face image.

[0059] Optionally, the three-dimensional model parameters include shape parameters, expression parameters and posture parameters, and the three-dimensional face model construction submodule includes:

[0060] A first face model construction unit, configured to construct a three-dimensional face model corresponding to the first face image according to shape parameters, expression parameters and posture parameters of the first face image;

[0061] The second face model construction unit is used to construct a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image and the expression parameters and posture parameters of the first face image.

[0062] Optionally, the expression parameter includes at least two basis vectors, and the second face model building unit includes:

[0063] An expression parameter adjustment subunit, used to adjust the coefficients of each basis vector in the expression parameter of the first face image to obtain adjusted expression parameters;

[0064] The face model construction subunit is used to construct a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image, the adjusted expression parameters and the posture parameters of the first face image.

[0065] Optionally, the target image generation module includes:

[0066] A texture mapping submodule, used to map the texture value of each pixel in the fused texture image to the three-dimensional face model corresponding to the second image to obtain a target face model;

[0067] A face image drawing submodule, used for drawing a third face image based on the target face model, wherein the third face image includes the area to be fused in the first face image;

[0068] The image rendering submodule is used to render the third face image into the second face image according to a second dynamic mask to obtain a target face image, wherein the second dynamic mask is used to indicate an area to be covered in the second face image.

[0069] Optionally, the texture image drawing submodule includes:

[0070] A first texture image drawing unit, used for mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space, to obtain a first texture image corresponding to the first face image;

[0071] The second texture image drawing unit is used to map the texture value of each coordinate point in the three-dimensional face model corresponding to the second face image to the texture space to obtain a second texture image corresponding to the second face image.

[0072] Optionally, the texture image acquisition module further includes:

[0073] A sample set acquisition submodule, used to acquire a training sample set, wherein the training sample set includes a face image and two-dimensional key point coordinates corresponding to the face image;

[0074] The 3D face reconstruction network training submodule is used to perform weakly supervised training on the pre-constructed 3D face reconstruction network based on the training sample set, and calculate the loss value of the 3D face reconstruction network until the loss value meets the convergence condition to obtain a trained 3D face reconstruction network.

[0075] Optionally, the 3D face reconstruction network training submodule includes:

[0076] A loss parameter determination unit, used to determine the loss parameters of the three-dimensional face network in each round of training, wherein the loss parameters include pixel loss value, key point distance loss value, identity loss value and regularization loss value;

[0077] The loss value calculation unit is used to perform weighted summation on each loss parameter according to preset weights to obtain the loss value of the three-dimensional face reconstruction network.

[0078] Optionally, the number of the first face images is greater than 1, and the texture image fusion module includes:

[0079] A first dynamic mask determination submodule, used to respectively determine a first dynamic mask corresponding to each first face image;

[0080] A first texture image fusion submodule is used to fuse the first texture images corresponding to each first face image according to the first dynamic mask corresponding to each first face image to obtain a third texture image;

[0081] A second dynamic mask determination submodule, used to determine a third dynamic mask corresponding to the third texture image;

[0082] The second texture image fusion submodule is used to fuse the third texture image and the second texture image based on the third dynamic mask to obtain a fused texture image.

[0083] On the other hand, an embodiment of the present invention discloses a device for image processing, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for executing one or more of the image processing methods described above.

[0084] On the other hand, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables a device to perform one or more of the image processing methods described above.

[0085] The embodiments of the present invention include the following advantages:

[0086] When generating a target facial image, the embodiment of the present invention first obtains a first texture image corresponding to the first facial image and a second texture image corresponding to the second facial image, and then fuses the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate the area to be fused in the first facial image; finally, the second facial image is rendered based on the fused texture image to obtain the target facial image. The embodiment of the present invention fuses the first texture image corresponding to the first facial image and the second texture image of the second facial image based on a dynamic mask, which can limit the fusion area of ​​the image within a valid range, fit the facial boundary of the pre-set area to be fused, and improve the accuracy of facial fusion; and the embodiment of the present invention renders the second facial image based on the fused texture image to generate a target facial image, which can ensure the color stability and authenticity of the generated target facial image and improve the image quality of the generated target facial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0088] Figure 1 is a flowchart of steps of an image processing method embodiment of the present invention;

[0089] Figure 2 It is a schematic diagram of skin detection and key point detection of the present invention;

[0090] Figure 3 is a structural block diagram of an embodiment of an image processing device of the present invention;

[0091] Figure 4 is a block diagram of an apparatus 800 for image processing according to the present invention;

[0092] Figure 5 It is a schematic diagram of the structure of the server in some embodiments of the present invention. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0094] Method Embodiment

[0095] Reference Figure 1 , shows a flowchart of an image processing method embodiment of the present invention, the method may specifically include the following steps:

[0096] Step 101: Obtain a first texture image corresponding to a first face image and a second texture image corresponding to a second face image.

[0097] Step 102: The first texture image and the second texture image are fused based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate a to-be-fused area in the first facial image.

[0098] Step 103: Render the second facial image based on the fused texture image to obtain a target facial image.

[0099] The image processing method provided by the embodiment of the present invention can be applied to face generation scenarios. Through the embodiment of the present invention, the area to be fused in the first face image can be fused into the second face image to obtain the target face image. The image processing method provided by the embodiment of the present invention can be applied to electronic devices, including but not limited to: servers, smart collections, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc.

[0100] In an embodiment of the present invention, the first facial image and / or the second facial image may be a partial face image (i.e., facial information is incomplete, such as a side face or a face partially obscured by hair, accessories, etc.), or a complete face image (i.e., facial information is complete, such as a proven unobstructed face). The first facial image and / or the second facial image may be a color image or a grayscale image. In addition, the embodiment of the present invention does not impose any restrictions on the image format. For example, the image format of the first facial image and / or the second facial image may be any format that can be recognized by an electronic device, such as jpg (Joint Photographic experts Group, a picture format), BMP (Bitmap, image file format) or RAW (RAW ImageFormat, a lossless compression format).

[0101] Of course, in the embodiment of the present invention, the faces in the first face image and / or the second face image are not limited to real faces, but may also be faces of cartoon characters, comic characters, etc. It can be understood that the embodiment of the present invention does not limit the category, age, and expression of the faces in the first face image and the second face image.

[0102] It should be noted that the first facial image and / or the second facial image can be a facial image of a target object acquired by the electronic device through an image acquisition device, or a facial image pre-stored in the electronic device, or a facial image downloaded by the electronic device from a network. The embodiment of the present invention does not limit the first facial image and the second facial image. Among them, the image acquisition device can be installed in the electronic device, or it can establish a connection with the electronic device. The connection can be a wired connection or a wireless connection. The embodiment of the present invention does not limit this. Exemplarily, the electronic device establishes a wifi connection with the camera, and acquires the first facial image and / or the second facial image through the camera.

[0103] The first facial image is a source facial image including an area to be fused, and the second facial image is a base image including an area to be covered. Through the image processing method provided in an embodiment of the present invention, it is possible to fuse the area to be fused in the first facial image to the area to be covered in the second facial image to obtain a target facial image.

[0104] Among them, the first face image and the second face image can be face images collected for the same target object, or face images collected for different target objects. For example, face images are collected for the same target object to obtain the first face image and the second face image, and the first face image can be used to compensate the details of the second face image to obtain a clearer target face image with more complete details; or the first face image and the second face image can be fused to generate a target face image that can reflect the change information such as the expression change, facial features change, and posture change of the target object. Face images are collected for different target objects respectively to obtain the first face image and the second face image. In one example, the first face image can be a user photo, and the second face image can be an image of an anime character. By fusing the first face image into the second face image, a target face image that can present the anime style of the second face image and have the user features in the first face image is generated; or the first face image can be a user photo, and the second face image can be a celebrity photo. By fusing the first face image into the second face image, a target face image that contains both user features and celebrity features can be generated, and so on.

[0105] The embodiment of the present invention generates a target facial image by fusing a first facial image and a second facial image. First, a first texture image corresponding to the first facial image and a second texture image corresponding to the second facial image are obtained. Specifically, a 3D (3 Dimensions, three-dimensional) scanning device can be used to obtain the first texture image corresponding to the first facial image and the second texture image corresponding to the second facial image. For example, a 3D facial model corresponding to the first facial image and a 3D facial model corresponding to the second facial image are obtained through a 3D scanning device, and a first mapping relationship and a second mapping relationship are established respectively, wherein the first mapping relationship refers to the mapping relationship between the points on the 3D facial model and the pixels in the texture image, and the second mapping relationship refers to the mapping relationship between the points on the 3D facial model and the pixels in the facial image. The electronic device can obtain the first texture image corresponding to the first facial image and the second texture image corresponding to the second facial image through the first mapping relationship and the second mapping relationship.

[0106] Then, the first texture image is fused into the second texture image based on the first dynamic mask. It should be noted that the mask is a binary image used to indicate the range of images to be fused in image fusion. In an embodiment of the present invention, the first dynamic mask is a binary image corresponding to the range to be fused in the first face image, which is used to indicate the area to be fused in the first face image. The first dynamic mask can be determined based on the face boundary of the area to be fused in the first face image. For example, if you want to fuse the mouth in the first face image into the second face image, that is, replace the mouth in the second face image with the mouth in the first face image, then the area indicated by the first dynamic mask is the area where the mouth is located in the first face image. Through the first mask, the fusion area of ​​the first texture image and the second texture image can be controlled within the effective range to avoid the fusion of textures in other areas affecting the accuracy of the target face image finally generated.

[0107] Finally, the second face image is rendered based on the fused texture image to obtain the target face image. Specifically, in the fused texture image, the texture values ​​of the pixels corresponding to the area to be fused in the first face image are the fused texture values ​​of the corresponding pixels in the first texture image and the second texture image, and the texture values ​​of other pixels are the texture values ​​of the corresponding pixels in the second texture image. The process of rendering the second face image based on the fused texture image is the process of adjusting the texture values ​​of each pixel in the area to be covered in the second face image to the texture values ​​of the corresponding pixels in the fused texture image. Among them, the area to be covered in the second face image can be consistent with the area to be fused in the first face image, or it can include a larger range than the area to be fused in the first face image, so as to ensure that when the expressions or postures of the second face image and the first face image are inconsistent, the area to be fused in the first face image can be completely included, so as to ensure the integrity and accuracy of the generated target face image.

[0108] It should be noted that, when rendering the second facial image, the embodiment of the present invention renders each frame of the image independently, and each frame of the target facial image generated is not affected by other image frames. Compared with the facial image generation method in the online technology that relies on a data set of multi-view images of the same person, the embodiment of the present invention can avoid the accumulation and amplification of defects when generating a longer video, thereby improving the stability of the generated target facial image.

[0109] In an optional embodiment of the present invention, the step 102 of fusing the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes:

[0110] Step S11, mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space according to the first dynamic mask to obtain a texture mask;

[0111] Step S12: Fusing the first texture image into the second texture image according to the texture mask to obtain a fused texture image.

[0112] When the first texture image and the second texture image are fused, the texture values ​​of each coordinate point in the three-dimensional face model corresponding to the first face image can be first mapped to the texture space according to the first dynamic mask to obtain the texture mask. Specifically, the mapping relationship between each coordinate point in the three-dimensional face model corresponding to the first face image and the coordinate value in the texture space can be pre-set, and then the texture values ​​of each coordinate point in the three-dimensional face model corresponding to the first face image can be mapped to the texture space based on the mapping relationship and the first dynamic mask to obtain the texture mask.

[0113] Then, the first texture image is fused into the second texture image according to the texture mask to obtain a fused texture image. Specifically, the first texture image and the second texture image can be fused by methods such as Poisson fusion and bilinear interpolation sampling, and the texture values ​​corresponding to the pixels in the second texture image that are within the range of the texture mask are adjusted to the fused texture of the corresponding pixels in the first texture image and the second texture image, and the current texture values ​​of the pixels in the second texture image that are outside the range of the texture mask are retained.

[0114] By using texture masking, the fusion area of ​​the first texture image and the second texture image can be controlled within a valid range, thereby avoiding fusion of textures in other areas to affect the accuracy of the target face image that is finally generated.

[0115] In an optional embodiment of the present invention, before the first texture image and the second texture image are fused based on the first dynamic mask to obtain the fused texture image in step 102, the method further includes:

[0116] Step S21, performing facial skin detection on the first facial image to obtain a skin detection range of the first facial image;

[0117] Step S22, performing key point detection on the first face image to obtain a key point detection range of the first face image;

[0118] Step S23: Calculate a first dynamic mask based on the skin detection range and the key point detection range.

[0119] In an embodiment of the present invention, the first dynamic mask can be calculated by performing skin detection and key point detection on the first face image. The purpose of skin detection is to automatically detect the uncovered face skin range in the first face image; the purpose of key point detection is to determine the key points of the target face to be fused in the first face image. The key points can depict the face contour and key parts of the face, such as eyebrows, eyes, nose, mouth, etc.

[0120] Optionally, the step S21 of performing facial skin detection on the first facial image to obtain a skin detection range of the first facial image includes:

[0121] Sub-step S211: inputting the first face image into a pre-trained skin detection model to perform skin detection processing, and obtaining a determination value of each pixel in the first face image, wherein the determination value is used to identify whether the pixel is a face pixel;

[0122] Sub-step S212: determining a skin detection range of the first facial image based on the determination value of each pixel point in the first facial image.

[0123] The judgment value of the pixel point is used to identify whether the pixel point is a pixel point corresponding to the unobstructed skin in the first face image. The essence of the skin detection model is to classify each pixel point in the input first face image. Therefore, in the embodiment of the present invention, any neural network model that is good at classification and pixel annotation can be selected as the skin detection model, and the skin detection model is trained through the face image to obtain a trained skin detection model.

[0124] Optionally, the step S22 of performing key point detection on the first facial image to obtain a key point detection range of the first facial image includes:

[0125] Sub-step S221, inputting the first face image into a pre-trained key point detection model to perform key point detection processing to obtain key point coordinates of the first face image;

[0126] Sub-step S222: determining a key point detection range of the first facial image according to the key point coordinates.

[0127] Among them, the key point detection model can use the ASM (Active Shape Model) detection algorithm, or any other neural network model that is good at key point detection, and the embodiment of the present invention does not make specific limitations on this.

[0128] After the key point coordinates of the first face image are determined, the key points can be further connected according to the key point coordinates to obtain key point lines, and the key point detection range in the first face image can be determined according to the key point lines.

[0129] Reference Figure 2 , showing a schematic diagram of skin detection and key point detection provided by an embodiment of the present invention. Figure 2 The first sub-image on the left is the first face image, the second sub-image in the middle is a schematic diagram of the skin detection range corresponding to the first face image, and the third sub-image on the right is a schematic diagram of the key point detection range corresponding to the first face image. Figure 2 As shown, the color of the face pixel points in the first face image is set to white, and the color of the non-face pixel points is set to black. The skin detection range corresponding to the first face image can be determined according to the color of each pixel point. Similarly, the key point detection range of the first face image is determined according to the connection lines of each key point corresponding to the first face image, and the color of each pixel point within the key point detection range is set to white, and the color of each pixel point outside the key point detection range is set to black. The key point detection range corresponding to the first face image can be intuitively determined according to the color of each pixel point.

[0130] After determining the skin detection range and key point detection range corresponding to the first face image, the intersection of the skin detection range and the key point detection range is obtained to obtain the first dynamic mask. Figure 2 For example, we can directly find the intersection of the second subgraph and the third subgraph. Figure 2 The three sub-images have the same size, and each pixel on each sub-image can find the pixel at the same position in the other two sub-images. Therefore, the coordinate set composed of the coordinate points where the pixel colors of the second and third sub-images are white is the area to be merged in the first sub-image, that is, the first dynamic mask.

[0131] In an optional embodiment of the present invention, the step 101 of obtaining a first texture image corresponding to the first face image and a second texture image corresponding to the second face image includes:

[0132] Step S31: input the first face image and the second face image into a pre-trained 3D face reconstruction network to perform 3D face reconstruction processing, thereby obtaining 3D model parameters of the first face image and 3D model parameters of the second face image;

[0133] Step S32: constructing a three-dimensional face model corresponding to the first face image and a three-dimensional face model corresponding to the second face image according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image;

[0134] Step S33: draw a first texture image corresponding to the first facial image based on the three-dimensional facial model corresponding to the first facial image, and draw a second texture image corresponding to the second facial image based on the three-dimensional facial model corresponding to the second facial image.

[0135] The 3D face reconstruction network can be any neural network model that is good at 3D face reconstruction in the art. The 3D face reconstruction network can be trained using a weakly supervised training method, and then the trained 3D face reconstruction network is used to perform 3D face reconstruction processing on the first face image and the second face image, respectively, to obtain the 3D model parameters of the first face image and the 3D model parameters of the second face image.

[0136] It should be noted that the 3D model parameters in the embodiment of the present invention include but are not limited to shape parameters, expression parameters and posture parameters in the face image. According to the 3D model parameters, a 3D face model corresponding to the first face image and a 3D face model corresponding to the second face image can be constructed.

[0137] Each pixel point in the first face image has a corresponding coordinate point in the three-dimensional face model corresponding to the first face image. Similarly, each pixel point in the second face image also has a corresponding coordinate point in the three-dimensional face model corresponding to the second face image. According to the pre-set mapping relationship between each coordinate point in the three-dimensional face model and each coordinate point in the texture space, the first texture image corresponding to the first face image and the second texture image corresponding to the second face image can be determined.

[0138] In an optional embodiment of the present invention, the three-dimensional model parameters include shape parameters, expression parameters and posture parameters, and step S32 constructs a three-dimensional face model corresponding to the first face image and a three-dimensional face model corresponding to the second face image according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image, including:

[0139] Sub-step S321, constructing a three-dimensional face model corresponding to the first face image according to the shape parameters, expression parameters and posture parameters of the first face image;

[0140] Sub-step S322: constructing a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image and the expression parameters and posture parameters of the first face image.

[0141] In an embodiment of the present invention, in order to further improve the fusion effect of the first facial image and the second facial image and improve the realism of the finally generated target facial image, when constructing a three-dimensional facial model corresponding to the second facial image, the expression parameters and posture parameters of the second facial image can be directly replaced with the expression parameters and posture parameters of the first facial image, and the three-dimensional facial model corresponding to the second facial image is constructed according to the shape parameters of the second facial image and the expression parameters and posture parameters of the first facial image, so that the expression changes in the generated target facial image are more natural.

[0142] In an optional embodiment of the present invention, the expression parameters in sub-step S322 include at least two basis vectors, and constructing a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image, the expression parameters and posture parameters of the first face image, includes:

[0143] P11, adjusting the coefficients of each basis vector in the expression parameters of the first facial image to obtain adjusted expression parameters;

[0144] P12. Construct a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image, the adjusted expression parameters and the posture parameters of the first face image.

[0145] In an embodiment of the present invention, when constructing a three-dimensional facial model corresponding to a second facial image, in addition to directly replacing the expression parameters and posture parameters of the second facial image with the expression parameters and posture parameters of the first facial image, the expression parameters of the first facial image can be further adjusted, and a three-dimensional facial model corresponding to the second facial image can be constructed based on the adjusted expression parameters, thereby improving the flexibility of expression changes, so that a target facial image with any expression can be generated according to actual needs.

[0146] In an optional embodiment of the present invention, the step S33 of drawing a first texture image corresponding to the first facial image based on the three-dimensional facial model corresponding to the first facial image, and drawing a second texture image corresponding to the second facial image based on the three-dimensional facial model corresponding to the second facial image, includes:

[0147] Sub-step S331, mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to the texture space, to obtain a first texture image corresponding to the first face image;

[0148] Sub-step S332: Mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the second face image to the texture space to obtain a second texture image corresponding to the second face image.

[0149] There is a correspondence between the coordinate points in the three-dimensional face model corresponding to the first face image and / or the second face image and the coordinates in the texture space, and the correspondence can be preset. According to the correspondence between the coordinate points in the three-dimensional face model and the coordinates in the texture space, the texture values ​​of each coordinate point in the three-dimensional face model corresponding to the first face image are mapped to the texture space, and the first texture image corresponding to the first face image can be obtained. Similarly, the texture values ​​of each coordinate point in the three-dimensional face model corresponding to the second face image are mapped to the texture space, and the second texture image can be obtained.

[0150] In an optional embodiment of the present invention, the step 103 of rendering the second face image based on the fused texture image to obtain the target face image includes:

[0151] Step S41, mapping the texture value of each pixel in the fused texture image to the three-dimensional face model corresponding to the second image to obtain a target face model;

[0152] Step S42: drawing a third face image based on the target face model, wherein the third face image includes the area to be fused in the first face image;

[0153] Step S43: Render the third facial image into the second facial image according to a second dynamic mask to obtain a target facial image, wherein the second dynamic mask is used to indicate an area to be covered in the second facial image.

[0154] Among them, the second dynamic mask is a binary image corresponding to the area to be fused in the second facial image, which is used to indicate the area to be fused in the second facial image, that is, the area to be covered in the second facial image. The calculation method of the second dynamic mask is the same as the calculation method of the first dynamic mask. Skin detection and key point detection are performed on the second facial image to obtain the skin detection range and key point detection range of the second facial image respectively, and then the second dynamic mask is obtained by finding the intersection of the skin detection range and the key point detection range. The coverage range of the second dynamic mask can be consistent with the coverage range of the first dynamic mask, or it can be larger than the coverage range of the first dynamic mask, so as to ensure that when the expressions or postures of the second facial image and the first facial image are inconsistent, the area to be fused in the first facial image can be completely included to ensure the integrity and accuracy of the generated target facial image.

[0155] It should be noted that, in the embodiment of the present invention, the sizes of the first face image, the second face image, the first texture image, the second texture image, the fused texture image, the third face image, and the target face image are all the same. Each pixel in the fused texture image has a corresponding coordinate point in the three-dimensional face model corresponding to the first face image and / or the second face image. Therefore, based on the mapping relationship between the pixel in the fused texture image and the coordinate point in the three-dimensional face model, the texture value of each pixel in the fused texture image can be mapped to the three-dimensional face model corresponding to the second face image, that is, the texture value of each coordinate point in the three-dimensional face model corresponding to the second face image is replaced with the texture value of the corresponding pixel in the fused texture image to obtain the target face model.

[0156] Then, a third face image is drawn based on the target face model, the third face image includes the area to be fused in the first face image, and the texture value of the area to be fused in the third face image is the fused texture of the first face image and the second face image.

[0157] Finally, the target face image can be obtained by rendering the third face image into the second face image according to the second dynamic mask. The depth value of the pixel points outside the second dynamic mask range in the third face image can be set above the preset buffer threshold, so that the pixel points outside the second dynamic mask range are ignored during the rendering process, and only the pixel points within the second dynamic mask range are rendered, thereby improving the accuracy of the rendering range.

[0158] It should be noted that, in the embodiment of the present invention, the second facial image may include a single face or multiple faces. When fusing the first facial image, multiple first facial images may be fused with the specified face in the second facial image to obtain multiple fused target facial images; or multiple first facial images may be fused with the specified face in the second facial image at the same time to obtain a fused target facial image. For example, there are multiple first facial images, corresponding to the faces of user A1, user B1, and user C1, respectively, and the second facial image includes the faces of user D2, user E2, and user F2. In an embodiment of the present invention, the first facial image of user A1 can be fused with the face of user D2 in the second facial image, the first facial image of user B1 can be fused with the face of user E2 in the second facial image, and the first facial image of user C1 can be fused with the face of user F2 in the second facial image to obtain fused target facial images A1D2, B1E2, and C1F2 respectively; the faces of user A1, user B1, and user C1 can also be fused with the face of user D2 at the same time to obtain the fused target facial image A1B1C1D2.

[0159] If multiple first face images are to be fused with the specified face in the second face image, it is only necessary to determine the first dynamic mask according to the area to be fused in the first face image, and fuse the first texture map of the first face image with the second texture image corresponding to the second face image based on the first dynamic mask to obtain a fused texture image; then determine the corresponding second dynamic mask according to the area to be fused in the second face image, and render the second face image based on the second dynamic mask and the fused texture image to obtain the target face image. For example, if the face of user A1 is to be fused with the face of user D2, it is only necessary to determine the first dynamic mask corresponding to the face of user A1 based on the area to which the face of user A1 belongs in the first face image, and determine the second dynamic mask corresponding to the face of user D2 based on the area to which the face of user D2 belongs in the second face image, and then perform the above image processing steps according to the determined first dynamic mask and second dynamic mask. Similarly, if the face of user B1 is to be merged with the face of user E2, it is only necessary to replace the first dynamic mask with the first dynamic mask corresponding to the face of user B1, and replace the second dynamic mask with the second dynamic mask corresponding to the face of user E2.

[0160] In an optional embodiment of the present invention, the number of the first face images is greater than 1, and the step 102 of fusing the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes:

[0161] Step S51, respectively determining a first dynamic mask corresponding to each first face image;

[0162] Step S52, performing fusion processing on the first texture images corresponding to each first face image according to the first dynamic mask corresponding to each first face image, to obtain a third texture image;

[0163] Step S53, determining a third dynamic mask corresponding to the third texture image;

[0164] Step S54: fusing the third texture image and the second texture image based on the third dynamic mask to obtain a fused texture image.

[0165] If multiple first facial images are simultaneously fused with a designated face in a second facial image to obtain a fused target facial image, the processing steps from step 101 to step 103 can be followed to iteratively fuse each first facial image in turn, and the target facial image after each round of fusion can be used as the second facial image. Taking the example of fusing the faces of user A1, user B1, and user C1 with the face of user D2 to obtain the fused target face image A1B1C1D2, the first face image corresponding to user A1 and the first face image corresponding to user B1 can be fused according to steps 101 to 103 to obtain the fused face image A1B1; then, the face image A1B1 is fused with the first face image corresponding to user C1 according to steps 101 to 103 to obtain the fused face image A1B1C1; finally, the face image A1B1C1 is fused with the second face image corresponding to user D2 according to steps 101 to 103 to obtain the fused target face image A1B1C1D2. It should be noted that in the iterative fusion process, the dynamic mask used is the dynamic mask corresponding to the face image to be fused in each round of fusion processing.

[0166] In addition to iterative fusion according to steps 101 to 103, multiple first face images can also be fused into one second face image at the same time according to steps S51 to S54. Let's continue to take the example of fusing the faces of user A1, user B1, and user C1 with the face of user D2 at the same time to obtain the fused target face image A1B1C1D2. First, the first dynamic masks corresponding to user A1, user B1, and user C1 are determined respectively, and then, based on the first dynamic masks corresponding to each user's face, the first texture images of the first face images corresponding to each user's face are fused to obtain a third texture image. Specifically, the first texture image corresponding to user A1 and the first texture image corresponding to user B1 can be fused based on the first dynamic mask corresponding to user A1 to obtain texture image a1b1; then, based on the first dynamic mask corresponding to user C1, the first texture image corresponding to user C1 and the texture image a1b1 are fused to obtain a third texture image a1b1c1. The first facial images corresponding to user A1, user B1, and user C1 may exist in one image at the same time, or may exist in different images.

[0167] In order to improve the accuracy of image fusion in the third texture image, it is necessary to further determine the third dynamic mask corresponding to the third texture image. The third dynamic mask is used to indicate the areas to be fused of each first facial image in the third texture image. Then, based on the third dynamic mask, the third texture image is fused with the second texture image of the second facial image to obtain a fused texture image. The specific fusion process can refer to step 102, and the embodiment of the present invention will not be further described here. After obtaining the fused texture image, the second facial image can be rendered based on the processing process according to step 103 to obtain the target facial image.

[0168] It should be noted that, in the embodiment of the present invention, the first facial image may contain a single face or multiple faces at the same time. That is to say, the faces of user A1, user B1, and user C1 may also exist in one first facial image at the same time. Whether it is to fuse multiple first facial images containing a single face with a specified face in the second facial image, or to fuse multiple faces in a first facial image containing multiple faces with a specified face in the second facial image, it is only necessary to determine the corresponding dynamic mask according to the face to be fused, and perform image fusion based on the dynamic mask.

[0169] In an optional embodiment of the present invention, the method further includes:

[0170] Step S51, obtaining a training sample set, wherein the training sample set includes a face image and two-dimensional key point coordinates corresponding to the face image;

[0171] Step S52: weakly supervised training is performed on the pre-constructed three-dimensional face reconstruction network based on the training sample set, and a loss value of the three-dimensional face reconstruction network is calculated until the loss value meets a convergence condition to obtain a trained three-dimensional face reconstruction network.

[0172] When training a 3D face reconstruction network, a weakly supervised training method can be used. Specifically, the training sample set does not rely on the 3D key point coordinates of the face image, but trains the 3D face reconstruction network based on the face image and its 2D key point coordinates, thereby eliminating the need to obtain the 3D key point coordinates of a large number of face images, reducing the difficulty of obtaining training samples and improving the training efficiency of the 3D face reconstruction network.

[0173] In an optional embodiment of the present invention, the step S52 of calculating the loss value of the 3D face reconstruction network includes:

[0174] Sub-step S521: in each round of training, determining the loss parameters of the three-dimensional face network, wherein the loss parameters include a pixel loss value, a key point distance loss value, an identity loss value, and a regularization loss value;

[0175] Sub-step S522: performing weighted summation on each loss parameter according to preset weights to obtain a loss value of the three-dimensional face reconstruction network.

[0176] Among them, the pixel loss value is used to identify the loss of effective pixels in the three-dimensional face model of the face image output by the three-dimensional face reconstruction network; the key point distance loss value is used to identify the error between each key point and the actual key point in the three-dimensional face model output by the three-dimensional face reconstruction network; the identity loss value is used to identify the loss of character feature information in the three-dimensional face model determined by the three-dimensional face reconstruction network; the regularization loss is used to identify the accuracy of the output result of the three-dimensional face reconstruction network. It should be noted that the loss parameters of the three-dimensional face reconstruction network can also include other loss parameters, and the above-listed loss parameters do not constitute a limitation on the embodiments of the present invention.

[0177] In order to improve the accuracy and robustness of the three-dimensional face reconstruction network, when training the three-dimensional face reconstruction network, multiple loss parameters can be considered at the same time, and the weighted sum of each loss parameter can be used as the loss value of the three-dimensional face reconstruction network. Based on the loss value, the network parameters of the three-dimensional face reconstruction network are adjusted until a three-dimensional face reconstruction network that meets the convergence conditions is obtained.

[0178] In summary, the embodiment of the present invention fuses the first texture image corresponding to the first facial image and the second texture image of the second facial image based on a dynamic mask, which can limit the fusion area of ​​the images within a valid range, fit the facial boundary of the pre-set area to be fused, and improve the accuracy of facial fusion; and, the embodiment of the present invention renders the second facial image based on the fused texture image to generate a target facial image, which can ensure the color stability and authenticity of the generated target facial image, and improve the image quality of the generated target facial image.

[0179] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0180] Device Embodiment

[0181] Reference Figure 3 , shows a structural block diagram of an image processing device embodiment of the present invention, the device may include:

[0182] A texture image acquisition module 201, used to acquire a first texture image corresponding to a first face image and a second texture image corresponding to a second face image;

[0183] A texture image fusion module 202, configured to fuse the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate a to-be-fused area in the first face image;

[0184] The target image generation module 203 is used to render the second face image based on the fused texture image to obtain a target face image.

[0185] Optionally, the texture image fusion module 202 includes:

[0186] a texture mask acquisition submodule, configured to map the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space according to the first dynamic mask, so as to obtain a texture mask;

[0187] The texture image fusion submodule is used to fuse the first texture image into the second texture image according to the texture mask to obtain a fused texture image.

[0188] Optionally, the device further comprises:

[0189] A skin detection module, configured to perform facial skin detection on the first facial image to obtain a skin detection range of the first facial image;

[0190] A key point detection module, used to perform key point detection on the first face image to obtain a key point detection range of the first face image;

[0191] A dynamic mask calculation module is used to calculate a first dynamic mask based on the skin detection range and the key point detection range.

[0192] Optionally, the texture image acquisition module 201 includes:

[0193] A 3D face reconstruction submodule, used to input the first face image and the second face image into a pre-trained 3D face reconstruction network to perform 3D face reconstruction processing, and obtain 3D model parameters of the first face image and 3D model parameters of the second face image;

[0194] a 3D face model construction submodule, configured to construct a 3D face model corresponding to the first face image and a 3D face model corresponding to the second face image according to the 3D model parameters of the first face image and the 3D model parameters of the second face image;

[0195] The texture image drawing submodule is used to draw a first texture image corresponding to the first face image based on the three-dimensional face model corresponding to the first face image, and to draw a second texture image corresponding to the second face image based on the three-dimensional face model corresponding to the second face image.

[0196] Optionally, the three-dimensional model parameters include shape parameters, expression parameters and posture parameters, and the three-dimensional face model construction submodule includes:

[0197] A first face model construction unit, configured to construct a three-dimensional face model corresponding to the first face image according to shape parameters, expression parameters and posture parameters of the first face image;

[0198] The second face model construction unit is used to construct a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image and the expression parameters and posture parameters of the first face image.

[0199] Optionally, the expression parameter includes at least two basis vectors, and the second face model building unit includes:

[0200] An expression parameter adjustment subunit, used to adjust the coefficients of each basis vector in the expression parameter of the first face image to obtain adjusted expression parameters;

[0201] The face model construction subunit is used to construct a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image, the adjusted expression parameters and the posture parameters of the first face image.

[0202] Optionally, the target image generation module includes:

[0203] A texture mapping submodule, used to map the texture value of each pixel in the fused texture image to the three-dimensional face model corresponding to the second image to obtain a target face model;

[0204] A face image drawing submodule, used for drawing a third face image based on the target face model, wherein the third face image includes the area to be fused in the first face image;

[0205] The image rendering submodule is used to render the third face image into the second face image according to a second dynamic mask to obtain a target face image, wherein the second dynamic mask is used to indicate an area to be covered in the second face image.

[0206] Optionally, the texture image drawing submodule includes:

[0207] A first texture image drawing unit, used for mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space, to obtain a first texture image corresponding to the first face image;

[0208] The second texture image drawing unit is used to map the texture value of each coordinate point in the three-dimensional face model corresponding to the second face image to the texture space to obtain a second texture image corresponding to the second face image.

[0209] Optionally, the texture image acquisition module 201 further includes:

[0210] A sample set acquisition submodule, used to acquire a training sample set, wherein the training sample set includes a face image and two-dimensional key point coordinates corresponding to the face image;

[0211] The 3D face reconstruction network training submodule is used to perform weakly supervised training on the pre-constructed 3D face reconstruction network based on the training sample set, and calculate the loss value of the 3D face reconstruction network until the loss value meets the convergence condition to obtain a trained 3D face reconstruction network.

[0212] Optionally, the 3D face reconstruction network training submodule includes:

[0213] A loss parameter determination unit, used to determine the loss parameters of the three-dimensional face network in each round of training, wherein the loss parameters include pixel loss value, key point distance loss value, identity loss value and regularization loss value;

[0214] The loss value calculation unit is used to perform weighted summation on each loss parameter according to preset weights to obtain the loss value of the three-dimensional face reconstruction network.

[0215] Optionally, the number of the first face images is greater than 1, and the texture image fusion module includes:

[0216] A first dynamic mask determination submodule, used to respectively determine a first dynamic mask corresponding to each first face image;

[0217] A first texture image fusion submodule is used to fuse the first texture images corresponding to each first face image according to the first dynamic mask corresponding to each first face image to obtain a third texture image;

[0218] A second dynamic mask determination submodule, used to determine a third dynamic mask corresponding to the third texture image;

[0219] The second texture image fusion submodule is used to fuse the third texture image and the second texture image based on the third dynamic mask to obtain a fused texture image.

[0220] In summary, the embodiment of the present invention fuses the first texture image corresponding to the first facial image and the second texture image of the second facial image based on a dynamic mask, which can limit the fusion area of ​​the images within a valid range, fit the facial boundary of the pre-set area to be fused, and improve the accuracy of facial fusion; and, the embodiment of the present invention renders the second facial image based on the fused texture image to generate a target facial image, which can ensure the color stability and authenticity of the generated target facial image, and improve the image quality of the generated target facial image.

[0221] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0222] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0223] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0224] An embodiment of the present invention provides a device for image processing, the device comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:

[0225] A11. Obtain a first texture image corresponding to the first face image and a second texture image corresponding to the second face image;

[0226] A12. Fusing the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate a to-be-fused area in the first facial image;

[0227] A13. Render the second facial image based on the fused texture image to obtain a target facial image.

[0228] Optionally, the fusing the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes:

[0229] Mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space according to the first dynamic mask to obtain a texture mask;

[0230] The first texture image is fused into the second texture image according to the texture mask to obtain a fused texture image.

[0231] Optionally, before fusing the first texture image and the second texture image based on the first dynamic mask to obtain the fused texture image, the method further includes:

[0232] Performing facial skin detection on the first facial image to obtain a skin detection range of the first facial image;

[0233] Performing key point detection on the first facial image to obtain a key point detection range of the first facial image;

[0234] A first dynamic mask is calculated based on the skin detection range and the key point detection range.

[0235] Optionally, the acquiring a first texture image corresponding to the first facial image and a second texture image corresponding to the second facial image includes:

[0236] Inputting the first face image and the second face image into a pre-trained 3D face reconstruction network to perform 3D face reconstruction processing, thereby obtaining 3D model parameters of the first face image and 3D model parameters of the second face image;

[0237] constructing a three-dimensional face model corresponding to the first face image and a three-dimensional face model corresponding to the second face image according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image;

[0238] A first texture image corresponding to the first facial image is drawn based on the three-dimensional facial model corresponding to the first facial image, and a second texture image corresponding to the second facial image is drawn based on the three-dimensional facial model corresponding to the second facial image.

[0239] Optionally, the three-dimensional model parameters include shape parameters, expression parameters and posture parameters, and constructing the three-dimensional face model corresponding to the first face image and the three-dimensional face model corresponding to the second face image according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image includes:

[0240] constructing a three-dimensional face model corresponding to the first face image according to the shape parameters, expression parameters and posture parameters of the first face image;

[0241] A three-dimensional face model corresponding to the second face image is constructed according to the shape parameters of the second face image and the expression parameters and posture parameters of the first face image.

[0242] Optionally, the expression parameters include at least two basis vectors, and constructing a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image, the expression parameters and the posture parameters of the first face image includes:

[0243] Adjusting the coefficients of each basis vector in the expression parameters of the first facial image to obtain adjusted expression parameters;

[0244] A three-dimensional face model corresponding to the second face image is constructed according to the shape parameters of the second face image, the adjusted expression parameters and the posture parameters of the first face image.

[0245] Optionally, rendering the second facial image based on the fused texture image to obtain a target facial image includes:

[0246] Mapping the texture value of each pixel in the fused texture image to the three-dimensional face model corresponding to the second image to obtain a target face model;

[0247] Drawing a third face image based on the target face model, wherein the third face image includes the area to be fused in the first face image;

[0248] The third facial image is rendered into the second facial image according to a second dynamic mask to obtain a target facial image, wherein the second dynamic mask is used to indicate an area to be covered in the second facial image.

[0249] Optionally, the step of drawing a first texture image corresponding to the first facial image based on the three-dimensional facial model corresponding to the first facial image, and drawing a second texture image corresponding to the second facial image based on the three-dimensional facial model corresponding to the second facial image, includes:

[0250] Mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to the texture space to obtain a first texture image corresponding to the first face image;

[0251] The texture value of each coordinate point in the three-dimensional face model corresponding to the second face image is mapped to the texture space to obtain a second texture image corresponding to the second face image.

[0252] Optionally, the method further comprises:

[0253] Acquire a training sample set, the training sample set including a face image and two-dimensional key point coordinates corresponding to the face image;

[0254] Based on the training sample set, weakly supervised training is performed on the pre-constructed three-dimensional face reconstruction network, and a loss value of the three-dimensional face reconstruction network is calculated until the loss value meets a convergence condition to obtain a trained three-dimensional face reconstruction network.

[0255] Optionally, calculating the loss value of the 3D face reconstruction network includes:

[0256] In each round of training, determining loss parameters of the three-dimensional face network, wherein the loss parameters include a pixel loss value, a key point distance loss value, an identity loss value, and a regularization loss value;

[0257] The loss parameters are weighted and summed according to preset weights to obtain the loss value of the three-dimensional face reconstruction network.

[0258] Optionally, the number of the first face images is greater than 1, and the fusing process of the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes:

[0259] Determine a first dynamic mask corresponding to each first face image respectively;

[0260] According to the first dynamic mask corresponding to each first facial image, the first texture images corresponding to each first facial image are fused to obtain a third texture image;

[0261] Determining a third dynamic mask corresponding to the third texture image;

[0262] The third texture image and the second texture image are fused based on the third dynamic mask to obtain a fused texture image.

[0263] Figure 4 8 is a block diagram of an apparatus 800 for image processing according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0264] Reference Figure 4 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0265] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0266] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0267] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0268] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0269] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0270] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0271] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also image the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0272] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0273] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0274] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0275] Figure 5 1 is a schematic diagram of the structure of the server in some embodiments of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0276] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0277] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to perform Figure 1 The image processing method shown.

[0278] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to perform an image processing method, the method comprising: obtaining a first texture image corresponding to a first facial image and a second texture image corresponding to a second facial image; fusing the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate an area to be fused in the first facial image; and rendering the second facial image based on the fused texture image to obtain a target facial image.

[0279] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0280] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0281] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0282] The above is a detailed introduction to an image processing method, an image processing device and a device for image processing provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Obtain a first texture image corresponding to a first facial image and a second texture image corresponding to a second facial image; the first texture image corresponding to the first facial image is drawn based on a three-dimensional facial model corresponding to the first facial image, and the second texture image corresponding to the second facial image is drawn based on a three-dimensional facial model corresponding to the second facial image; the three-dimensional facial model corresponding to the first facial image is constructed based on shape parameters, expression parameters and posture parameters of the first facial image; the three-dimensional facial model corresponding to the second facial image is constructed based on shape parameters of the second facial image, expression parameters of the adjusted first facial image and posture parameters of the first facial image; the expression parameters of the adjusted first facial image are obtained by adjusting the coefficients of each basis vector in the expression parameters of the first facial image; Performing a fusion process on the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate a to-be-fused area in the first facial image; Rendering the second facial image based on the fused texture image to obtain a target facial image; The rendering of the second face image based on the fused texture image to obtain a target face image includes: Mapping the texture value of each pixel in the fused texture image to the three-dimensional face model corresponding to the second face image to obtain a target face model; Drawing a third face image based on the target face model, wherein the third face image includes the area to be fused in the first face image; The third facial image is rendered into the second facial image according to a second dynamic mask to obtain a target facial image, wherein the second dynamic mask is used to indicate an area to be covered in the second facial image.

2. The method according to claim 1, characterized in that The step of fusing the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes: Mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space according to the first dynamic mask to obtain a texture mask; The first texture image is fused into the second texture image according to the texture mask to obtain a fused texture image.

3. The method according to claim 1, characterized in that Before fusing the first texture image and the second texture image based on the first dynamic mask to obtain the fused texture image, the method further includes: Performing facial skin detection on the first facial image to obtain a skin detection range of the first facial image; Performing key point detection on the first facial image to obtain a key point detection range of the first facial image; A first dynamic mask is calculated based on the skin detection range and the key point detection range.

4. The method according to claim 1, characterized in that: The obtaining of a first texture image corresponding to the first face image and a second texture image corresponding to the second face image includes: Inputting the first face image and the second face image into a pre-trained 3D face reconstruction network to perform 3D face reconstruction processing, thereby obtaining 3D model parameters of the first face image and 3D model parameters of the second face image; constructing a three-dimensional face model corresponding to the first face image and a three-dimensional face model corresponding to the second face image according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image; A first texture image corresponding to the first facial image is drawn based on the three-dimensional facial model corresponding to the first facial image, and a second texture image corresponding to the second facial image is drawn based on the three-dimensional facial model corresponding to the second facial image.

5. The method according to claim 4, characterized in that The three-dimensional model parameters include shape parameters, expression parameters, and posture parameters. The three-dimensional face model corresponding to the first face image and the three-dimensional face model corresponding to the second face image are constructed according to the three-dimensional model parameters of the first face image and the three-dimensional model parameters of the second face image, including: constructing a three-dimensional face model corresponding to the first face image according to the shape parameters, expression parameters and posture parameters of the first face image; A three-dimensional face model corresponding to the second face image is constructed according to the shape parameters of the second face image and the expression parameters and posture parameters of the first face image.

6. The method according to claim 5, characterized in that The expression parameters include at least two basis vectors, and constructing a three-dimensional face model corresponding to the second face image according to the shape parameters of the second face image, the expression parameters and the posture parameters of the first face image includes: Adjusting the coefficients of each basis vector in the expression parameters of the first facial image to obtain adjusted expression parameters; A three-dimensional face model corresponding to the second face image is constructed according to the shape parameters of the second face image, the adjusted expression parameters and the posture parameters of the first face image.

7. The method according to any one of claims 4 to 6, characterized in that: The step of drawing a first texture image corresponding to the first facial image based on the three-dimensional facial model corresponding to the first facial image, and drawing a second texture image corresponding to the second facial image based on the three-dimensional facial model corresponding to the second facial image, comprises: Mapping the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to the texture space to obtain a first texture image corresponding to the first face image; The texture value of each coordinate point in the three-dimensional face model corresponding to the second face image is mapped to the texture space to obtain a second texture image corresponding to the second face image.

8. The method according to claim 4, characterized in that The method further comprises: Acquire a training sample set, the training sample set including a face image and two-dimensional key point coordinates corresponding to the face image; Based on the training sample set, weakly supervised training is performed on the pre-constructed three-dimensional face reconstruction network, and a loss value of the three-dimensional face reconstruction network is calculated until the loss value meets a convergence condition to obtain a trained three-dimensional face reconstruction network.

9. The method according to claim 8, characterized in that The calculating the loss value of the three-dimensional face reconstruction network includes: In each round of training, determining loss parameters of the 3D face reconstruction network, wherein the loss parameters include a pixel loss value, a key point distance loss value, an identity loss value, and a regularization loss value; The loss parameters are weighted and summed according to preset weights to obtain the loss value of the three-dimensional face reconstruction network.

10. The method according to claim 1, characterized in that The number of the first face images is greater than 1, and the fusing process of the first texture image and the second texture image based on the first dynamic mask to obtain a fused texture image includes: Determine a first dynamic mask corresponding to each first face image respectively; According to the first dynamic mask corresponding to each first facial image, the first texture images corresponding to each first facial image are fused to obtain a third texture image; Determining a third dynamic mask corresponding to the third texture image; The third texture image and the second texture image are fused based on the third dynamic mask to obtain a fused texture image.

11. An image processing device, characterized in that: The device comprises: A texture image acquisition module, used to acquire a first texture image corresponding to a first facial image and a second texture image corresponding to a second facial image; the first texture image corresponding to the first facial image is drawn based on a three-dimensional facial model corresponding to the first facial image, and the second texture image corresponding to the second facial image is drawn based on a three-dimensional facial model corresponding to the second facial image; the three-dimensional facial model corresponding to the first facial image is constructed based on shape parameters, expression parameters and posture parameters of the first facial image; the three-dimensional facial model corresponding to the second facial image is constructed based on shape parameters of the second facial image, expression parameters of the adjusted first facial image and posture parameters of the first facial image; the expression parameters of the adjusted first facial image are obtained by adjusting the coefficients of each basis vector in the expression parameters of the first facial image; a texture image fusion module, configured to fuse the first texture image and the second texture image based on a first dynamic mask to obtain a fused texture image, wherein the first dynamic mask is used to indicate a to-be-fused area in the first facial image; A target image generation module is used to render the second facial image based on the fused texture image to obtain a target facial image; map the texture value of each pixel in the fused texture image to a three-dimensional facial model corresponding to the second facial image to obtain a target facial model; draw a third facial image based on the target facial model, the third facial image including the area to be fused in the first facial image; render the third facial image into the second facial image according to a second dynamic mask to obtain a target facial image, the second dynamic mask being used to indicate the area to be covered in the second facial image.

12. The device according to claim 11, characterized in that The texture image fusion module comprises: a texture mask acquisition submodule, configured to map the texture value of each coordinate point in the three-dimensional face model corresponding to the first face image to a texture space according to the first dynamic mask, so as to obtain a texture mask; The texture image fusion submodule is used to fuse the first texture image into the second texture image according to the texture mask to obtain a fused texture image.

13. A device for image processing, characterized in that: The device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for executing the image processing method as described in any one of claims 1 to 10.

14. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes a device to perform the image processing method according to any one of claims 1 to 10.