Image processing method, device, storage medium and electronic device

Abstract

The present specification discloses an image processing method, device, storage medium and electronic device, the method comprising: obtaining a source image and a reference image; performing semantic segmentation processing on the reference image to obtain a reference semantic segmentation map containing non-facial components of the reference image; processing the source image, the reference image and the reference semantic segmentation map to obtain a pending image that fuses the features of the source image and the reference image at the position corresponding to the facial component of the reference image; and performing color processing on the part of the pending image corresponding to the facial component of the reference image according to the key point information of the reference image to obtain a composite image. Through the above scheme, not only can the features of the source image be represented in the fusion result, but also the features of the facial components of the source image in the composite image can be matched with the area where the internal features of the reference image are located.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0002] In advertising, film and television, entertainment and other scenarios, facial image replacement is becoming more and more widely used.

[0003] Face replacement refers to replacing the face in the reference image with the face in the source image. The target composite image contains the face in the source image and the non-face parts such as the hair, body, and background of the person in the reference image. In other words, the goal of face replacement is to retain the identity information of the source image in the composite image (this identity information can be represented by several dimensions such as facial features, expression, and skin color), rather than replacing the identity information of the person in the source image with the identity information of the person in the reference image in the composite image.

[0004] The face-changing methods in the related art mostly encode the source image and the reference image, and then decode the encoded file to obtain a composite image. However, this face-changing method will merge the identity information of the person in the source image and the reference image, so that the composite image contains the identity information of the person in both the source image and the reference image. When the identity information of the person in the source image and the reference image is very different, this "identity confusion" phenomenon is more obvious. It can be seen that the face-changing method in the related art cannot achieve the target face-changing effect. Summary of the invention

[0005] The embodiments of this specification provide an image processing method, device, storage medium and electronic device to partially solve the above problems existing in the prior art.

[0006] The embodiments of this specification adopt the following technical solutions:

[0007] In a first aspect, the present application provides an image processing method, comprising:

[0008] Obtaining a source image and a reference image;

[0009] Performing semantic segmentation processing on the reference image to obtain a reference semantic segmentation map including non-facial components of the reference image;

[0010] Processing the source image, the reference image, and the reference semantic segmentation map to obtain a pending map that fuses features of the source image and the reference image at a position corresponding to a facial component of the reference image;

[0011] According to the key point information of the reference image, color processing is performed on the part of the undetermined image corresponding to the facial component of the reference image to obtain a composite image.

[0012] In an optional embodiment of the present specification, color processing is performed on the position of the internal component of the undetermined image corresponding to the reference image according to the key point information of the reference image to obtain a composite image, including:

[0013] Determine a hybrid weight map according to the key point information of the reference image; wherein the hybrid weight map reflects the weight distribution of different pixel points on the reference image, and the weight corresponding to the key point on the reference image is less than the weight corresponding to the non-key point on the reference image;

[0014] According to the mixed weight map and the reference image, color processing is performed on the positions of the facial components of the undetermined image corresponding to the reference image to obtain a composite image.

[0015] In an optional embodiment of the present specification, the source image, the reference image, and the reference semantic segmentation map are processed to obtain a pending map that fuses features of the source image and the reference image at a position corresponding to a facial component of the reference image, including:

[0016] fusing the source image and the reference image to obtain a first image including features of the source image and features of the reference image;

[0017] According to the boundary between the facial component and the non-facial component shown in the reference semantic segmentation graph, determining the portion of the first graph corresponding to the facial component as a reconstruction area;

[0018] The reconstruction area is reconstructed to obtain a pending map.

[0019] In an optional embodiment of the present specification, the source image and the reference image are fused to obtain a first image including features of the source image and features of the reference image, including:

[0020] Performing semantic segmentation processing on the source image to obtain a source semantic segmentation map containing facial components of the source image;

[0021] The first image is obtained based on the source semantic segmentation image, the source image and the reference image.

[0022] In an optional embodiment of the present specification, obtaining the first image based on the source semantic segmentation image, the source image and the reference image includes:

[0023] constructing a three-dimensional fusion model according to the source image and the reference image;

[0024] The first map is obtained according to the three-dimensional fusion model and the source semantic segmentation map.

[0025] In an optional embodiment of the present specification, the image processing method further includes:

[0026] Determine the key point detection model;

[0027] The key point detection model is used to perform key point detection on the reference image to obtain reference key point information of the reference image.

[0028] In an optional embodiment of the present specification, determining a key point detection model includes:

[0029] Determine the model to be trained;

[0030] Determine the first loss and the second loss based on the detection results output by the model to be trained based on the sample set; wherein the first loss is used to characterize the loss caused by at least one of the following: facial posture, expression, and degree of occlusion. The second loss is used to characterize the loss caused by the accuracy of the training samples and annotations;

[0031] Taking minimizing the sum of the first loss and the second loss as the training objective, the model training is performed until convergence to obtain the key point detection model.

[0032] In a second aspect, the present specification provides an image processing device, the device comprising:

[0033] An image acquisition module is configured to: acquire a source image and a reference image;

[0034] A semantic segmentation processing module is configured to: perform semantic segmentation processing on the reference image to obtain a reference semantic segmentation map containing non-facial components of the reference image;

[0035] A pending image generation module is configured to: process the source image, the reference image and the reference semantic segmentation map to obtain a pending image that integrates features of the source image and the reference image at a position corresponding to a facial component of the reference image;

[0036] The synthetic image module is configured to: perform color processing on the part of the face component of the pending image corresponding to the reference image according to the key point information of the reference image to obtain a synthetic image.

[0037] The electronic device provided in this specification includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the image processing method is implemented when the processor executes the program.

[0038] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0039] In the image processing method, device, storage medium and electronic device in the embodiments of this specification, in the process of obtaining a composite image, the reference image is segmented and the facial components in the reference image are segmented out, so that the reference semantic segmentation map is obtained, which retains the external components in the reference image. Thereafter, the composite image obtained based on the reference semantic segmentation map can also include the external components in the reference image. In addition, the method in this specification can fuse the features of the source image and the reference image at the position of the facial components of the reference image. On the one hand, the features of the source image can be represented in the fusion result; on the other hand, the facial component features of the source image in the composite image can be matched with the area where the internal features of the reference image are located. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0041] Figure 1 A flowchart of an image processing method provided in an embodiment of this specification;

[0042] Figure 2 A schematic diagram of a flow chart of obtaining an intermediate image in at least part of the steps of the image processing method provided in an embodiment of this specification;

[0043] Figure 3 A schematic diagram of a flow chart of semantic segmentation processing in the image processing method provided in the embodiments of this specification;

[0044] Figure 4 A schematic diagram of a flow chart of obtaining corresponding data by using a model in at least some stages of the image processing method provided in an embodiment of this specification;

[0045] Figure 5 A schematic diagram of the process of training a key point detection model using the image processing method provided in the embodiments of this specification;

[0046] Figure 6 A schematic diagram of the process of constructing a three-dimensional fusion model using the image processing method provided in the embodiments of this specification;

[0047] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of this specification;

[0048] Figure 8 The embodiments of this specification provide corresponding to Figure 1Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0050] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0051] In order to solve the problem in the prior art that face swapping may cause the loss of identity information of the person in the source image in the synthesized image, the present specification provides an image processing process so that the identity information of the person in the source image is retained as much as possible in the synthesized image.

[0052] Before describing the process in this specification, at least some of the concepts involved in this specification are described.

[0053] The source image and reference image in this specification can be a frame of image obtained by shooting; or it can be a frame of image in a video. By implementing the image processing process in this specification on at least part of the frames in the video, the face of the person in the video can be replaced.

[0054] In the definition of this specification, the foreground of the image is composed of several components, so that the parts other than the internal components are exposed in the various parts of the human body represented by the image, and the parts other than the internal components are external components. The internal components include at least one of the following: eyes, eyebrows, nose, mouth, ears. The external components include at least one of the following: hair, cheeks, forehead. In addition, the part of the human body not exposed in the image can be the foreground part blocked by clothes. In order to conform to the image processing process in this specification, this specification divides the composition of the image into facial components and non-facial components. Facial components are components belonging to the face of the human body among the various parts of the human body exposed in the image, and can include: eyes, eyebrows, nose, mouth, ears, cheeks, forehead. Non-facial components are components in the image other than facial components, and can include: hair, arms, clothes, background.

[0055] The identity information referred to in this specification is information represented by facial components and used to distinguish people. For example, the identity information may include at least one of the following: the shape of facial features, the proportion of facial features, the relative position of facial features on the face, the expression of the person, and the skin color of the person.

[0056] The characters in this manual may be real people (such as Zhang San) or virtual people (such as a cartoon character).

[0057] Key points are pre-defined points that can reflect at least some of the facial component features of a face, such as the left inner eye corner, the right mouth corner, the nose tip, etc. The key points of a face can be identified by numbers. For example, the key point set M = [M1, M2...Mn], numbers 1 to n correspond to a face key point in a certain order, then n is the number of key points, and the value of n is determined by the number of face key points to be detected that is pre-set by the face recognition algorithm. Commonly used n can be 68, 83, 101, etc., and Mn represents the two-dimensional coordinates (xn, yn) of the face key point numbered n.

[0058] Figure 1 The image processing process provided in the embodiments of this specification may specifically include one or more of the following steps:

[0059] S100: Acquire a source image and a reference image.

[0060] The method in this specification is performed by an image processing device. The process in this specification is to replace the face of a person in a source image with a reference image to obtain a composite image. The source image and the reference image both contain the face of a person (ie, a human face).

[0061] The acquisition operation performed on the source image and / or the reference image in this step may be performed by the user uploading the source image and / or the reference image to the image processing device so that the image processing device acquires the image; or the image processing device may perform frame extraction processing on the video to acquire the image. This specification may also have other ways of acquiring images, which are not described in detail here.

[0062] S102: Performing semantic segmentation processing on the reference image to obtain a reference semantic segmentation map including non-facial components of the reference image.

[0063] For example, the image processing process in this specification is as follows: Figure 2 As shown. This step can at least distinguish the area where the facial component of the reference image is located in the reference image from the area where the non-facial component is located in the reference image. The reference semantic segmentation map obtained by this step can at least represent: the area where the non-facial component in the reference image is located in the reference image.

[0064] In an optional embodiment of the present specification, an image is composed of a plurality of pixels, and the image can be represented as a collection of information represented by each pixel contained therein. Then the semantic segmentation processing performed in this step can be: among the pixels constituting the reference image, a pixel corresponding to the facial component of the reference image is determined as the first pixel. The first pixel is deleted from the pixels constituting the reference image to obtain a reference semantic segmentation map.

[0065] It can be seen that this step aims to segment part of the facial components of the reference image from the reference image, rather than distinguishing between the various facial components. In other words, the segmentation in this specification only needs to distinguish the face of the person, without distinguishing the eyes and nose in the face. It can be seen that the amount of data processed by the semantic segmentation processing in this specification is less than the amount of data processed when distinguishing between facial components, which is conducive to reducing the resources consumed in the image processing process.

[0066] S104: Processing the source image, the reference image and the reference semantic segmentation map to obtain a pending map that fuses features of the source image and the reference image at a position corresponding to a facial component of the reference image.

[0067] The processing in this step is intended to form an image that fuses the features of the facial components of the reference image and the features of the facial components of the source image at the locations of the facial components of the reference image, while retaining the features of the reference image at the locations of the non-facial components.

[0068] In an optional embodiment of the present specification, the process of obtaining the pending image may be: obtaining the features of the facial components of the source image as the first feature; obtaining the features of the facial components of the reference image as the second feature. The first feature and the second feature are fused to form a synthetic feature. The synthetic feature is added to the area of ​​the facial component represented in the reference semantic segmentation map to obtain the pending image. Optionally, the first feature in the present specification is used to characterize the position of the key points of the source image, and the second feature is used to characterize the position of the key points in the reference image. Then the synthetic feature can at least represent: when the key points corresponding to the facial components can match the non-facial components of the reference image, the position of the key points of the source image in the pending image.

[0069] That is to say, the undetermined image obtained through this step can at least indicate the position of elements representing identity information such as facial features of the person in the source image in the target synthetic image.

[0070] S106: Performing color processing on the portion of the pending image corresponding to the facial component of the reference image according to the key point information of the reference image to obtain a composite image.

[0071] The image processing process in this specification performs color processing on the position of the facial components on the determined image corresponding to the reference image based on the color information of the source image (which can represent the skin color of the characters in the source image). This can replace the face in the source image with the reference image while making the characters on the resulting composite image reflect the skin color of the characters in the source image.

[0072] In an optional embodiment of the present specification, the color information of the pixel points corresponding to the facial components on the source image can be used as the target information. The target information is used as the color information of the pixel points corresponding to the facial components on the pending image; and the key point information of the reference image is used to perform color processing on the part of the pending image corresponding to the facial components of the reference image. Specifically, the pixel points in the pending image corresponding to the key points of the reference image are determined as designated points. The color of the pixel points in the reference image corresponding to the non-key points of the pending image is used to render the corresponding points on the pending image to obtain a composite image.

[0073] In the image processing method in the embodiment of this specification, in the process of obtaining the composite image, the reference image is segmented and the facial components in the reference image are segmented out, so that the reference semantic segmentation map retains the non-facial components in the reference image. After that, the composite image obtained based on the reference semantic segmentation map can also include the non-facial components in the reference image. In addition, the method in this specification can fuse the features of the source image and the reference image at the position of the facial components of the reference image. On the one hand, the features of the source image can be represented in the fusion result; on the other hand, the facial component features of the source image in the composite image can be matched with the area where the internal features of the reference image are located. Further, the process in this specification also performs color processing on the part of the subject image corresponding to the facial components of the reference image according to the key point information of the reference image, so that the skin color (a kind of identity information) of the source image can be restored in the area corresponding to the facial components of the reference image in the composite image. In addition, the image processing process in this specification can restore the skin color of the person in the source image to the composite image, and can make the skin color of the person in the composite image reflect the light perception characteristics of the face in the reference image caused by factors such as angle and face shape.

[0074] For example, the skin color of the person in the color-processed composite image at the position corresponding to the key point retains the skin color of the person in the source image. At the position of non-key points, such as the hairline and chin, the skin color of the person in the source image is retained, and the "highlight" effect caused by the position of the light source relative to the person and the light intensity in the reference image can be reflected.

[0075] In such Figure 2As shown in the a area, the source image and the reference image are first fused to obtain a first image including the features of the source image and the features of the reference image. Then, the first image is processed to obtain a pending image.

[0076] The process of obtaining the first image is now described. In an optional embodiment of the present specification, the process of obtaining the first image may include at least one of the following stages: semantic segmentation, key point information extraction, and face fitting.

[0077] 1. Semantic segmentation.

[0078] The semantic segmentation in this specification can at least determine the boundary between facial components and non-facial components. In an optional embodiment of this specification, the semantic segmentation process is performed by a semantic segmentation model. Figure 3 As shown, the process of performing semantic segmentation processing on an image (for example, a source image and a reference image in this specification) may be:

[0079] S300: Make a bounding rectangle for the area in the image that contains the person's face, and use the area enclosed by the bounding rectangle as a sub-image.

[0080] In an optional embodiment of the present specification, a face detector (such as BlazeFace) is used to process the image I to determine the size of the bounding rectangle and the position of the bounding rectangle in the image.

[0081] According to the detection results (including the detected key points) of the key point detection model (which will be introduced below), the area selected by the circumscribed rectangle in the image is aligned.

[0082] Afterwards, RoI Tanh-warping is used to warp the result obtained by the alignment process into a fixed scale (this process can be called forward deformation processing) to obtain a sub-image I'.

[0083] The semantic segmentation model in this specification includes: a first backbone network, a feature extraction module and an output module. The first backbone network can be a MobileNetV3 lightweight network. The sub-image in this step can be obtained based on the first backbone network. The MobileNetV3 lightweight network has the characteristics of small size and fast data processing speed.

[0084] S302: Perform feature extraction on the sub-image to obtain facial component features and / or non-facial component features. The facial component features can at least represent the area on the sub-image that corresponds to the face of the person, and the non-facial component features can at least represent the area on the sub-image that does not correspond to the face of the person. It can be seen that the facial component features and the non-facial component features have a certain degree of mutual exclusivity when representing their corresponding relationship with the face of the person.

[0085] Then in an optional embodiment of the present specification, for each pixel point on the sub-image, the facial component feature of the pixel point is determined (including but not limited to the probability that the pixel point is a point on the facial component), and the non-facial component feature of the pixel point is determined (including but not limited to the probability that the pixel point is not a point on the facial component). If the characteristic of the facial component feature of the pixel point is stronger than the characteristic of the non-facial component feature, the pixel point is determined to be a pixel point on the facial component; otherwise, the pixel point is determined to be a pixel point on the non-facial component.

[0086] In an optional embodiment of the present invention, a feature extraction module of a semantic segmentation model may be used, wherein the feature extraction module includes multiple convolutional layers to extract global features of the sub-images. Then, facial component features and / or non-facial component features are obtained based on the global features.

[0087] S304: Obtain an intermediate semantic graph according to facial component features and / or non-facial component features.

[0088] Each pixel in the sub-image may be clustered according to the characteristic features of the facial component features and / or non-facial component features to obtain an intermediate semantic map. The intermediate semantic map obtained by this step can at least represent the area corresponding to the non-facial component in the sub-image.

[0089] S306: Use RoI Tanh-warping to perform reverse deformation processing on the intermediate semantic map to obtain a semantic segmentation map.

[0090] In an optional embodiment of the present specification, this step can be performed by the output module of the semantic segmentation model. Figure 4 As shown, when the processing object of the semantic segmentation processing is the source image, the output of this step is the source language segmentation map; when the processing object of the semantic segmentation processing is the reference image, the output of this step is the reference language segmentation map.

[0091] It can be seen that in the image processing process in this specification, the source image can be semantically segmented to obtain a source semantic segmentation map containing the facial components of the source image; based on the source semantic segmentation map, the source image and the reference image, the first image is obtained.

[0092] 2. Extraction of key point information.

[0093] The key point information in this specification can at least represent the position of at least some points (which can be pixels) on the face of a person in an image (for example, the source image and reference image in this specification) and the components to which they belong. For example, the coordinates of pixel a on the image are (xa, ya), and pixel a is the point at the inner corner of the right eye.

[0094] The key point information extraction can be implemented using the key point detection model designed in this specification. The key point detection model includes: a second backbone network, a convolutional network, and a fully connected network. Optionally, the second backbone is an optimized MobileNetV3 network. Compared with the general MobileNetV3 network, the Inverted ResidualBlock in the optimized MobileNetV3 network is replaced with a Ghost Module. In addition, this specification also appropriately reduces the number of convolution kernels and units of the convolutional network and the fully connected network to reduce the number of model parameters.

[0095] The process of obtaining the key point information of the image may be to input the image into the key point detection model and obtain the key point information output by the key point detection model.

[0096] It can be seen that the key point information extraction effect in this description is related to the performance of the key point detection model. Figure 4 and Figure 5 As shown, this specification provides a process for training a key point detection model:

[0097] S500: Determine a model to be trained.

[0098] After determining the second backbone network, the convolutional network, and the fully connected network, at least some of their parameters are initialized to obtain a model to be trained.

[0099] S502: Obtain the detection result output by the model to be trained according to the sample set.

[0100] S504: Determine the first loss and the second loss according to the detection result.

[0101] The first loss is used to characterize the loss caused by at least one of the following: facial posture, expression, and occlusion degree. The second loss represents the loss caused by the accuracy of training samples and annotations.

[0102] In an optional embodiment of the present specification, the first loss (L1) is based on: the number of samples in the sample set (M), the number of predicted key points (N), the yaw angle of the face toward a certain direction (θ 1 ), pitch angle (θ 2 ) and roll angle (θ 3), the type of face (C, such as frontal face, side face, head-up, head-down expression, and degree of occlusion) is calculated.

[0103] Among them, the type of face can be specified by weight Characterization, the role of the specified weight is: for data with a relatively large number of samples (such as frontal faces, that is, the Euler angles are relatively small), a small specified weight is given, so that when the gradient is back-propagated, the contribution to the model training is smaller; for data with a relatively small number of samples (side faces, lowered heads, raised heads, extreme expressions), a larger specified weight is given, so that when the gradient is back-propagated, the contribution to the model training is greater.

[0104] The first loss (L1) can be calculated by the following formula 1.

[0105]

[0106] In the formula, ‖·‖ is an evaluation index used to measure the prediction error of the nth key point of the mth sample. The evaluation index can be L2 loss, that is, Ridge loss.

[0107] The second loss is used to characterize the loss caused by the accuracy of the annotation of the sample set. In an optional embodiment of the present specification, the second loss can be calculated by the following formula 2.

[0108]

[0109] Among them, t is a coefficient related to training accuracy. The higher the training accuracy requirement, the smaller t is, which is used to limit the range of the nonlinear part to [-t, t]. The value of ∈ is a very small value, because it will make the network training unstable and cause the gradient explosion problem due to a small error. F = tt ln(1+∈x). x is the coordinate of the key point in the image.

[0110] S506: Taking minimizing the sum of the first loss and the second loss as the training objective, perform model training until convergence to obtain a key point detection model.

[0111] In addition, in order to improve the key point detection effect of the key point detection model under conditions of closed eyes and large postures (such as lowering the head), the sample augmentation method is adopted to amplify the data of a certain face type with a small amount of data in the sample set, and the amplified sample set is used for training.

[0112] The key point information obtained by the key point detection model in this specification can be used to determine the size of the bounding rectangle and the position of the bounding rectangle in the image in step S200. Specifically, when aligning the area selected by the bounding rectangle in the image in step S200, the key point information output by the key point detection model is used as a basis.

[0113] 3. Facial fitting.

[0114] The facial features in this specification can be regarded as a collection of key point information. At the micro level, facial features at least indicate the position of the key points on the image; at the macro level, facial features can indicate the orientation of the person's face, the shape of the facial features, the proportions of the facial features, the relative positions of the facial features on the face, the person's expression, etc.

[0115] The facial fitting in this specification aims to fuse the facial features of the source image and the facial features of the reference image, so that the fused synthetic features can represent the shape of the facial features, the proportion of the facial features, the relative position of the facial features on the face, the expression of the person in the source image, and the orientation of the face of the person in the reference image. In this way, the synthetic image obtained based on the synthetic features can contain the identity information of the person in the source image, and can also match the facial components of the person in the synthetic image with the non-facial components to avoid the phenomenon of "disharmony".

[0116] In an optional embodiment of this specification, if Figure 4 and Figure 6 As shown, the following steps can be used to achieve face fitting:

[0117] S600: Obtaining first model parameters according to key point information of the source image; and obtaining second model parameters according to key point information of the reference image.

[0118] The first model parameter is used to characterize the three-dimensional features of the person in the source image, and the second model parameter is used to characterize the three-dimensional features of the person in the reference image.

[0119] In an optional embodiment of the present specification, a reference face model set can be used to obtain the first model parameter and / or the second model parameter. Specifically, a reference face model set is first determined. The reference face model set includes several reference three-dimensional face models. The reference face model set can be represented as Shapes = [S_1, S_2, ... S_ n ], where S_ n is the nth reference 3D facial model. Different reference 3D facial models represent different facial orientations, facial features, expressions, etc.

[0120] Then, each reference 3D facial model in the reference facial model set is used to fit the key point information in the image. For example, for any reference 3D facial model, a projection of the reference 3D facial model in the plane where the image is located is determined as the reference image. With the goal of minimizing the difference between the key point information of the reference image and the key point information of the image (e.g., source image, reference image), the mixing parameter P of the reference 3D facial model is determined. Mix , a rotation matrix, a translation amount and a scaling amount are used as model parameters of the reference 3D facial model. The set of model parameters of each reference 3D facial model in the reference facial model set is used as the first model parameter (in this case, the aforementioned image is the source image) or the second model parameter (in this case, the aforementioned image is the reference image).

[0121] Among them, the model parameters of the i-th reference 3D facial model can be expressed as: rotation matrix

[0122] Specifically, the process of obtaining the model parameters can be expressed by the following formula 3 and formula 4:

[0123]

[0124] L p =S*R*(P Mix ·Shapes)+T Formula 4

[0125] In the formula, Loss L is the difference between the key point information and the key point information of the image. The process of determining the model parameters is to find the Loss L Minimum process. L real is the coordinate of the key point represented by the key point information, L proj It is the key coordinate point of each benchmark 3D facial model projected onto the plane where the image is located (represented by the key point information of the benchmark figure). p For a three-dimensional model constructed based on undetermined model parameters, before the model parameters are solved, the model parameters determined by Formula 3 and Formula 4 can be referred to as undetermined model parameters.

[0126] In an optional embodiment of the present specification, the Gauss-Newton method may be used to iteratively solve the model parameters.

[0127] S602: construct a first three-dimensional model according to the first model parameters; and construct a second three-dimensional model according to the second model parameters.

[0128] Specifically, each reference 3D face model in the reference face model set can be adjusted according to the first model parameters obtained in the above steps to obtain a first 3D model S SAccording to the second model parameters obtained in the previous step, each reference three-dimensional face model in the reference face model set is adjusted to obtain a second three-dimensional model S T .

[0129] S604: Construct a three-dimensional fusion model according to the first three-dimensional model and the second three-dimensional model.

[0130] The three-dimensional fusion model in this specification includes the features of the first three-dimensional model and the second three-dimensional model. In an optional embodiment of this specification, the process of constructing the three-dimensional fusion model can be: aligning the first three-dimensional model and the second three-dimensional model using the Proctor analysis method to obtain the three-dimensional fusion model S ALIGNED .

[0131] In order to make the three-dimensional fusion model S ALIGNED It is possible to more accurately represent the structural information of the face. In an optional embodiment of this specification, the three-dimensional fusion model S ALIGNED As the pending model, align the pending model to the source semantic segmentation map, and update the 3D fusion model S according to the obtained results. ALIGNED Later, based on the new 3D fusion model S ALIGNED , perform the subsequent steps.

[0132] It can be seen that in the image processing process in this specification, a three-dimensional fusion model can be constructed based on the source image and the reference image; and the first image can be obtained based on the three-dimensional fusion model and the source semantic segmentation map.

[0133] 4. Face reconstruction.

[0134] Since the facial contours of the characters in the source image and the reference image are likely to be different, and in some cases, due to the occlusion of external components such as hair, the difference may be more obvious. In other words, it is possible that directly adding the first image obtained in the above step to the facial position of the reference semantic segmentation image will result in some positions not being fully filled. In this case, it is necessary to reconstruct the face based on the first image obtained in the above step.

[0135] In an optional embodiment of the present specification, according to the boundary between the facial component and the non-facial component shown in the reference semantic segmentation graph, the portion of the first graph corresponding to the facial component can be determined as the reconstruction area. Then, the reconstruction area is reconstructed to obtain a pending graph.

[0136] Specifically, the face reconstruction in this specification is implemented using the reconstruction model Gr. Optionally, the reconstruction model Gr uses ResNet18 as the backbone network and is trained with pixel-by-pixel l1 loss - that is, MAE (mean abserror) - as the loss function. The dataset used to train the reconstruction model Gr is CelebAMask-HQ, and each sample image in CelebAMask-HQ corresponds to the segmentation mask of the face attributes of CelebA.

[0137] The process of training the reconstruction model Gr can be: preprocessing the data in the dataset so that each sample in the dataset contains the facial component region I F and the face mask M F For each sample, the mask M of the sample F Perform random local occlusion processing to obtain the processed facial component area I′ F and the processed face mask M F '. Take the processed facial component area I' F , and the facial mask M F As the input of the reconstruction model Gr, the facial component region I F , and the processed face mask M F ′ is taken as output and the reconstruction model Gr is trained until convergence.

[0138] After the reconstruction model Gr is trained, the first image and the reference semantic segmentation component are used as inputs of the reconstruction model Gr, and the output of the reconstruction model Gr includes a reconstructed facial image, which is used as a pending image in this specification.

[0139] 5. Color processing.

[0140] The first image obtained by the aforementioned "face fitting" step can reflect the structural information of the synthesized face, such as the position and expression of the facial features. However, the first image at this time cannot reflect the skin color of the person. In an optional embodiment of this specification, the first image output by the aforementioned "face fitting" is used as the image to be processed. Obtaining the three-dimensional fusion model S ALIGNED (It can be the updated 3D fusion model S ALIGNED ) Mesh information. Based on Mesh information and 3D fusion model S ALIGNED The key point coordinates obtained by projection (i.e., the aforementioned “L proj ”), rendering the colors of the key points of the source image to the image to be processed, and updating the first image to the result obtained by the rendering. The first image obtained in this embodiment can reflect the skin color of the person in the source image.

[0141] As can be seen from the foregoing, the image processing process in this specification can at least achieve the retention of the identity information of the source image in the resulting composite image to a large extent, and the identity information includes skin color. On the one hand, adding the skin color of the face of the person in the source image to the part of the composite image corresponding to the face of the person can achieve the retention of the identity information of the person in the source image; but on the other hand, due to the different environments, lighting conditions and other conditions of the people in the source image and the reference image, it is possible that there is a sense of disharmony between the face and other parts of the person in the composite image. If the face of the person in the source image is mechanically replaced with the person in the reference image, the area of ​​the face and the non-face area on the resulting composite image will have different light perceptions, causing the face in the composite image to be "disharmonic" with other areas.

[0142] In view of this, in an optional embodiment of the present specification, a hybrid weight map is determined according to the key point information of the reference image obtained in the above steps. According to the hybrid weight map and the reference image, the position of the facial component of the determined image corresponding to the reference image is color processed to obtain a composite image.

[0143] The mixed weight map reflects the weight distribution of different pixels on the reference image, and the weight corresponding to the key points on the reference image is smaller than the weight corresponding to the non-key points on the reference image. For example, the weight corresponding to the key points is 0.05, and the weight corresponding to the non-key points is 0.1.

[0144] Specifically, when obtaining a composite image based on a hybrid weight map, for each reference pixel on the reference image, the weight value corresponding to the reference pixel can be determined from the hybrid weight map, and the target pixel corresponding to the reference pixel can be determined from the pending image (optionally, when the pending image and the reference pixel have the same size and are in the same coordinate system, if the coordinates of a reference pixel are the same as the coordinates of a pixel on the pending image, then the pixel on the pending image is the target pixel corresponding to the reference pixel). The product of the color value of the reference pixel and its corresponding weight value is superimposed on the color value of the corresponding target pixel to obtain the color value of the composite pixel corresponding to the reference pixel on the composite image.

[0145] It can be seen that the image processing process in this specification can restore the skin color of the characters in the source image to the synthesized image, and at the same time, can make the skin color of the characters in the synthesized image reflect the light characteristics of the faces in the reference image caused by factors such as angle and face shape, so that the faces in the source image in the synthesized image are consistent with the external components of the reference image, thereby minimizing the sense of disharmony caused by face swapping.

[0146] Based on the same idea, the present specification also provides a corresponding embodiment Figure 1 An image processing device for the process shown in FIG. Figure 7 shown.

[0147] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this specification. The image processing device may include one or more of the following modules:

[0148] The image acquisition module 700 is configured to: acquire a source image and a reference image;

[0149] The semantic segmentation processing module 702 is configured to: perform semantic segmentation processing on the reference image to obtain a reference semantic segmentation map including non-facial components of the reference image;

[0150] The pending image generation module 704 is configured to: process the source image, the reference image and the reference semantic segmentation map to obtain a pending image that integrates features of the source image and the reference image at a position corresponding to a facial component of the reference image;

[0151] The synthetic image module 706 is configured to: perform color processing on the part of the pending image corresponding to the facial component of the reference image according to the key point information of the reference image, so as to obtain a synthetic image.

[0152] In an optional embodiment of the present specification, the composite image module 706 is specifically configured as follows: determining a hybrid weight map based on key point information of the reference image; wherein the hybrid weight map reflects the weight distribution of different pixel points on the reference image, and the weights corresponding to the key points on the reference image are smaller than the weights corresponding to the non-key points on the reference image; and performing color processing on the positions of the facial components of the undetermined image corresponding to the reference image based on the hybrid weight map and the reference image to obtain a composite image.

[0153] In an optional embodiment of the present specification, the pending image generation module 704 is specifically configured to: fuse the source image and the reference image to obtain a first image that includes features of the source image and features of the reference image; determine the portion of the first image that corresponds to the facial component as a reconstruction area according to the boundaries between facial components and non-facial components shown in the reference semantic segmentation chart; and reconstruct the reconstruction area to obtain the pending image.

[0154] In an optional embodiment of the present specification, the pending image generation module 704 is specifically configured to: perform semantic segmentation processing on the source image to obtain a source semantic segmentation map containing facial components of the source image; and obtain the first map based on the source semantic segmentation map, the source image and the reference image.

[0155] In an optional embodiment of the present specification, the pending image generation module 704 is specifically configured to: construct a three-dimensional fusion model according to the source image and the reference image; and obtain the first image according to the three-dimensional fusion model and the source semantic segmentation image.

[0156] In an optional embodiment of the present specification, the image processing device also includes a key point detection module 708, which is configured to: determine a key point detection model; use the key point detection model to perform key point detection on the reference image to obtain reference key point information of the reference image.

[0157] In an optional embodiment of the present specification, the image processing device further includes a training module 710, configured to: determine a model to be trained; determine a first loss and a second loss according to the detection result output by the model to be trained based on the sample set; wherein the first loss is used to characterize the loss caused by at least one of the following: facial posture, expression, and degree of occlusion. The second loss is used to characterize the loss caused by the accuracy of the training samples and annotations; with the sum of the first loss and the second loss minimized as the training goal, perform model training until convergence, and obtain a key point detection model.

[0158] The embodiments of this specification also provide a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides the process of image processing.

[0159] The embodiments of this specification also propose Figure 8 The schematic structure diagram of the electronic device shown in FIG. Figure 8 At the hardware level, the electronic device may include a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement any of the above-mentioned image processing processes.

[0160] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as the combination of logic device XOR software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic device.

[0161] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0162] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0163] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0164] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0165] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0175] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0176] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. An image processing method, characterized in that: include: Obtaining a source image and a reference image; Performing semantic segmentation processing on the reference image to obtain a reference semantic segmentation map including non-facial components of the reference image; Processing the source image, the reference image, and the reference semantic segmentation map to obtain a pending map that fuses features of the source image and the reference image at a position corresponding to a facial component of the reference image; According to the key point information of the reference image, color processing is performed on the part of the image to be determined corresponding to the facial component of the reference image to obtain a composite image; The key point information is obtained by performing key point detection on the reference image through a key point detection model, and the parameters related to the training target of the key point detection model at least include: a first loss and a second loss; The first loss is used to characterize the loss caused by at least one of the following: facial posture, expression, and occlusion degree, and the second loss represents the loss caused by the accuracy of training samples and annotations.

2. The method according to claim 1, characterized in that According to the key point information of the reference image, color processing is performed on the position of the internal component of the undetermined image corresponding to the reference image to obtain a composite image, including: Determine a hybrid weight map according to the key point information of the reference image; wherein the hybrid weight map reflects the weight distribution of different pixel points on the reference image, and the weight corresponding to the key point on the reference image is less than the weight corresponding to the non-key point on the reference image; According to the mixed weight map and the reference image, color processing is performed on the positions of the facial components of the undetermined image corresponding to the reference image to obtain a composite image.

3. The method according to claim 1, characterized in that The source image, the reference image, and the reference semantic segmentation map are processed to obtain a pending map that fuses features of the source image and the reference image at a position corresponding to a facial component of the reference image, including: fusing the source image and the reference image to obtain a first image including features of the source image and features of the reference image; According to the boundary between the facial component and the non-facial component shown in the reference semantic segmentation graph, determining the portion of the first graph corresponding to the facial component as a reconstruction area; The reconstruction area is reconstructed to obtain a pending map.

4. The method according to claim 3, characterized in that The source image and the reference image are fused to obtain a first image including features of the source image and features of the reference image, including: Performing semantic segmentation processing on the source image to obtain a source semantic segmentation map containing facial components of the source image; The first image is obtained based on the source semantic segmentation image, the source image and the reference image.

5. The method according to claim 4, characterized in that The first image is obtained based on the source semantic segmentation image, the source image and the reference image, including: constructing a three-dimensional fusion model according to the source image and the reference image; The first map is obtained according to the three-dimensional fusion model and the source semantic segmentation map.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Determine the key point detection model; The key point detection model is used to perform key point detection on the reference image to obtain reference key point information of the reference image.

7. The method according to claim 6, characterized in that Determine the key point detection model, including: Determine the model to be trained; Determine a first loss and a second loss according to the detection results output by the model to be trained based on the sample set; wherein the first loss is used to characterize the loss caused by at least one of the following: facial posture, expression, and occlusion degree; and the second loss is used to characterize the loss caused by the accuracy of the training samples and annotations; Taking minimizing the sum of the first loss and the second loss as the training objective, the model training is performed until convergence to obtain the key point detection model.

8. An image processing device, characterized in that: The device comprises: An image acquisition module is configured to: acquire a source image and a reference image; A semantic segmentation processing module is configured to: perform semantic segmentation processing on the reference image to obtain a reference semantic segmentation map containing non-facial components of the reference image; A pending image generation module is configured to: process the source image, the reference image and the reference semantic segmentation map to obtain a pending image that integrates features of the source image and the reference image at a position corresponding to a facial component of the reference image; A synthetic image module is configured to: perform color processing on a portion of the image to be determined corresponding to a facial component of the reference image according to key point information of the reference image, so as to obtain a synthetic image; wherein the key point information is obtained by performing key point detection on the reference image by a key point detection model, and parameters related to a training target of the key point detection model at least include: a first loss and a second loss; The first loss is used to characterize the loss caused by at least one of the following: facial posture, expression, and occlusion degree, and the second loss represents the loss caused by the accuracy of training samples and annotations.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps of the image processing method described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images