Image processing method, device and electronic equipment

Abstract

The embodiment of the present invention provides an image processing method, device, system and electronic device, wherein the method includes: obtaining a first face image and a second face image; respectively determining the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image; according to the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image, correspondingly replacing the face area pixels in the second face image with the face area pixels in the first face image to form a transformed second face image. The scheme of the embodiment of the present invention can realize face image swapping at large angles, make the face posture and expression after swapping more natural, and improve the generalization ability of image swapping.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, device and electronic equipment. Background Art

[0002] Image face swapping means that if there are two face images, face image A and face image B, the task of image face swapping is to replace the face in face image B with the face in face image A, and at the same time, the face image formed after replacement still maintains the posture and expression of the face in the original face image B. Image face swapping has a broad application prospect. For example, in film and television production, the entire TV series or movie can be shot through a stand-in, and the face of the protagonist of the video can be swapped later through image face swapping technology to reduce the shooting cost; in terms of user entertainment, users can choose their favorite TV series and replace the characters of interest with themselves, so that everyone can be the protagonist, which not only entertains the audience, but also attracts a lot of traffic for websites, businesses, etc.

[0003] The existing image face swapping method mainly divides the face of face image A into multiple small areas (such as 3mmx3mm), and then fits the divided small areas to the corresponding positions of the face of face image B to achieve image face swapping. This image face swapping method is relatively rough and is only applicable to restricted scenarios (such as face images with fixed posture, expression, lighting, etc.): First, face image B needs to be carefully designed, such as the posture must be frontal, the expression must be natural, the face must be a common face, and the face image quality requirements are high; secondly, face image A also has similar requirements, and the posture and expression are as consistent as possible with face image B.

[0004] Therefore, the existing image face-swapping solutions have poor generalization capabilities and are unable to achieve face-swapping of facial images at large angles and postures. For exaggerated expressions, or when there is a large difference in facial expressions between face image A and face image B, the swapped faces are easily severely distorted and unnatural, which cannot meet the needs of actual application scenarios. Summary of the invention

[0005] The present invention provides an image processing method, device and electronic device, which can realize face swapping of facial images at large angles, make the posture and expression of the swapped face more natural, and improve the generalization ability of image face swapping.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, an image processing method is provided, comprising:

[0008] Acquire a first face image and a second face image;

[0009] Determine mapping relationship information between pixel points in the face area of ​​the first face image and the second face image and pixel points in the set face image respectively;

[0010] According to the mapping relationship information between the pixel points in the face area of ​​the first face image and the second face image and the pixel points in the set face image, the pixel points in the face area of ​​the second face image are replaced with the pixel points in the face area of ​​the first face image to form a transformed second face image.

[0011] In a second aspect, an image processing device is provided, comprising:

[0012] A face image acquisition module, used to acquire a first face image and a second face image;

[0013] A mapping information determination module, used to respectively determine mapping relationship information between pixel points in the face area of ​​the first face image and the second face image and pixel points in the set face image;

[0014] A face transformation processing module is used to replace the face area pixels in the second face image with the face area pixels in the first face image according to the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image, so as to form a transformed second face image.

[0015] In a third aspect, an image processing method is provided, comprising:

[0016] Collect the face image to be replaced and determine the face image to be replaced;

[0017] Determine mapping relationship information between pixel points in the face area of ​​the face image to be replaced and the face image to be replaced and pixel points in the set face image respectively;

[0018] According to the mapping relationship information between the face image to be replaced and the face area pixels in the replaced face image and the pixel positions in the set face image, the face area pixels in the replaced face image are correspondingly replaced with the face area pixels in the face image to be replaced to form a transformed replaced face image.

[0019] In a fourth aspect, an electronic device is provided, including:

[0020] Memory, used to store programs;

[0021] The processor is coupled to the memory and is used to execute the program. When the program is running, the image processing method provided by the present invention is executed.

[0022] The present invention provides an image processing method, device and electronic device, which pre-sets a set face image; obtains a first face image and a second face image, and respectively determines the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image; according to the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image, the face area pixels in the second face image are correspondingly replaced with the face area pixels in the first face image to form a transformed second face image, thereby achieving the purpose of replacing the face in the second face image with the face in the first face image. In this scheme, the pixel positions in the set face image are used as a conversion bridge, so that a mapping relationship is established between the face area pixels of the two face images that are replaced and replaced. Since the number of pixels is relatively dense, it can solve the problems of unnatural image distortion caused by large-angle postures and exaggerated expressions in image face swapping, and improve the generalization ability of image face swapping.

[0023] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0025] Figure 1 A schematic diagram of image processing logic according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the training logic of the U-Net neural network model according to an embodiment of the present invention;

[0027] Figure 3 Constructing a logical schematic diagram for the Laplacian pyramid of an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of the Laplacian pyramid fusion logic of an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a Gaussian pyramid reconstruction logic according to an embodiment of the present invention;

[0030] Figure 6 The image processing method of the embodiment of the present invention is as follows: Figure 1 ;

[0031] Figure 7 The image processing method of the embodiment of the present invention is as follows: Figure 2 ;

[0032] Figure 8 The image processing method of the embodiment of the present invention is as follows: Figure 3 ;

[0033] Fig. 9 The image processing method of the embodiment of the present invention is as follows: Figure 4 ;

[0034] Fig.10 The image processing method of the embodiment of the present invention is as follows: Figure 5 ;

[0035] Fig.11 The structure of the image processing device according to the embodiment of the present invention is Figure 1 ;

[0036] Fig.12 A structural diagram of a face transformation processing module according to an embodiment of the present invention;

[0037] Fig.13 The structure of the image processing device according to the embodiment of the present invention is Figure 2 ;

[0038] Fig.14 The structure of the image processing device according to the embodiment of the present invention is Figure 3 ;

[0039] Fig.15 This is a structural diagram of a skin color fusion processing module according to an embodiment of the present invention;

[0040] Fig.16 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] The embodiment of the present invention improves the defects of the prior art in face-changing processing of face images, which are prone to unnatural image distortion and poor generalization ability. The core idea is to pre-train and learn the position transformation information M of the face area pixel points in any face image to the predefined pixel point positions in the set face image through neural network (such as U-Net neural network) algorithm training, and by setting the pixel point positions in the face image, a mapping relationship for the face area pixel points is formed between the two face images to be replaced and the replaced face images, so as to realize the position transformation of the face area pixel points based on the mapping relationship, and then realize face-changing. This face-changing method based on face area pixel point mapping can make the key points in the face, especially the pixel points corresponding to the facial features, more accurate in the face-changing operation, maintain the similarity of the face before and after the face-changing, and make the facial expression more natural.

[0043] Specifically, after obtaining the first face image and the second face image, the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image can be determined respectively, such as respectively determining the position transformation information M1 and the position transformation information M2 corresponding to the transformation of the face area pixels in the first face image and the second face image to the pixel positions in the set face image; then, the position inverse transformation information M2 of the position transformation information M1 and the position transformation information M2 is sequentially executed on the face area pixels of the first face image. -1 The corresponding transformation processing forms a transformed face, and the transformed face is used to replace the face in the second facial image to form a transformed second facial image.

[0044] like Figure 1 FIG. 1 is a schematic diagram of the image processing logic of an embodiment of the present invention, wherein the processing logic includes a face image A to be replaced (Image A), a face image B to be replaced (Image B), a set face image (Standard Template), and a U-Net neural network model. Among them, the image sizes of Image A and Image B are the same. The purpose of the image processing in the figure is to replace the face part in Image B with the face part in Image A. The processing logic is as follows:

[0045] 1. Use the U-Net neural network model to calculate the position transformation information M1 and position transformation information M2 corresponding to the pixel points in the face area of ​​Image A and Image B (the pixel points within the box selection range, the box selection area has the same image size as the stand template) to the pixel point position in the stand template. Among them, M1 and M2 can be position transformation matrices.

[0046] Among them, the overall process of U-Net neural network model training is as follows:

[0047] The main function of the U-Net neural network model is to calculate the position mapping relationship between the pixels in the face area of ​​the input image (such as Image A, Image B) and the pixels in the predefined set face image (stand template). The most important thing in image face swapping is the position of the facial features. Therefore, in order to calculate the position of the facial features as accurately as possible, this solution sets a weight mask for the information of the key pixel positions of the facial features, and uses L2 Loss to allow the network to better regress the facial features information.

[0048] like Figure 2 As shown in , a face image is input into the U-Net neural network, as well as the offset information from the pixel points in the face area to the pixel points in the set face image after the actual annotation, which is recorded as Gound truth M (M is the position transformation information). The U-Net neural network calculates the offset information from each pixel point in the face area of ​​the face image to the pixel point position in the set face image, which is recorded as Predict M (M is the position transformation information), and then calculates the error between Predict M and Gound truth M. The error is fed back to the U-Net neural network through the back propagation algorithm to adjust the parameters of the network, so that Predict M is closer to Gound truth M.

[0049] The calculation formula (loss function) of the error between Predict M and Gound truth M is as follows:

[0050]

[0051] Among them, the loss function Loss mainly calculates the position of each pixel in the face area of ​​the input face image (Pos(u,v) is the offset prediction value, is the actual value of the offset), i.e., the Euclidean distance. The larger the distance, the greater the error. Since face swapping mainly focuses on the facial features, we hope that the error in the prediction of the facial features is as small as possible. Therefore, in this solution, the weight Weight with Mask is added to the calculation of the loss function value (the weight W(u,v) of the facial features in the Mask is larger, and the weights of other positions are smaller), so that the loss function pays more attention to the accuracy of the calculation of the facial features.

[0052] 2. According to Image A and its corresponding position transformation information M1, perform image transformation (Transform) on the pixels in the face area of ​​Image A to obtain the set face image corresponding to Image A.

[0053] 3. Perform an inverse transform on the set face image corresponding to Image A using the position transformation information M2 corresponding to Image B to obtain a transformed face.

[0054] For example, first invert the position transformation information M2 to obtain the position inverse transformation information (M2) -1 Then, Image A is set to the face image and (M2) is performed. -1 Transform to get the transformed face.

[0055] 4. Replace the face area in Image B with the transformed face (Fusion) to obtain a transformed Image B with the same posture and expression as the face in Image B.

[0056] Furthermore, after the transformed second facial image is formed, skin color fusion processing may be performed on the transformed second facial image according to the skin color in the first facial image and / or the second facial image to obtain a skin color fusion image with a more natural skin color.

[0057] For example, the skin color fusion processing of the transformed second face image can be completed based on the following three steps:

[0058] The first step is to construct the Laplacian pyramid of Image A and Image B.

[0059] like Figure 3 As shown, taking the above-mentioned Image B as an example, the process of establishing its corresponding Laplacian pyramid is as follows:

[0060] (1) Downsampling

[0061] Downsampling makes the image smaller as the level of the image pyramid increases. Assuming that each level is numbered from bottom to top, the size of level (i+1) is represented by G i+1 . Then the steps to obtain the i+1th layer image from the i-th layer image are:

[0062] Convolve Gi with a Gaussian kernel:

[0063]

[0064] b) Remove all even-numbered rows and columns.

[0065] After the above processing, the resulting image is 1 / 4 of the original image size. Continuously iterating the above steps can get Figure 3 The middle left image pyramid (also called Gaussian pyramid).

[0066] (2) Upsampling

[0067] Upsampling makes the image larger. Steps:

[0068] Will Figure 3 The bottommost image of the pyramid on the left side of the middle image is expanded to twice its original size in the row and column directions, and the newly added rows and columns are filled with 0;

[0069] Convolve the upscaled image with the same Gaussian kernel as before (multiplied by 4) to get an approximation of the added pixels.

[0070] (3) Laplace pyramid calculation

[0071] The information data lost by the downsampling operation forms a Laplacian pyramid. The i-th level of the Laplacian pyramid is defined as:

[0072]

[0073] Among them, UP is the upsampling operator, g 5×5 is the Gaussian kernel.

[0074] The second step is to fuse the Laplacian pyramids of Image A and Image B.

[0075] After the Laplacian pyramids of Image A and Image B are established, a multi-scale Gaussian pyramid mask is established for each layer, and then the weighted fusion of the Laplacian pyramids of Image A and Image B is achieved through masks of different scales.

[0076] like Figure 4 As shown in FIG. 1 , the mask may include two types of graphics: one is a mask that displays a face image (such as A mask map), and the other is a mask that displays a background image other than the face image (such as B maskmap). The two masks are used to extract the Laplacian image A Laplacian map of Image A and the Laplacian image B Laplacian map of Image B in the same layer (such as Level N), and then the extracted images are synthesized into a Laplacian image according to the pixel positions. Each layer is operated in this way to obtain a fused Laplacian pyramid.

[0077] The third step is to reconstruct the image.

[0078] like Figure 5 As shown, for Figure 1The transformed Image B finally obtained in the step is used to construct a Gaussian pyramid, and starting from the top layer, the transformed Image B is upsampled, and the upsampled image obtained at each layer is added to the image of the same layer in the fused Laplacian pyramid obtained in the second step to obtain the skin color fusion image corresponding to the layer. The skin color fusion image obtained at each layer is continuously enlarged (upsampled) from top to bottom and added to the fused Laplacian image of the same layer, so as to finally obtain the skin color fusion image corresponding to the transformed Image B.

[0079] Compared with the existing image face-changing technology, this solution has the following advantages:

[0080] (1) The deep neural network U-Net is used to learn the mapping relationship information between the pixels in the face area of ​​any face image and the pixel positions in the set face image, thereby establishing a mapping relationship between the pixel positions of the two face images to be replaced and the replaced face images. The pixels are dense, which solves the problem of unnatural image distortion during face replacement of face images with large angles and exaggerated expressions.

[0081] (2) The key points of face mapping in the U-Net neural network are more accurate, especially the position information of facial features, which maintains the similarity of the face before and after face swapping.

[0082] (3) The skin color fusion algorithm is used to solve the problem of unnatural skin color fusion in various scenes, further improving the generalization ability of the image face-changing technology shown in this scheme.

[0083] The technical solution of the present application is further illustrated below through multiple embodiments.

[0084] Embodiment 1

[0085] Figure 6 The image processing method provided by the embodiment of the present invention Figure 1 , which can be achieved Figure 1 The image processing logic shown in the figure, the image processing method comprises the following steps:

[0086] S610: Acquire a first face image and a second face image.

[0087] In order to facilitate uniform processing of face images, the acquired face images have the same size. Figure 1 Image A and Image B in the image.

[0088] S620: Determine mapping relationship information between pixel points in the face area of ​​the first face image and the second face image and positions of pixel points in the set face image respectively.

[0089] The most important thing in face-swapping is the location of the facial features. Therefore, in order to calculate the location of the facial features as accurately as possible and reduce the amount of data calculation, this solution selects the pixels of the face area (the area where the facial features are concentrated) in the face image as the object to be processed for face-swapping. For example, the face area can be Figure 1 The area enclosed by the boxes in Image A and Image B.

[0090] In this solution, a set face image is pre-set. To facilitate the mapping of face pixels in any face image to the pixel positions in the set face image, the set face image can be the same size as the face area in Image A and Image B, and the face posture can be frontal, the expression is natural, the face shape is common, and the face image quality is high.

[0091] The so-called mapping relationship of pixel positions refers to the correspondence between each pixel position of the face area in two face images and the real face position it points to. For example, there is a mapping relationship between the pixels (which may be multiple pixels) at the eye position in two face images. This mapping relationship can be described by the position offset between the pixels.

[0092] In this solution, the set of position offsets describing the mapping relationship between all pixels in the two images is called position transformation information, and the position transformation information will be opposite to each other according to the different mapping directions (pixel transformation directions) of the two images.

[0093] For example, when determining the mapping relationship information between the pixel points in the face area in the first face image and the second face image and the pixel point positions in the set face image, the position transformation information M1 and the position transformation information M2 corresponding to the transformation of the pixel points in the face area in the first face image and the second face image to the pixel point positions in the set face image can be determined respectively.

[0094] When the size of the face area and the set face image is a rectangle, the corresponding position transformation information can be described by a position transformation matrix, and each element value in the position transformation matrix is ​​defined as the position offset corresponding to the transformation of the corresponding pixel point in the face area to the pixel point position in the set face image.

[0095] In order to quickly and accurately calculate the position transformation information corresponding to the pixel point transformation of the face area in the face image to the pixel point position in the set face image, the U-Net neural network can be used for modeling calculation. Specifically, Figure 7 The method steps shown perform the modeling calculation process:

[0096] S710, transforming the pixel points in the face area of ​​the labeled sample image to the position transformation information corresponding to the pixel point position in the set face image, using a U-Net neural network for training and learning, and building a U-Net neural network model.

[0097] Among them, the main function of the U-Net neural network model is to calculate the position mapping relationship between the face area pixels of the input image (such as Image A, Image B) and the pixels in the set face image, such as the position transformation information of the former to the latter. The most important thing in image face swapping is the position of the facial features. Therefore, in order to make the calculation of the position of the facial features as accurate as possible, this solution sets a weight mask for the information of the key pixel positions of the facial features, and uses L2 Loss to allow the network to better regress the facial features information.

[0098] The specific model building process can be Figure 2 The processing flow shown is completed and will not be repeated here.

[0099] S720, using a U-Net neural network model to perform model calculation on the pixels in the face area of ​​the face image, and obtain position transformation information corresponding to the position of the pixels in the face area of ​​the face image to the pixel positions in the set face image.

[0100] After obtaining the U-Net neural network model, the position transformation information corresponding to the pixel points in the face area of ​​the first face image and the second face image and the pixel point positions in the set face image can be calculated based on the network model to obtain the mapping relationship information between the two.

[0101] S630, according to the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image, the face area pixels in the second face image are replaced with the face area pixels in the first face image to form a transformed second face image.

[0102] Specifically, after obtaining the mapping relationship information between the pixel points in the face area of ​​the first face image and the second face image and the pixel points in the set face image respectively, it is possible to replace the pixel points in the face area of ​​the second face image with the pixel points in the face area of ​​the first face image based on the mapping relationship information and using the pixel points in the set face image as a conversion bridge, thereby forming a transformed second face image.

[0103] For example, after determining that the face area pixels in the first face image and the second face image are transformed to the position transformation information M1 and the position transformation information M2 corresponding to the pixel positions in the set face image, correspondingly, by executing Figure 8 The method steps shown implement the processing process of this step:

[0104] S810, performing transformation processing corresponding to the position transformation information M1 on the pixel points in the face area of ​​the first face image to obtain a set face image corresponding to the first face image.

[0105] For example, according to Figure 1 Image A and its corresponding position transformation information M1 are used to perform image transformation (Transform) on the pixels in the face area of ​​Image A to obtain the set face image corresponding to Image A.

[0106] S820, determine the position inverse transformation information (M2) corresponding to the position transformation information M2 -1 .

[0107] Among them, the transformation information M2 and the corresponding position inverse transformation information (M2) -1 The position offsets of the corresponding pixels described in are opposite to each other to indicate different pixel transformation directions.

[0108] S830, performing position inverse transformation information (M2) on the pixel points in the set face image corresponding to the first face image -1 The corresponding transformation processing is performed to obtain the transformed face.

[0109] For example, set Image A to the face image (M2) -1 Transform to get the transformed face.

[0110] S840: Replace the face region in the second face image with the transformed face to form a transformed second face image.

[0111] For example, Figure 1 The transformed face in Image A replaces the face area in Image B (Fusion) to obtain the transformed Image B with the same posture and expression as the face in Image B.

[0112] The image processing method provided by the present invention pre-sets a set face image; obtains a first face image and a second face image, and respectively determines the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image; according to the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image, the face area pixels in the second face image are correspondingly replaced with the face area pixels in the first face image to form a transformed second face image, thereby achieving the purpose of replacing the face in the second face image with the face in the first face image. In this scheme, the pixel positions in the set face image are used as a conversion bridge to establish a mapping relationship between the face area pixels of the two face images to be replaced and replaced. Since the number of pixels is relatively dense, it can solve the problems of unnatural image distortion caused by large-angle postures and exaggerated expressions in image face swapping, thereby improving the generalization ability of image face swapping.

[0113] Furthermore, by constructing a U-Net neural network model and calculating the position transformation information M1 and position transformation information M2 corresponding to the transformation of the pixel points in the face area in the first face image and the second face image to the pixel point positions in the set face image based on the model, the mapping relationship information from the pixel points in the face area in the face image to the pixel point positions in the set face image can be obtained conveniently and quickly.

[0114] Furthermore, by sequentially performing the transformation processing corresponding to the position transformation information M1 and the position inverse transformation information (M2) on the pixel points in the face area of ​​the first face image -1 The corresponding transformation processing is performed to obtain a transformed face; the face area in the second face image is replaced by the transformed face, thereby forming a transformed second face image.

[0115] Embodiment 2

[0116] Fig. 9 The image processing method provided by the embodiment of the present invention Figure 4 , this method can be used as Figure 6 to Figure 8 The supplementary processing process of any one of the image processing methods is used to perform skin color fusion processing on the generated transformed second face image, that is, the transformed second face image is subjected to skin color fusion processing according to the image skin color in the first face image and / or the second face image to obtain a skin color fusion image corresponding to the transformed second face image. Fig. 9 As shown, the image processing method may specifically include the following steps:

[0117] S910 , forming Laplacian pyramids corresponding to the first face image and the second face image respectively.

[0118] like Figure 3As shown, taking the above-mentioned Image B as an example, the process of establishing its corresponding Laplacian pyramid is as follows:

[0119] (1) Downsampling

[0120] Downsampling makes the image smaller as the level of the image pyramid increases. Assuming that each level is numbered from bottom to top, the size of level (i+1) is represented by G i+1 . Then the steps to obtain the i+1th layer image from the i-th layer image are:

[0121] Convolve Gi with a Gaussian kernel:

[0122]

[0123] b) Remove all even-numbered rows and columns.

[0124] After the above processing, the resulting image is 1 / 4 of the original image size. Continuously iterating the above steps can get Figure 3 The middle left image pyramid (also called Gaussian pyramid).

[0125] (2) Upsampling

[0126] Upsampling makes the image larger. Steps:

[0127] Will Figure 3 The bottommost image of the pyramid on the left side of the middle image is expanded to twice its original size in the row and column directions, and the newly added rows and columns are filled with 0;

[0128] Convolve the upscaled image with the same Gaussian kernel as before (multiplied by 4) to get an approximation of the added pixels.

[0129] (3) Laplace pyramid calculation

[0130] The information data lost by the downsampling operation forms a Laplacian pyramid. The i-th level of the Laplacian pyramid is defined as:

[0131]

[0132] Among them, UP is the upsampling operator, g 5×5 is the Gaussian kernel.

[0133] S920, performing fusion processing on corresponding layers in the Laplacian pyramid corresponding to the first face image and the second face image to obtain a fused Laplacian pyramid.

[0134] For example, when performing fusion, the area corresponding to the face in the Laplacian image corresponding to the first face image in the corresponding layer and the area corresponding to the face other than the face in the Laplacian image corresponding to the second face image can be selected for fusion processing to obtain a fused Laplacian pyramid.

[0135] For example, after the Laplacian pyramids of Image A and Image B are established, a multi-scale Gaussian pyramid mask may be established for each layer, and then the weighted fusion of the Laplacian pyramids of Image A and Image B may be achieved through masks of different scales.

[0136] like Figure 4 As shown, the mask may include two types of graphics: one is a mask that displays a face image (such as A mask map), and the other is a mask that displays a background image other than the face image (such as B mask map). The two mask graphics are used to extract the Laplacian image A Laplacian map of Image A and the Laplacian image B Laplacian map of Image B in the same layer (such as Level N), and then the extracted images are synthesized into a Laplacian image according to the pixel positions. Each layer is operated in this way to obtain a fused Laplacian pyramid.

[0137] Of course, it is also possible to select only the Laplacian map A of Image A or the Laplacian map B of Image B to form the fused Laplacian pyramid.

[0138] S930, constructing a Gaussian pyramid corresponding to the transformed second facial image, and starting from the top layer of the Gaussian pyramid from top to bottom, using corresponding layers of the fused Laplacian pyramid to perform layer reconstruction to obtain a skin color fused image corresponding to the transformed second facial image.

[0139] For example, you can use Fig.10 The method steps shown are used to reconstruct the layers of the Gaussian pyramid:

[0140] S101, using the top layer image of the Gaussian pyramid as the skin color fusion image of the layer, upsampling the skin color fusion image of the top layer and performing Gaussian convolution on the image to obtain a predicted image of the next layer.

[0141] S102, adding the predicted image of the next layer to the corresponding layer in the fused Laplacian pyramid to obtain a skin color fused image corresponding to the next layer.

[0142] S103, and so on, until a skin color fusion image corresponding to the bottom layer of the Gaussian pyramid is formed as a skin color fusion image corresponding to the transformed second face image.

[0143] like Figure 5 As shown, for Figure 1 The transformed Image B finally obtained in the step is used to construct a Gaussian pyramid, and starting from the top layer, the transformed Image B is upsampled, and the result of the upsampled image (and Gaussian convolution) obtained at each layer is added to the image of the same layer in the fused Laplacian pyramid obtained in the second step to obtain the skin color fusion image corresponding to the layer. The skin color fusion image obtained at each layer is continuously enlarged (upsampled) from top to bottom and added to the fused Laplacian image of the same layer, and finally the skin color fusion image corresponding to the transformed Image B is obtained.

[0144] The image processing method provided by the present invention, based on any of the image processing methods shown in Example 1, further performs skin color fusion processing on the transformed second face image according to the image skin color in the first face image and / or the second face image, to obtain a skin color fusion image corresponding to the transformed second face image, so that the image after face replacement is more natural, thereby improving the generalization ability of image face replacement of this scheme.

[0145] Embodiment 3

[0146] like Fig.11 FIG. 1 is a diagram showing the structure of an image processing device according to an embodiment of the present invention. Figure 1 The image processing device can control the execution of Figure 6 The method steps shown in , which include:

[0147] A face image acquisition module 111, used to acquire a first face image and a second face image;

[0148] A mapping information determination module 112, used to respectively determine mapping relationship information between pixel points in the face area of ​​the first face image and the second face image and pixel points in the set face image;

[0149] The face transformation processing module 113 is used to replace the face area pixels in the second face image with the face area pixels in the first face image according to the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image, so as to form a transformed second face image.

[0150] Furthermore, the mapping information determination module 112 may be used to respectively determine position transformation information M1 and position transformation information M2 corresponding to the transformation of the face area pixels in the first face image and the second face image to the pixel positions in the set face image.

[0151] Furthermore, if Fig.12 As shown, Fig.11The face transformation processing module 113 in the image processing device shown in FIG. 1 may include:

[0152] An image transformation unit 121 is used to perform a transformation process corresponding to the position transformation information M1 on the pixel points in the face area of ​​the first face image to obtain a set face image corresponding to the first face image;

[0153] The inverse transformation determination unit 122 is used to determine the position inverse transformation information (M2) corresponding to the position transformation information M2. -1 ;

[0154] The image inverse transformation unit 123 is used to perform position inverse transformation information (M2) on the pixel points in the set face image corresponding to the first face image. -1 Corresponding transformation processing is performed to obtain a transformed face;

[0155] The image replacement unit 124 is used to replace the face area in the second face image with the transformed face to form a transformed second face image.

[0156] Fig.12 The module structure shown can be used to implement Figure 8 The method steps are shown.

[0157] Furthermore, if Fig.13 As shown, Fig.11 The image processing device shown in may also include:

[0158] The network model training module 131 is used to transform the pixel points in the face area of ​​the labeled sample image to the position transformation information corresponding to the pixel point position in the set face image, use the U-Net neural network for training and learning, and build a U-Net neural network model;

[0159] The mapping information determination module 112 can be used to use the U-Net neural network model to perform model calculation on the pixel points in the face area of ​​the face image to obtain the position transformation information corresponding to the pixel points in the face area of ​​the face image to the pixel points in the set face image.

[0160] Fig.13 The image processing apparatus shown can be used to perform Figure 7 The method steps are shown.

[0161] Furthermore, if Fig.14 As shown, Figure 11 to Figure 13 Any of the image processing devices shown may also include:

[0162] The skin color fusion processing module 141 is used to perform skin color fusion processing on the transformed second face image according to the image skin color in the first face image and / or the second face image to obtain a skin color fusion image corresponding to the transformed second face image.

[0163] Furthermore, if Fig.14 The image processing device may further include:

[0164] A pyramid construction module 142, used to form Laplacian pyramids corresponding to the first face image and the second face image respectively;

[0165] A pyramid fusion module 143 is used to fuse the corresponding layers in the Laplacian pyramid corresponding to the first face image and the second face image to obtain a fused Laplacian pyramid;

[0166] The skin color fusion processing module 141 can be used to construct a Gaussian pyramid corresponding to the transformed second face image, and start from the top of the Gaussian pyramid from top to bottom, and use the corresponding layers of the fused Laplacian pyramid to perform layer reconstruction to obtain a skin color fusion image corresponding to the transformed second face image.

[0167] Fig.14 The image processing apparatus shown can be used to perform Fig. 9 The method steps are shown.

[0168] Furthermore, if Fig.15 As shown, Fig.14 The skin color fusion processing module 141 in the image processing device shown in FIG. 1 may include:

[0169] An upsampling processing unit 151 is used to use the top layer image of the Gaussian pyramid as the skin color fusion image of the layer, upsample the skin color fusion image of the top layer and perform Gaussian convolution to obtain a predicted image of the next layer;

[0170] An image fusion unit 152 is used to add the predicted image of the next layer to the corresponding layer in the fused Laplacian pyramid to obtain a skin color fused image corresponding to the next layer;

[0171] This is deduced in this way until a skin color fusion image corresponding to the bottom layer of the Gaussian pyramid is formed as a skin color fusion image corresponding to the transformed second face image.

[0172] Fig.15 The module structure shown can be used to implement Fig.10 The method steps are shown.

[0173] Furthermore, the pyramid fusion module 143 may be configured to select an area corresponding to the face in the Laplacian image corresponding to the first face image in the corresponding layer and an area corresponding to other than the face in the Laplacian image corresponding to the second face image for fusion processing to obtain the fused Laplacian pyramid.

[0174] The image processing device provided by the present invention pre-sets a set face image; obtains a first face image and a second face image, and respectively determines the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image; according to the mapping relationship information between the face area pixels in the first face image and the second face image and the pixel positions in the set face image, the face area pixels in the second face image are correspondingly replaced with the face area pixels in the first face image to form a transformed second face image, thereby achieving the purpose of replacing the face in the second face image with the face in the first face image. In this scheme, the pixel positions in the set face image are used as a conversion bridge, so that a mapping relationship is established between the face area pixels of the two face images that are replaced and replaced. Since the number of pixels is relatively dense, it can solve the problems of unnatural image distortion caused by large-angle postures and exaggerated expressions in image face swapping, and improve the generalization ability of image face swapping.

[0175] Furthermore, by constructing a U-Net neural network model and calculating the position transformation information M1 and position transformation information M2 corresponding to the transformation of the pixel points in the face area in the first face image and the second face image to the pixel point positions in the set face image based on the model, the mapping relationship information from the pixel points in the face area in the face image to the pixel point positions in the set face image can be obtained conveniently and quickly.

[0176] Furthermore, by sequentially performing the transformation processing corresponding to the position transformation information M1 and the position inverse transformation information (M2) on the pixel points in the face area of ​​the first face image -1 The corresponding transformation processing is performed to obtain a transformed face; the face area in the second face image is replaced by the transformed face, thereby forming a transformed second face image.

[0177] Furthermore, skin color fusion processing is performed on the transformed second facial image according to the skin color in the first facial image and / or the second facial image to obtain a skin color fusion image corresponding to the transformed second facial image, so that the image after face replacement is more natural, thereby improving the generalization ability of image face replacement of this scheme.

[0178] Embodiment 4

[0179] The embodiment of the present invention provides an image processing method, which can be used as an application of the image processing method in the above embodiment in actual business processing. The image processing method may specifically include the following steps:

[0180] Collect the face image to be replaced and determine the face image to be replaced;

[0181] The face image to be replaced is used as the target face image after replacement, which can be Figure 1Image A; the replaced face image is the target face image to be replaced, which can be Figure 1 Image B.

[0182] In actual business scenarios, when a user wants to replace a replaced facial image (such as a star or actor) with his or her own facial image, he or she can trigger the business system to collect his or her own facial image as a replacement facial image, and at the same time specify the replaced facial image to the business system.

[0183] Determine mapping relationship information between pixel points in the face area of ​​the face image to be replaced and the face image to be replaced and pixel points in the set face image respectively;

[0184] According to the mapping relationship information between the pixel points of the face area in the face image to be replaced and the replaced face image and the pixel points in the set face image, the pixel points of the face area in the replaced face image are replaced with the pixel points of the face area in the face image to be replaced, so as to form a transformed replaced face image.

[0185] Specifically, after the business system collects the user's facial image and determines the facial image to be replaced, it can use the image processing method in the aforementioned embodiment to replace the pixels in the facial area of ​​the facial image to be replaced with the pixels in the facial area of ​​the user's facial image to form a transformed facial image to be replaced, thereby satisfying the user's entertainment needs of changing their face to look like a star or actor.

[0186] Embodiment 5

[0187] The above embodiments describe the overall architecture of the image processing device. The functions of the device can be implemented by means of an electronic device, such as Fig.16 As shown, it is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which specifically includes: a memory 161 and a processor 162.

[0188] The memory 161 is used to store programs.

[0189] In addition to the above-mentioned programs, the memory 161 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0190] Memory 161 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0191] The processor 162 is coupled to the memory 161 and is used to execute the program in the memory 161. When the program is executed, Figure 6 to Figure 10 The image processing method shown.

[0192] The above-mentioned specific processing operations have been described in detail in the previous embodiments and will not be repeated here.

[0193] Further, if Fig.16 As shown, the electronic device may also include: a communication component 163, a power component 164, an audio component 165, a display 166 and other components. Fig.16 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.16 Components shown.

[0194] The communication component 163 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 163 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 163 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0195] The power supply component 164 provides power to various components of the electronic device. The power supply component 164 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0196] The audio component 165 is configured to output and / or input audio signals. For example, the audio component 165 includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 161 or sent via the communication component 163. In some embodiments, the audio component 165 also includes a speaker for outputting audio signals.

[0197] The display 166 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0198] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, comprising: Acquire a first face image and a second face image; Determine mapping relationship information between pixel points in the face area of ​​the first face image and the second face image and pixel points in the set face image respectively; According to the mapping relationship information between the face area pixels in the first face image and the second face image and the positions of the pixels in the set face image, the face area pixels in the second face image are replaced with the face area pixels in the first face image to form a transformed second face image; The mapping relationship information of respectively determining the pixel points of the face area in the first face image and the second face image and the pixel point positions in the set face image includes: Position transformation information M1 and position transformation information M2 corresponding to the transformation of the pixel points in the face area of ​​the first face image and the second face image to the pixel points in the set face image are determined respectively.

2. The method according to claim 1, wherein: The step of replacing the face area pixels in the second face image with the face area pixels in the first face image according to the mapping relationship information between the face area pixels in the first face image and the second face image and the positions of the pixels in the set face image to form the transformed second face image comprises: Performing transformation processing corresponding to the position transformation information M1 on the pixel points in the face area of ​​the first face image to obtain a set face image corresponding to the first face image; Determine the position inverse transformation information (M2) corresponding to the position transformation information M2 -1 ; Executing the position inverse transformation information (M2) on the pixel points in the set face image corresponding to the first face image -1 Corresponding transformation processing is performed to obtain a transformed face; The face region in the second face image is replaced with the transformed face to form a transformed second face image.

3. The method according to claim 1, wherein: The method further comprises: The position transformation information of the pixel points in the face area of ​​the labeled sample image to the position of the pixel points in the set face image is transformed, and a U-Net neural network is used for training and learning to build a U-Net neural network model; The position transformation information corresponding to the transformation of the pixel point in the face area of ​​the face image to the pixel point position in the set face image includes: The U-Net neural network model is used to perform model calculation on the pixel points in the face area of ​​the face image to obtain position transformation information corresponding to the pixel points in the face area of ​​the face image to the pixel points in the set face image.

4. The method according to any one of claims 1 to 3, wherein: The method comprises: The transformed second facial image is subjected to skin color fusion processing according to the image skin color in the first facial image and / or the second facial image to obtain a skin color fusion image corresponding to the transformed second facial image.

5. The method according to claim 4, wherein: The method further comprises: Forming Laplacian pyramids corresponding to the first face image and the second face image respectively; Performing fusion processing on corresponding layers in the Laplacian pyramid corresponding to the first face image and the second face image to obtain a fused Laplacian pyramid; The step of performing skin color fusion processing on the transformed second face image according to the skin colors of the first face image and the second face image to obtain a skin color fusion image corresponding to the transformed second face image comprises: A Gaussian pyramid corresponding to the transformed second face image is constructed, and layer reconstruction is performed from the top layer of the Gaussian pyramid downward using corresponding layers of the fused Laplacian pyramid to obtain a skin color fused image corresponding to the transformed second face image.

6. The method according to claim 5, wherein: The step of starting from the top layer of the Gaussian pyramid and reconstructing the layers using the corresponding layers of the fused Laplacian pyramid to obtain the skin color fused image corresponding to the transformed second face image comprises: The top layer image of the Gaussian pyramid is used as the skin color fusion image of the layer, and the skin color fusion image of the top layer is upsampled and Gaussian convolved to obtain a predicted image of the next layer; Adding the predicted image of the next layer to the corresponding layer in the fused Laplacian pyramid to obtain a skin color fused image corresponding to the next layer; And so on, until the skin color fusion image corresponding to the bottom layer of the Gaussian pyramid is formed as the skin color fusion image corresponding to the transformed second face image.

7. The method according to claim 5, wherein: The fusing corresponding layers in the Laplacian pyramid corresponding to the first face image and the second face image to obtain a fused Laplacian pyramid includes: An area corresponding to the face in the Laplacian image corresponding to the first face image in the corresponding layer and an area corresponding to other than the face in the Laplacian image corresponding to the second face image are selected for fusion processing to obtain the fused Laplacian pyramid.

8. An image processing device, comprising: A face image acquisition module, used to acquire a first face image and a second face image; A mapping information determination module, used to respectively determine mapping relationship information between pixel points in the face area of ​​the first face image and the second face image and pixel points in the set face image; a face transformation processing module, configured to replace the face area pixels in the second face image with the face area pixels in the first face image according to the mapping relationship information between the face area pixels in the first face image and the second face image and the positions of the pixels in the set face image, so as to form a transformed second face image; The mapping information determination module is used to respectively determine position transformation information M1 and position transformation information M2 corresponding to the transformation of face area pixels in the first face image and the second face image to pixel positions in the set face image.

9. The device according to claim 8, wherein: The face transformation processing module comprises: An image transformation unit, configured to perform a transformation process corresponding to the position transformation information M1 on the pixels of the face area of ​​the first face image, so as to obtain a set face image corresponding to the first face image; An inverse transformation determination unit, configured to determine the position inverse transformation information (M2) corresponding to the position transformation information M2 -1 ; An image inverse transformation unit, configured to perform the position inverse transformation information (M2) on pixel points in a set face image corresponding to the first face image. -1 Corresponding transformation processing is performed to obtain a transformed face; The image replacement unit is used to replace the face area in the second face image with the transformed face to form a transformed second face image.

10. The device according to claim 8, wherein: The device also includes: A network model training module is used to transform the pixel points in the face area of ​​the labeled sample image to the position transformation information corresponding to the pixel point position in the set face image, and use the U-Net neural network for training and learning to build a U-Net neural network model; The mapping information determination module is used to use the U-Net neural network model to perform model calculation on the pixel points in the face area of ​​the face image to obtain the position transformation information corresponding to the pixel point position in the face area of ​​the face image to the set face image.

11. The device according to any one of claims 8 to 10, wherein: The device also includes: The skin color fusion processing module is used to perform skin color fusion processing on the transformed second face image according to the image skin color in the first face image and / or the second face image to obtain a skin color fusion image corresponding to the transformed second face image.

12. The device according to claim 11, wherein: The device also includes: A pyramid construction module, used to form Laplacian pyramids corresponding to the first face image and the second face image respectively; A pyramid fusion module, used for fusing corresponding layers in the Laplacian pyramid corresponding to the first face image and the second face image to obtain a fused Laplacian pyramid; The skin color fusion processing module is used to construct a Gaussian pyramid corresponding to the transformed second face image, and start from the top layer of the Gaussian pyramid from top to bottom, using the corresponding layers of the fused Laplacian pyramid to perform layer reconstruction to obtain a skin color fusion image corresponding to the transformed second face image.

13. The device according to claim 12, wherein: The skin color fusion processing module comprises: An upsampling processing unit, used to use the top layer image of the Gaussian pyramid as the skin color fusion image of the layer, upsample the skin color fusion image of the top layer and perform Gaussian convolution to obtain a predicted image of the next layer; An image fusion unit, used for adding the predicted image of the next layer to the corresponding layer in the fused Laplacian pyramid to obtain a skin color fused image corresponding to the next layer; And so on, until the skin color fusion image corresponding to the bottom layer of the Gaussian pyramid is formed as the skin color fusion image corresponding to the transformed second face image.

14. The device according to claim 12, wherein: The pyramid fusion module is used to An area corresponding to the face in the Laplacian image corresponding to the first face image in the corresponding layer and an area corresponding to other than the face in the Laplacian image corresponding to the second face image are selected for fusion processing to obtain the fused Laplacian pyramid.

15. An image processing method, comprising: Collect the face image to be replaced and determine the face image to be replaced; Determining respectively the mapping relationship information between the face area pixels in the face image to be replaced and the replaced face image and the pixel positions in the set face image, including: respectively calculating the position transformation information M1 and the position transformation information M2 corresponding to the face area pixels in the replacement face image and the replaced face image to the pixel positions in the set face image; According to the mapping relationship information between the face image to be replaced and the face area pixels in the replaced face image and the pixel positions in the set face image, the face area pixels in the replaced face image are correspondingly replaced with the face area pixels in the face image to be replaced to form a transformed replaced face image.

16. An electronic device comprising: Memory, used to store programs; A processor, coupled to the memory, is used to execute the program, and when the program is run, the image processing method described in any one of claims 1-7 and 15 is executed.

Images