Image processing method and device, electronic device, and storage medium

Abstract

The embodiments of the present application disclose an image processing method and apparatus, an electronic device, and a storage medium. The method includes: aligning a facial region of a first facial image with a facial region of a second facial image to obtain a target facial image after the alignment of the first facial image; extracting features of the target facial image and the second facial image at multiple scales to obtain corresponding first multi-scale feature maps and second multi-scale feature maps; obtaining a first mask of the target facial image and a second mask of the second facial image; and performing facial fusion processing on the target facial image and the second facial image based on a first residual link established between the first mask and the first multi-scale feature map, and a second residual link established between the second mask and the second multi-scale feature map to obtain a fused facial image. The technology of the embodiments of the present application can adaptively fuse any two facial images, such as fusing any two frames of facial images in a multimedia video.

Description

Technical Field

[0001] The present application relates to the technical fields of image processing and computer vision, and in particular to an image processing method and apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] Face fusion / face swapping is a key topic in image processing and computer vision. Existing face fusion techniques can be divided into two main categories: methods that use face alignment and Bosch fusion, and neural network-based approaches, such as deepfake (an AI face-swapping tool).

[0003] In the first type of method, the face areas of the original image and the target image are first found through face detection or facial key points, and then stretching and alignment are performed. Then, methods such as Poisson fusion are used to fit the faces. The advantage of this method is that it is easy to use. For any original image and target image, as long as the face area can be detected, face swapping and fusion can be performed. However, it also has obvious disadvantages. Poisson fusion is slow and the effect cannot be guaranteed. Obvious boundaries and artifacts will appear on images with large color differences.

[0004] In recent years, convolutional neural networks (CNNs) have gradually become the mainstream method in the fields of image processing and computer vision. Therefore, in another type of method, Deepfake and similar schemes are used to train an end-to-end neural network on a large number of photos of two people to complete face replacement. Compared with the first method, this method has a more natural fusion boundary and can automatically correct the angle and orientation. However, this method is relatively cumbersome to use. To swap any two faces, a large number of pictures of both faces must be collected and a separate model must be trained, which is a cumbersome process. Summary of the Invention

[0005] To solve the above technical problems, the embodiments of the present application provide an image processing method and device, an electronic device, and a computer-readable storage medium, which can quickly realize face fusion and replacement of facial images.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0007] According to one aspect of an embodiment of the present application, an image processing method is provided, including: aligning a facial region of a first facial image with a facial region of a second facial image to obtain a target facial image after the first facial image is aligned; extracting features of the target facial image and the second facial image at multiple scales to obtain corresponding first multi-scale feature maps and second multi-scale feature maps; obtaining a first mask of the target facial image and a second mask of the second facial image; and performing facial fusion processing on the target facial image and the second facial image based on a first residual link established between the first mask and the first multi-scale feature map, and a second residual link established between the second mask and the second multi-scale feature map to obtain a fused facial image.

[0008] According to one aspect of an embodiment of the present application, an image processing device is provided, comprising: a face alignment module configured to align a face region of a first face image with a face region of a second face image to obtain a target face image after the first face image is aligned; a multi-scale feature map acquisition module configured to extract features of the target face image and the second face image at multiple scales to obtain corresponding first and second multi-scale feature maps; a mask acquisition module configured to obtain a first mask of the target face image and a second mask of the second face image; and a face fusion module configured to perform face fusion processing on the target face image and the second face image based on a first residual link established between the first mask and the first multi-scale feature map, and a second residual link established between the second mask and the second multi-scale feature map, to obtain a fused face image. According to one aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the image processing method described above is implemented.

[0009] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the image processing method described above.

[0010] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional embodiments described above.

[0011] In the technical solution provided in the embodiments of this application, two aligned facial images are directly input, a residual link between the multi-scale feature maps and masks in the aligned facial images is established, and face fusion processing is performed on the residually linked multi-scale feature maps and masks to generate the fused facial image. This face fusion method is fast, does not require a large amount of data training, and can adaptively fuse any two facial images.

[0012] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0014] Figure 1 A schematic diagram of an implementation environment involved in this application;

[0015] Figure 2 is a flowchart of an image processing method shown in an exemplary embodiment of the present application;

[0016] Figure 3 1 is a schematic diagram showing the principle of face fusion using a multi-scale encoder-decoder with residual links, as shown in an exemplary embodiment of the present application;

[0017] Figure 4 is a flowchart of an image processing method shown in another exemplary embodiment of the present application;

[0018] Figure 5 This application Figure 2 A flow chart of step S210 in the illustrated embodiment in an exemplary embodiment;

[0019] Figure 6 This application Figure 5 A flow chart of step S510 in the illustrated embodiment in an exemplary embodiment;

[0020] Figure 7 is a schematic structural diagram illustrating an exemplary embodiment of the present application and a target detection algorithm;

[0021] Figure 8 1 is a schematic diagram of the structure of a facial key point detection network shown in an exemplary embodiment of the present application;

[0022] Figure 9 This application Figure 6 A flow chart of step S630 in the illustrated embodiment in an exemplary embodiment;

[0023] Figure 10 This application Figure 5 A flow chart of step S530 in the illustrated embodiment in an exemplary embodiment;

[0024] Figure 11 is a structural diagram of an image processing device shown in an exemplary embodiment of the present application;

[0025] Figure 12 It is a structural diagram of a computer system suitable for implementing an electronic device, shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0029] It should also be noted that the term "plurality" used in this application refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0030] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0031] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0032] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0033] The image processing methods and devices, electronic devices, and storage media proposed in the embodiments of this application involve artificial intelligence technology and machine learning technology, and these embodiments will be described in detail below.

[0034] First see Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application, which includes a user terminal 100 and a server 200, which communicate with each other via a wired or wireless network.

[0035] The user terminal 100 is used to collect facial images to be subjected to facial fusion processing, and at the same time input the collected facial images into the server side 200. The server side 200 processes the facial images, performs facial fusion on the received facial images, and then sends the effect image after facial fusion to the user terminal 100. The effect image after facial fusion can be visually displayed through the display module of the user terminal 100.

[0036] Exemplarily, after receiving a set of first face images and second face images to be subjected to face fusion processing, the user terminal 100 sends the first face image and the second face image to the server side 200. The server side 200 aligns the face area of ​​the first face image with the face area of ​​the second face image to obtain a target face image after the first face image is aligned, and then extracts the features of the target face image and the second face image at multiple scales respectively to obtain the corresponding first multi-scale feature map and second multi-scale feature map, and then obtains the first mask of the target face image and the second mask of the second face image. According to the first residual link established between the first mask and the first multi-scale feature map, and the second residual link established between the second mask and the second multi-scale feature map, face fusion processing is performed on the target face image and the second face image to obtain a fused face image.

[0037] The user terminal 100 includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, and in-vehicle terminals. For example, it can be any electronic device capable of image visualization, such as a smartphone, tablet, laptop, or computer, and is not limited here. The server end 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, wherein multiple servers can form a blockchain, and the server is a node on the blockchain. The server end 200 can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, and is not limited here.

[0038] Figure 2 FIG. 1 is a flow chart showing an image processing method according to an exemplary embodiment. Figure 2 As shown, in an exemplary embodiment, the method may include steps S210 to S270, which are described in detail as follows:

[0039] Step S210: aligning the facial region of the first facial image with the facial region of the second facial image to obtain a target facial image after the alignment of the first facial image.

[0040] In this embodiment, a group of face images that need to be fused are obtained: a first face image and a second face image. First, the face directions in the two face images are aligned. For example, if the face direction in the first face image needs to be aligned with the face direction in the second face image, a face alignment method is used to align the face in the first face image with the face in the second face image, and a target face image after the first face image is aligned is obtained.

[0041] Of course, in other embodiments, a face alignment method may be used to align the face in the second face image with the face in the first face image to obtain a target face image after the second face image is aligned.

[0042] Step S230: extracting features of the target face image and the second face image at multiple scales respectively, and obtaining corresponding first multi-scale feature maps and second multi-scale feature maps.

[0043] In this embodiment, the face image is processed by an encoder to extract feature maps of different scales of the face image to obtain a multi-scale feature map.

[0044] In a specific embodiment, the target face image and the second face image are processed by two encoders with the same structure to extract a first multi-scale feature map of the target face image and a second multi-scale feature map of the second face image, respectively.

[0045] Exemplarily, before this step, a mask of the target face image can be obtained, and then color correction is performed on the face area of ​​the target face image based on the mask of the target face image to provide a face image that is easy to process for the subsequent face fusion process.

[0046] Step S250: Acquire a first mask of the target face image and a second mask of the second face image.

[0047] In this embodiment, masks of different sizes are obtained for the target facial image to form a first mask for the target image, and masks of different sizes are obtained for the second facial image to form a second mask. The first mask and the second mask of different sizes can be obtained by scaling the masks.

[0048] Step S270: performing face fusion processing on the target face image and the second face image based on the first residual link established between the first mask and the first multi-scale feature map, and the second residual link established between the second mask and the second multi-scale feature map, to obtain a fused face image.

[0049] In this embodiment, the first mask and the first multi-scale feature map, as well as the second mask and the second multi-scale feature map, are respectively input into the decoder to perform face fusion of the corresponding target face image and the second face image. Since a residual link is established between the decoder and the encoder, the residual link corresponding to the first mask and the first multi-scale feature map is referred to as the first residual link, and the link corresponding to the second mask and the second multi-scale feature map is referred to as the second residual link. It should be noted that in this embodiment, the multi-scale feature map is obtained through the encoder, and the multi-scale feature map and the mask are jointly input into the decoder connected to the encoder residual link for face fusion. The purpose is to use the residual link between the encoder and the decoder to help produce a result with richer details in the face fusion process and improve the face fusion effect.

[0050] In a specific embodiment, after obtaining the first multi-scale feature map and the second multi-scale feature map, multiple first multi-scale feature maps of different sizes are multiplied by the first mask scaled to the corresponding size and used as the input of the decoder. At the same time, multiple second multi-scale feature maps of different sizes are multiplied by the second mask scaled to the corresponding size and used as the input of the decoder. Of course, here the multi-scale feature map and the mask of the same size are multiplied as the input of the decoder, but in other embodiments, other methods such as adding the multi-scale feature map and the mask of the same size, or establishing a linear relationship between the multi-scale feature map and the mask of the same size can also be used as the input of the decoder, and no specific limitation is made here.

[0051] In this embodiment, face fusion of the target face image and the second face image is completed by processing a multi-scale encoder-decoder with residual links. Before the multi-scale encoder-decoder with residual links is applied to the face fusion of two aligned face images, the multi-scale encoder-decoder with residual links can be trained first. For example, a large number of aligned face images and corresponding masks are obtained using the above-mentioned method. Then, Poisson fusion or other image fusion methods are used to synthesize face data. After manual screening, the data with poor synthesis effect are removed, and the data with good synthesis effect are retained. Finally, a training framework based on generative adversarial networks (GAN) is used to train the multi-scale encoder-decoder with residual links, and finally a multi-scale encoder-decoder with residual links that can be applied to the face fusion of two aligned face images is obtained.

[0052] When using a training framework based on a generative adversarial network to train the multi-scale encoder-decoder with residual links, the loss function of the multi-scale encoder-decoder with residual links is a weighted sum of L1 loss, adversarial loss, and multi-scale perceptual loss. The specific loss functions are as follows:

[0053] L1Loss=∑|yG(x)|

[0054] L adv =E (x，y) [log D(x,y)]+E (y) [log(1-D(x, G(x)))]

[0055] L perceptual =∑|VGG(y)-VGG(G(x))|

[0056] Among them, L1Loss is the pixel-by-pixel L1 loss between the network output and the true value label, y is the label, G is the generator, x is the network input, L adv represents adversarial loss, D is the discriminator, E is the mathematical expectation, which means x and y obey a certain distribution, VGG is the pre-trained VGG (convolutional neural network) feature extraction network, L perceptual Represents the perceptual loss between the network output and the label, which is calculated as the sum of the L1 losses of the extracted features at each layer.

[0057] refer to Figure 3 , is a schematic diagram of a face fusion using the multi-scale encoder-decoder with residual link in a specific embodiment. In the figure, the encoder is used to obtain multi-scale feature maps of two face images respectively. Taking the target face image as an example, the first multi-scale feature map of the target face image is obtained by the encoder ( Figure 3 , the solid rectangular box is regarded as a feature map of different scales), and then the first multi-scale feature map is respectively connected with the first mask of the same size to establish a first residual link ( Figure 3 The "X" shaped image in the middle is used to establish a residual link), and the same process is performed on the second face image. Then, the first residual link established by the first mask and the first multi-scale feature map, and the second residual link established by the second mask and the second multi-scale feature map are input into the decoder for face fusion ( Figure 3 In the figure, “+” represents the first residual link and the second residual link of different sizes input to the decoder, and the hollow rectangle represents the output of different sizes obtained by the decoder), and finally the fused face image is obtained.

[0058] In this embodiment, a multi-scale encoder-decoder with residual links can directly process two input face images and output a fused result. Compared with existing fusion methods, it is faster and can effectively avoid artifacts and contour distortion problems. When swapping any two faces, there is no need to collect a large number of images of the two faces to train a separate model. The process is simple and can realize a pipeline (R&D pipeline / product line) based on face fusion. It can process any two faces without retraining the model, and the effect is better than traditional methods.

[0059] Figure 4 is a flow chart showing an image processing method according to another exemplary embodiment, which may be located in Figure 2 Before step S230 proposed in the embodiment shown, Figure 4 As shown, the method may include steps S410 to S430, which are described in detail as follows:

[0060] Step S410: adjusting the angle of the face in the target face image.

[0061] In this embodiment, for Figure 2 After the alignment processing of step S210 in the shown embodiment, there is a problem of excessive difference in the facial orientation in the target facial image, and the facial angle in the target facial image is adjusted. Specifically, the generator can be used to extract the features of the target facial image, and then convolution is performed to obtain the target facial image after the facial angle is adjusted.

[0062] Step S430: If the difference between the adjusted angle and the angle of the face in the second facial image is not within the preset error threshold range, continue to adjust the angle of the face in the target facial image until the difference between the adjusted angle and the angle of the face in the second facial image is within the preset error threshold range.

[0063] In this embodiment, it is determined whether the face orientation of the target face image after adjusting the face angle is within an error range with the face orientation in the second face image. If not, the angle of the target face image is continued to be adjusted by the generator until the difference between the adjusted angle and the angle of the face in the second face image is within a preset error threshold.

[0064] The specific judgment method is to use the discriminator to judge the authenticity of the target face image and the second face image after adjusting the face angle, and the generator will not stop working until the authenticity of the discriminator output is within the allowable error range.

[0065] In this embodiment, the face angle of the target face image is adjusted based on the generator and the discriminator by using the generative adversarial network latent space interpolation method to solve the problem of poor fusion effect caused by the large difference in face orientation of the target face image. Figure 2 Steps S230 to S270 provided in the embodiment are used to perform face fusion, thereby improving the effect of face fusion.

[0066] Figure 5 yes Figure 2 The flowchart of step S210 in the embodiment shown is in an exemplary embodiment. Figure 5As shown, in an exemplary embodiment, the process of aligning the facial region of the first facial image with the facial region of the second facial image to obtain the target facial image after the first facial image is aligned may include steps S510 to S530, which are described in detail as follows:

[0067] Step S510: Obtain the coordinates of facial key points of the facial image.

[0068] In this embodiment, two facial images to be processed for facial fusion are first obtained, namely the first facial image and the second facial image, and then the facial key point coordinates of the two facial images are obtained, including the first facial key point coordinates of the first facial image and the second facial key point coordinates of the second facial image.

[0069] Step S530: Aligning the facial region of the first facial image with the facial region of the second facial image according to the coordinates of the first facial key points and the coordinates of the second facial key points to obtain a target facial image after the alignment of the first facial image.

[0070] In this embodiment, face alignment is performed based on the coordinates of the first facial key points and the coordinates of the second facial key points. If the face direction in the first facial image needs to be aligned with the face direction in the second facial image, the coordinates of the first facial key points are adjusted based on the coordinates of the second facial key points, so that the face in the first facial image is aligned with the face in the second facial image, and the target facial image after the first facial image is aligned is obtained.

[0071] Of course, in other embodiments, the coordinates of the second facial key points may be adjusted so that the face in the second facial image is aligned with the face in the first facial image to obtain a target facial image after the second facial image is aligned.

[0072] In this embodiment, after obtaining the target face image that has undergone face alignment processing, face fusion processing is performed on the target face and the face image that has not undergone alignment processing to obtain a fused face image.

[0073] For example, in one embodiment, if the target facial image is obtained after alignment processing of the first facial image, then the target facial image and the second facial image are subjected to facial fusion processing to obtain a fused facial image; if the target facial image is obtained after alignment processing of the second facial image, then the target facial image and the first facial image are subjected to facial fusion processing to obtain a fused facial image.

[0074] Preferably, before performing face fusion processing on the target face image and the unaligned face image to obtain the fused face image, it includes: obtaining the mask of the target face image, and then performing color correction on the face area of ​​the target face image according to the mask of the target face image, so as to provide a face image that is easy to process for subsequent face fusion processing.

[0075] In this embodiment, by obtaining the coordinates of the facial key points of the two images to be fused, the faces are aligned based on the facial key points of the two images to improve the accuracy of face alignment, and finally the aligned images are fused to obtain a fused face image. This method is simple and can improve the speed and effect of face fusion.

[0076] Figure 6 yes Figure 5 The flowchart of step S510 in the embodiment shown is in an exemplary embodiment. Figure 6 As shown, in an exemplary embodiment, the process of obtaining the coordinates of key facial points of a facial image may include steps S610 to S650, which are described in detail as follows:

[0077] Step S610: Obtain the position coordinates of the face frame in the face image.

[0078] In this embodiment, the face in the face image is first detected, specifically by inputting an RGB (three primary colors) color image, then cutting out a face frame, and outputting the position coordinates of the face frame.

[0079] The face frame covers the face area in the face image, and the output face frame position coordinates are (x, y, w, h), where x and y are the coordinates of the upper left corner of the face frame in the rectangular coordinate system, and w and h are the width and height of the face frame. The face frame can be rectangular or square. For example, when the face frame is square, w = h.

[0080] In a specific embodiment, face frame recognition can be performed using a target detection algorithm (Retinanet). The structure diagram of the target detection algorithm can be found in Figure 7Specifically, the target detection algorithm includes a residual network (ResNet) that extracts the backbone features of the face image, a multi-scale target detection network (featurepyramid net) that fuses the backbone features to obtain multiple-scale features, and a target box regression and classification regression subnet (class+box subnets) that outputs the key point positions; among them, the target box regression subnet (class subnet) uses 4 256-channel convolutions and 1 num_priors x num_classes convolution, num_priors refers to the number of prior boxes owned by the feature layer, and num_classes refers to the total number of classes of targets detected by the network; the classification regression subnet (box subnet) uses 4 256-channel convolutions and 1 num_priors x 4 convolution, num_priors refers to the number of prior boxes owned by the feature layer, and 4 refers to the adjustment of the prior box. Finally, the face box in the face image is output through the class subnets, and the box subnets output the coordinates of the face box. Of course, the above is only one face frame detection method of this embodiment. Other face frame detection methods may be used in other embodiments.

[0081] Step S630: Obtain the positions of facial key points in the face frame.

[0082] In this embodiment, after obtaining the face frame, the face area cut out by the face frame in the face image is input into the face key point detection network, and the positions of the face key points are output.

[0083] Step S650: Obtain the facial key point coordinates of the facial key point positions according to the facial key point positions and the position coordinates of the face frame.

[0084] In this embodiment, the position coordinates of the face frame are used as a reference, and the facial key point coordinates of the facial key point positions are obtained according to the facial key point positions.

[0085] This embodiment can limit the number of key points. For example, when inputting the facial key point detection network, it is set to 106 key points, then the positions of the 106 key points are input, and finally the data of (106, 2) is obtained, that is, the coordinates of the 106 key points on the facial image. Of course, it can also be other numbers of key points, which can be pre-set according to specific needs.

[0086] In a specific embodiment, the Figure 8The facial key point detection network shown in the figure performs facial key point detection. Specifically, the facial area corresponding to the face frame cut out of the face image is input into the facial key point detection network. Through the funnel-shaped network, upsampling and downsampling are performed multiple times to obtain the position features of the facial key points at multiple scales, and finally the coordinates of the facial key points are obtained.

[0087] In this embodiment, the face frame is cut out, the coordinates of the face frame are obtained, and the face frame is cut out for facial key point recognition to obtain the position of the facial key point. At the same time, the coordinates of the face frame are used as the reference coordinate system to output the coordinates of the facial key points on the face image. By obtaining the precise position of the facial key points on the face image, the foundation is laid for subsequent face alignment.

[0088] Figure 9 yes Figure 6 The flowchart of step S630 in the embodiment shown is in an exemplary embodiment. Figure 9 As shown, in an exemplary embodiment, the process of obtaining the positions of key facial points in the face frame may include steps S910 to S930, which are described in detail as follows:

[0089] Step S910: converting the size of the face frame to a preset size;

[0090] In this embodiment, before obtaining the coordinates of the facial key points, the size of the face frame is converted to a preset size. For example, the size of the face frame can be scaled to a resolution of 256*256 or other sizes.

[0091] Step S930: Obtain the positions of facial key points in the face frame converted to a preset size.

[0092] In this embodiment, the corresponding face area in the face frame converted to a preset size is input into the face key point detection network to obtain the position of the face key points after size conversion.

[0093] In this embodiment, by converting the face frame size, the face frame sizes in the face images to be aligned are made to be within the same range, that is, the reference systems of the coordinates of the facial key points in the obtained face images are made the same, thereby improving the accuracy of face alignment.

[0094] Figure 10 yes Figure 5 The flowchart of step S530 in the embodiment shown is in an exemplary embodiment. Figure 6 As shown, in an exemplary embodiment, the process of aligning the face region of the first face image with the face region of the second face image based on the first face key point coordinates and the second face key point coordinates to obtain the target face image after the first face image is aligned includes steps S101 to S105, which are detailed as follows:

[0095] Step S101: performing feature normalization processing on the coordinates of the first facial key point and the coordinates of the second facial key point respectively to obtain the first key point feature and the second key point feature.

[0096] In this embodiment, feature normalization is performed on the key point coordinates of the two facial images. Specifically, the means of the first facial key point coordinates and the second facial key point coordinates are respectively subtracted and then divided by the variance to obtain the first key point feature and the second key point feature.

[0097] Step S103: Regularize the first key point feature and the second key point feature, and perform singular value decomposition on the data obtained after the regularization process to obtain an affine transformation matrix.

[0098] In this embodiment, the first key point feature and the second key point feature are regularized, and then SVD (singular value decomposition) is used to obtain an affine transformation matrix.

[0099] In a specific embodiment, the affine transformation matrix can be obtained by:

[0100] R=arg min Ω ||ΩA-B||subject to Ω T Ω=I

[0101] Where A and B are the coordinates of the first facial key point of the first face image and the second facial key point of the second face image, respectively. Ω is the least squares coefficient, R is the affine transformation matrix, I is the identity matrix, and T is the transpose. The second face image is used as the standard shape, and the least squares method is used to calculate the affine transformation matrix from the first face image to the second face image.

[0102] Step S105: performing affine transformation on the first facial image using an affine transformation matrix to obtain a target facial image.

[0103] After obtaining the affine transformation matrix, the affine transformation is performed on the face image to be aligned using the affine transformation matrix to obtain a target face image, the face angle and orientation of the target face image are the same as those of the other face image that is not aligned.

[0104] In one embodiment, when the facial area of ​​the first facial image needs to be aligned with the facial area of ​​the second facial image, the first facial image is affine transformed through an affine transformation matrix to obtain a target facial image after the first facial image is aligned, and the facial angle and orientation of the target facial image are the same as the facial angle and orientation of the second facial image; when the facial area of ​​the second facial image needs to be aligned with the facial area of ​​the first facial image, the second facial image is affine transformed through an affine transformation matrix to obtain a target facial image after the second facial image is aligned, and the facial angle and orientation of the target facial image are the same as the facial angle and orientation of the first facial image.

[0105] In this embodiment, by aligning the facial area of ​​the facial image to be aligned with the facial area of ​​another facial image, a target facial image with the same angle and orientation as the other facial image is obtained. The alignment process is simple and the processing effect is high, which can provide a basis for subsequent facial fusion processing.

[0106] Figure 11 FIG. 1 is a structural diagram of an image processing device according to an exemplary embodiment. Figure 11 As shown, the device includes:

[0107] The face alignment module 111 is configured to align the face region of the first face image with the face region of the second face image to obtain a target face image after the first face image is aligned;

[0108] A multi-scale feature map acquisition module 113 is configured to extract features of the target face image and the second face image at multiple scales, respectively, to obtain corresponding first multi-scale feature maps and second multi-scale feature maps;

[0109] a mask acquisition module 115 configured to acquire a first mask of a target facial image and a second mask of a second facial image;

[0110] The face fusion module 117 is configured to perform face fusion processing on the target face image and the second face image based on the first residual link established between the first mask and the first multi-scale feature map, and the second residual link established between the second mask and the second multi-scale feature map to obtain a fused face image.

[0111] This embodiment provides an image processing device, based on which face fusion between any two face images can be achieved.

[0112] In another exemplary embodiment, the image processing apparatus further includes:

[0113] The color correction module is configured to perform color correction on the face area of ​​the target face image according to the mask of the target face image.

[0114] In another exemplary embodiment, the image processing apparatus further includes:

[0115] The angle adjustment module is configured to adjust the angle of the face in the target face image.

[0116] The angle verification module is configured to continue adjusting the angle of the face in the target facial image until the difference between the adjusted angle and the angle of the face in the second facial image is within the preset error threshold if the difference between the adjusted angle and the angle of the face in the second facial image is not within the preset error threshold.

[0117] The angle adjustment module and angle verification module proposed in this embodiment can also work together with the face alignment module 111, the multi-scale feature map acquisition module 113, the mask acquisition module 115, and the face fusion module 117 in the above-mentioned embodiment for face fusion. Specifically, face alignment is performed by the face alignment module 111, and then the face angle of the target image is adjusted by the angle adjustment module and the angle verification module until the difference between the face angle of the target image and the angle of the face in the second face image is within a preset error threshold. Then, the target image after the face angle is adjusted is subjected to face fusion processing with the second face image through the multi-scale feature map acquisition module 113, the mask acquisition module 115, and the face fusion module 117 to obtain a fused face image.

[0118] In another exemplary embodiment, the face alignment module 111 includes:

[0119] A key point coordinate acquisition unit configured to acquire the coordinates of facial key points of a facial image; the facial key point coordinates of the facial image include the coordinates of the first facial key point of the first facial image and the coordinates of the second facial key point of the second facial image;

[0120] a face alignment unit configured to align a face region of the first face image with a face region of the second face image based on the coordinates of the first face key points and the coordinates of the second face key points, to obtain a target face image after the first face image is aligned;

[0121] In another exemplary embodiment, the key point coordinate acquiring unit includes:

[0122] The face frame position acquisition module is configured to obtain the position coordinates of the face frame in the face image; the face frame covers the face area in the face image;

[0123] The facial key point position acquisition module is configured to obtain the facial key point positions in the face frame;

[0124] The facial key point coordinate acquisition module is configured to obtain the facial key point coordinates of the facial key point positions based on the facial key point positions and the position coordinates of the face frame.

[0125] In another exemplary embodiment, the facial key point position acquisition section includes:

[0126] a size conversion block configured to convert the size of the face frame to a preset size;

[0127] The facial key point position acquisition block is configured to obtain the facial key point positions in a face frame converted to a preset size.

[0128] In another exemplary embodiment, the face alignment unit includes:

[0129] A key point feature acquisition module is configured to perform feature normalization processing on the coordinates of the first facial key point and the second facial key point respectively to obtain the first key point feature and the second key point feature;

[0130] An affine transformation matrix acquisition section is configured to perform regularization processing on the first key point feature and the second key point feature, and perform singular value decomposition on the data obtained after the regularization processing to obtain an affine transformation matrix;

[0131] The target face image acquisition module is configured to perform affine transformation on the first face image through an affine transformation matrix to obtain a target face image.

[0132] It should be noted that the apparatus provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0133] An embodiment of the present application further provides an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, which implement the image processing method as described above when executed by the processor.

[0134] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0135] It should be noted that Figure 12 The computer system 1600 of the electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0136] like Figure 12As shown, computer system 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1602 or programs loaded from storage portion 1608 into random access memory (RAM) 1603, such as executing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1603. CPU 1601, ROM 1602, and RAM 1603 are connected to each other via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0137] The following components are connected to the I / O interface 1605: an input section 1606 including a keyboard, a mouse, and the like; an output section 1607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1608 including a hard disk; and a communication section 1609 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as needed. Removable media 1611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1610 as needed, so that computer programs read from the removable media can be installed in the storage section 1608 as needed.

[0138] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1609, and / or installed from a removable medium 1611. When the computer program is executed by the central processing unit (CPU) 1601, the various functions defined in the system of the present application are executed.

[0139] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0142] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device.

[0143] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above embodiments.

[0144] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main ideas and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. An image processing method, characterized in that: include: Aligning the facial region of the first facial image with the facial region of the second facial image to obtain a target facial image after the first facial image is aligned; Extracting features of the target facial image and the second facial image at multiple scales respectively to obtain corresponding first multi-scale feature maps and second multi-scale feature maps; Acquire a first mask of the target facial image and a second mask of the second facial image; Based on a first residual link established between the first mask and the first multi-scale feature map, and a second residual link established between the second mask and the second multi-scale feature map, face fusion processing is performed on the target face image and the second face image to obtain a fused face image.

2. The method according to claim 1, characterized in that Before respectively extracting features of the target facial image and the second facial image at multiple scales to obtain corresponding first multi-scale feature maps and second multi-scale feature maps, the method further includes the steps of: Color correction is performed on the face region of the target face image according to the mask of the target face image.

3. The method according to claim 1, characterized in that Before respectively extracting features of the target facial image and the second facial image at multiple scales to obtain corresponding first multi-scale feature maps and second multi-scale feature maps, the method further includes the steps of: Adjusting the angle of the face in the target face image; If the difference between the adjusted angle and the angle of the face in the second facial image is not within the preset error threshold, continue to adjust the angle of the face in the target facial image until the difference between the adjusted angle and the angle of the face in the second facial image is within the preset error threshold.

4. The method according to claim 1, wherein The aligning the facial region of the first facial image with the facial region of the second facial image to obtain a target facial image after the alignment of the first facial image includes: Obtaining facial key point coordinates of a facial image; the facial key point coordinates of the facial image include the first facial key point coordinates of the first facial image and the second facial key point coordinates of the second facial image; According to the coordinates of the first facial key points and the coordinates of the second facial key points, the facial region of the first facial image is aligned with the facial region of the second facial image to obtain a target facial image after the alignment of the first facial image.

5. The method according to claim 4, characterized in that The step of aligning the facial region of the first facial image with the facial region of the second facial image based on the first facial key point coordinates and the second facial key point coordinates to obtain a target facial image after the first facial image is aligned includes: Performing feature normalization processing on the first facial key point coordinates and the second facial key point coordinates respectively to obtain first key point features and second key point features; Regularizing the first key point feature and the second key point feature, and performing singular value decomposition on the data obtained after the regularization process to obtain an affine transformation matrix; Performing an affine transformation on the first facial image using the affine transformation matrix to obtain the target facial image.

6. The method according to claim 4, characterized in that The step of obtaining the coordinates of key facial points of a facial image includes: Obtaining the position coordinates of a face frame in the face image; wherein the face frame covers the face area in the face image; Obtaining the positions of key facial points in the face frame; The facial key point coordinates of the facial key point position are obtained according to the facial key point position and the position coordinates of the face frame.

7. The method according to claim 6, characterized in that The obtaining of the positions of key facial points in the face frame includes: Converting the size of the face frame to a preset size; Gets the positions of facial landmarks in a face frame converted to a preset size.

8. An image processing device, characterized in that: include: a face alignment module configured to align a face region of the first face image with a face region of the second face image to obtain a target face image after the first face image is aligned; a multi-scale feature map acquisition module, configured to extract features of the target facial image and the second facial image at multiple scales, respectively, to obtain corresponding first multi-scale feature maps and second multi-scale feature maps; a mask acquisition module configured to acquire a first mask of the target facial image and a second mask of the second facial image; The face fusion module is configured to perform face fusion processing on the target face image and the second face image based on a first residual link established between the first mask and the first multi-scale feature map, and a second residual link established between the second mask and the second multi-scale feature map, to obtain a fused face image.

9. An electronic device, characterized in that: include: a memory storing computer-readable instructions; The processor reads the computer-readable instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images