A face image processing method and related device

Abstract

The application provides a face image processing method and related equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining a full-face feature point set of a face image to be processed; adjusting corresponding feature points in the full-face feature point set according to face feature adjustment information, to obtain an adjusted full-face feature point set; inputting the adjusted full-face feature point set and the face image to be processed into a trained image completion model, to obtain a full-face image in which a face non-display area in the face image to be processed is completed. The method completes the face image to be processed based on the full-face feature point set, realizes more accurate face completion, and realizes more abundant and accurate completion of the face image by accurately adjusting the full-face feature point set.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a facial image processing method and related equipment. Background Technology

[0002] With the continuous development of science and technology, image completion technology has gained increasing popularity. Face image completion technology refers to completing a face image containing incomplete regions to obtain a complete face image. This typically involves extracting feature maps from the face image containing the incomplete regions, then predicting the incomplete face image based on these feature maps, and finally obtaining the completed face image. Current face completion methods usually extract hierarchical feature images from the incomplete face image, performing non-linear representation and completion of the incomplete face image. Summary of the Invention

[0003] This application provides a facial image processing method and related equipment, which provides a facial image processing method based on feature points, and can adjust facial features as needed to obtain a corresponding full-face image.

[0004] Firstly, a facial image processing method is provided, including:

[0005] Obtain a face image to be processed, the face image to be processed including a portion of the face that is not displayed;

[0006] The face image to be processed is input into a trained facial feature point prediction model to obtain the full-face feature point set of the face image to be processed.

[0007] Based on the facial feature adjustment information, the corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set;

[0008] The adjusted set of full-face feature points and the face image to be processed are input into a trained image completion model to obtain a full-face image with the undisplayed areas of the face in the face image to be processed completed.

[0009] Secondly, a facial image processing method is provided, including:

[0010] In response to a face image completion input operation on the face image completion interface, a completion request is sent to the server so that the server processes the face image to be processed and the face feature adjustment information according to any of the methods described in the first aspect to obtain a full-face image after face image completion; wherein, the completion request includes the face image to be processed and the face feature adjustment information;

[0011] Obtain and display the full-face image returned by the server.

[0012] Thirdly, a facial image processing apparatus is provided, comprising:

[0013] A transceiver module is used to obtain a face image to be processed, the face image to be processed including a portion of the face that is not displayed;

[0014] The processing module is configured to input the face image to be processed into a trained facial feature point prediction model to obtain a full-face feature point set of the face image to be processed; and, according to facial feature adjustment information, adjust the corresponding feature points in the full-face feature point set to obtain an adjusted full-face feature point set; and, input the adjusted full-face feature point set and the face image to be processed into a trained image completion model to obtain a full-face image with the undisplayed facial areas in the face image to be processed completed.

[0015] In one possible embodiment, the facial feature point prediction model includes a first encoding module, multiple second encoding modules, and a fully connected module, wherein the processing module is specifically used for:

[0016] The first encoding module extracts features from the face image to be processed at multiple scales to obtain a first feature map set.

[0017] The first feature map set is subjected to convolutional pooling processing through each of the plurality of second encoding modules to obtain the second feature value set;

[0018] The fully connected module is used to concatenate multiple sets of second feature values ​​to obtain a full-face feature point set.

[0019] In one possible embodiment, the facial feature adjustment information is an expression classification label, and the processing module is specifically used for:

[0020] The expression classification labels and the full-face feature point set are input into the face adjustment model to obtain the adjusted full-face feature point set;

[0021] The facial adjustment model is based on a set of sample facial expression images, where each sample facial expression image in the set is labeled with a corresponding expression classification label.

[0022] In one possible embodiment, the processing module is specifically used for:

[0023] Extract the facial contour features corresponding to the facial feature adjustment information; wherein, the facial contour features are used to represent the contour formed by facial key points;

[0024] Based on the facial contour features, the coordinates of the corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set.

[0025] In one possible embodiment, the image completion model includes a third encoding module, a dilated convolution module, and a decoding module, wherein the processing module is specifically used for:

[0026] The adjusted full-face feature point set is converted into a facial feature point map;

[0027] The third encoding module performs convolution processing on the facial feature point map and the face image to be processed at different scales to obtain a third feature map set; wherein, the third feature map set includes a convolution-processed feature map set, and the convolution-processed feature map set is the final convolution feature map output by the third encoding module during the process of performing convolution processing on the facial feature point map and the face image to be processed at different scales.

[0028] The fourth feature map is obtained by performing dilated convolution on the convolution-processed feature map set through the residual unit in the dilated convolution module.

[0029] The fifth feature map is obtained by extracting the fourth feature map set and the local features of the convolutional feature map set through the attention unit in the dilated convolution module.

[0030] The decoding module upsamples the fifth feature set and the third feature set to obtain a full-face image with the undisplayed areas of the face in the face image to be processed.

[0031] In one possible embodiment, the third feature map set further includes a first intermediate feature map set and a second intermediate feature map set. The first intermediate feature map set is an intermediate convolution result obtained by sequentially performing a preset number of convolution processes on the facial feature point map and the facial image to be processed. The second intermediate feature map set is an intermediate convolution result obtained by performing a preset number of convolution processes on the first intermediate feature map set. The processing module is specifically used for:

[0032] The decoding module performs upsampling processing on the fifth feature map set to obtain the sixth feature map set;

[0033] The decoding module performs weighted processing on the second intermediate feature map set and the sixth feature map set to obtain the seventh feature map set.

[0034] The decoding module performs weighted and upsampling processing on the seventh feature map set and the first intermediate feature map set to obtain the eighth feature map set.

[0035] The decoding module performs convolution processing on the eighth feature map set to obtain a full-face image with the undisplayed areas of the face in the face image to be processed completed.

[0036] In one possible embodiment, the facial feature point prediction model is trained through the following steps:

[0037] Obtain a first sample dataset; wherein the first sample dataset includes the set of sample face images to be processed, and the set of ground value sample full-face feature points corresponding to each sample face image to be processed;

[0038] Based on the first sample dataset, a facial feature point prediction model is trained;

[0039] The trained facial feature point prediction model is obtained when the prediction loss of the facial feature point prediction model meets the first preset condition; wherein, the prediction loss is used to represent the loss between the predicted sample full-face feature point set and the ground value sample full-face feature point set.

[0040] In one possible embodiment, the image completion model includes a generative sub-model and a discriminative sub-model, and the loss of the image completion model is obtained by weighting pixel-wise loss, perceptual loss, style loss, total differential loss, and adversarial loss.

[0041] Wherein, the pixel-wise loss is used to represent the pixel value error between the completed sample full-face image and the sample face image to be processed; the perceptual loss is used to represent the error between the feature map of the completed sample full-face image and the feature map of the sample face image to be processed; the style loss is used to represent the sum of the Gram matrix differences between the feature maps of the completed sample full-face image with the occluded region and the sample face image to be processed with the occluded region, after vectorization of each channel information; the total differential loss is used to represent the ratio between the sum of the horizontal and vertical derivatives of the completed sample full-face image and the total number of pixels included in the sample face image to be processed; and the generation loss is used to represent the loss of the discriminant sub-model on the discrimination result of the generation sub-model and the true source of the generation result.

[0042] Secondly, a facial image processing method is provided, including:

[0043] In response to a face image completion input operation on the face image completion interface, a completion request is sent to the server so that the server processes the face image to be processed and the face feature adjustment information according to any of the methods described in the first aspect to obtain a full-face image after face image completion; wherein, the completion request includes the face image to be processed and the face feature adjustment information;

[0044] Obtain and display the full-face image returned by the server.

[0045] Thirdly, a facial image processing apparatus is provided, comprising:

[0046] A transceiver module is used to obtain a face image to be processed, the face image to be processed including a portion of the face that is not displayed;

[0047] The processing module is configured to input the face image to be processed into a trained facial feature point prediction model to obtain a full-face feature point set of the face image to be processed; and, according to facial feature adjustment information, adjust the corresponding feature points in the full-face feature point set to obtain an adjusted full-face feature point set; and, input the adjusted full-face feature point set and the face image to be processed into a trained image completion model to obtain a full-face image with the undisplayed facial areas in the face image to be processed completed.

[0048] Fourthly, a terminal device is provided, comprising:

[0049] A sending module is configured to respond to a face image completion input operation on a face image completion interface, send a completion request to a server, so that the server processes the face image to be processed and the face feature adjustment information according to any of the methods described in the first aspect to obtain a full-face image after face image completion; wherein, the completion request includes the face image to be processed and the face feature adjustment information; and obtain the full-face image returned by the server;

[0050] The display module is used to display the full-face image.

[0051] Fifthly, a facial image processing device is provided, comprising:

[0052] At least one processor, and

[0053] A memory that is communicatively connected to the at least one processor;

[0054] The memory stores instructions executable by the at least one processor, which implements the method as described in any one of the first or second aspects by executing the instructions stored in the memory.

[0055] A sixth aspect provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first or second aspects.

[0056] Since the embodiments of this application adopt the above-described technical solution, they have at least the following technical effects:

[0057] In this embodiment, a face completion method based on feature points is implemented by obtaining a full-face feature point set of the face image to be processed. Furthermore, by precisely modifying the full-face feature point set, a more accurate adjusted full-face image is obtained. Since the user does not need to manually adjust the full-face feature point set, the efficiency of adjusting the full-face feature point set can be improved, thereby increasing the efficiency of face image processing. Further, different facial feature adjustment information can be set as needed to generate different full-face images to meet the personalized needs of different users and enhance the fun of face completion. By adjusting facial feature points, various aspects of the image, such as pose, expression, and facial contour, can be modified, realizing an adjustable image completion scheme. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the structure of a facial image processing device provided in an embodiment of this application;

[0059] Figure 2 A schematic diagram illustrating an application scenario of a facial image processing method provided in this application embodiment;

[0060] Figure 3 A schematic diagram illustrating the principle of a facial image processing method provided in this application embodiment;

[0061] Figure 4 A schematic diagram showing the distribution of the various models provided in the embodiments of this application;

[0062] Figure 5 Sample face images to be processed generated by the two occlusion methods provided in the embodiments of this application;

[0063] Figure 6 An example diagram illustrating the process of adjusting a full-face feature point set provided in an embodiment of this application;

[0064] Figure 7 This application provides a schematic diagram illustrating a process for adjusting a full-face feature point set.

[0065] Figure 8 An interactive schematic diagram of a facial image processing method provided in an embodiment of this application;

[0066] Figure 9 A schematic diagram of a face image completion interface provided in an embodiment of this application;

[0067] Figure 10 This application provides a schematic diagram of a process for generating an image to be processed by adding occlusion to an original image, as provided in an embodiment of the present application.

[0068] Figure 11An example of a process for completing a full-face image provided in this application embodiment. Figure 1 ;

[0069] Figure 12 An example of a process for completing a full-face image provided in this application embodiment. Figure 2 ;

[0070] Figure 13 An example of a process for completing a full-face image provided in this application embodiment. Figure 3 ;

[0071] Figure 14 An example of a process for completing a full-face image provided in this application embodiment. Figure 4 ;

[0072] Figure 15 An example of a process for completing a full-face image provided in this application embodiment. Figure 5 ;

[0073] Figure 16 An example of a process for completing a full-face image provided in this application embodiment. Figure 6 ;

[0074] Figure 17 A schematic diagram of an interface for displaying a full-face image provided in this application embodiment. Figure 1 ;

[0075] Figure 18 A schematic diagram of an interface for displaying a full-face image provided in this application embodiment. Figure 2 ;

[0076] Figure 19 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0077] Figure 20 A schematic diagram of the structure of a terminal device provided in this application embodiment. Figure 1 ;

[0078] Figure 21 A schematic diagram of the structure of a terminal device provided in this application embodiment. Figure 2 . Detailed Implementation

[0079] To better understand the technical solutions provided in the embodiments of this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0080] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, the following is an introduction to the technical terms involved in this application.

[0081] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0082] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0083] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0084] Face image completion: Image completion refers to the process of supplementing incomplete facial images and editing facial attributes. The generated full-face image can be as accurate as the original full-face image or maintain consistency in content with the complete face image, making the completed full-face image look realistic. Editing facial attributes includes modifying one or more of facial expressions, facial poses, and facial contours. Face image completion can be applied to everyday photo processing as well as in security applications.

[0085] The face image to be processed refers to a face image that includes areas of the face that are not displayed, such as a partially incomplete face or a face that is occluded. The face referred to in this application includes, but is not limited to, human or animal faces.

[0086] Facial feature adjustment information: This information is used to adjust the coordinates of various feature points in the full-face feature point set. It can be used for facial contours, expressions, and poses. The format of facial feature adjustment information can be coordinate points, images, or even simple line drawings, etc.

[0087] It should be noted that in this application, "face" refers to a human face or an animal face. This application mainly uses a human face as an example for detailed explanation, but the same principle can be used to complete the process for animal faces.

[0088] Design Concept

[0089] In various situations, such as poor lighting conditions during shooting or damage to the image caused by human or natural factors, facial images may be incomplete. For example, when a camera is used to photograph criminals at night, if a complete facial image is not captured, a facial completion method can be used to process the facial image.

[0090] Currently, facial image methods generally express and complete facial images non-linearly, which cannot achieve accurate completion or precise modification of the facial images to be processed.

[0091] In view of this, the inventors of this application provide a facial image processing method. First, the full-face feature point set of the face image to be processed is predicted. Then, based on facial feature adjustment information, the full-face feature point set is adjusted to obtain an adjusted full-face feature point set. Finally, based on the adjusted full-face feature point set and the face image to be processed, the face image to be processed is completed. Since this application extracts the full-face feature point set, it obtains information for accurate facial quantification, which is beneficial for accurately completing the face image. Furthermore, it facilitates more precise modification of the full-face feature point set. After modifying the full-face feature point set, the completed face image will also differ, enriching the completed face image and meeting the different needs of different users for full-face completion.

[0092] The inventors of this application further considered that manually adjusting the full-face feature point set is inefficient and cannot guarantee the accuracy of the full-face image generated from the adjusted feature point set. Therefore, the inventors of this application considered introducing a face adjustment model. Users only need to select the corresponding expression classification label, and the face adjustment model can achieve precise adjustment of the full-face feature point set.

[0093] Application scenario examples

[0094] Please refer to Figure 1 This is a schematic diagram of the structure of a face image processing device that performs the face image processing method in the embodiments of this application. The face image processing device 100 includes one or more input devices 101, one or more processors 102, one or more memories 103, and one or more output devices 104.

[0095] Input device 101 provides an input interface to acquire facial images to be processed from external devices or user input. After acquiring the input facial image, input device 101 sends it to processor 102. Processor 102 uses program instructions stored in memory 103 to complete the facial image and obtain the completed full-face image. The full-face image is then output through output device 104.

[0096] Input device 101 may include, but is not limited to, one or more of the following: physical keyboard, function keys, trackball, mouse, touchscreen, joystick, etc. Processor 102 may be a central processing unit (CPU) or a digital processing unit, etc. Memory 103 may be volatile memory, such as random-access memory (RAM); memory 103 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 103 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures, accessible by a computer, but is not limited thereto. Memory 103 may be a combination of the above-mentioned memories. Output device 104 may include, for example, a monitor, speaker, and printer, etc.

[0097] In possible embodiments, the facial image processing device 100 can be a user-end device or a server-end device. The user-end device can be a mobile terminal, fixed terminal, or portable terminal, such as a mobile phone, site, unit, device, multimedia computer, multimedia tablet, internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio receiver, e-book device, gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the facial image processing device 100 can support any type of user-facing interface (e.g., wearable devices). The server-end device can be a server providing various services, a large computing device, etc. The server can be one or more servers. The server can also be a physical server or a virtual server, etc.

[0098] In one possible application scenario, please refer to Figure 2 This describes an example application scenario where the facial image processing device 100 is implemented via a server 220. A user determines the facial image to be processed through a client 211 in a terminal device 210. The client 211 generates a completion request. The server 220 performs the aforementioned processing on the facial image to be processed, obtains a full-face image, and returns the full-face image to the client 211. The client 211 generally refers to any client 211 capable of facial completion, such as a photo-taking app or an image processing app. This application does not limit the specific type of client 211.

[0099] Working principle

[0100] Please refer to Figure 3 This is a schematic diagram illustrating the principle of processing facial images. The image processing process mainly includes three parts: predicting the full-face feature point set 310, adjusting the full-face feature point set 320, and completing the facial image to be processed 330.

[0101] Specifically, the face image to be processed is I, and the undisplayed area of ​​this face image can be represented as M. Therefore, the face image to be processed can be represented as:

[0102]

[0103] The period "." represents the Hadamard product of matrices.

[0104] The facial image processing device 100 predicts a set of full-face feature points L, adjusts the set of full-face feature points L according to facial feature adjustment information to obtain L', and then inputs L' and the facial image I to be processed into the image completion model to obtain the full-face image. The final full-face image It can be represented as follows:

[0105]

[0106] in, This indicates the area to be filled in for areas that are not displayed.

[0107] The following section explains part 310, which describes the prediction of the full-face feature point set.

[0108] The face image processing device 100 can obtain the full-face feature point set of the face image to be processed through a face feature point prediction model.

[0109] In one possible embodiment, please refer to Figure 4 The facial feature point prediction model 410 may include a first encoding module 411, multiple second encoding modules 412, and a fully connected module 413, wherein:

[0110] The first encoding module 411 includes convolutional layers of various scales. After an image passes through each layer in the first encoding module 411 in sequence, each convolutional layer can obtain a corresponding feature map, thereby obtaining a first feature map set. That is, the first feature map set includes multiple feature maps. It should be noted that, without explicit limitation, the feature map set in this application may include one feature map or multiple feature maps.

[0111] After obtaining the first feature map set Figure 4 The example uses three second encoding modules 412, but the actual number is not limited. Each second encoding module 412 includes one or more convolutional layers and one or more pooling layers. Different second encoding modules 412 can perform different convolutional and pooling processes on the first feature map set. Each second encoding module 412 outputs a corresponding feature value set, thereby obtaining the second feature value set. The convolutional and pooling processes can be performed by first performing convolution and then performing average pooling on the convolution result, or by first performing convolution and then performing max pooling on the convolution result, etc.

[0112] As one example, there are two coding modules with different scales in multiple second coding models 412.

[0113] In this embodiment, the first set of feature maps is processed by convolutional pooling through multiple second encoding modules 412. Processing at different scales can preserve different features of the first set of feature maps, making subsequent feature point prediction more accurate.

[0114] The second feature value set is concatenated using the fully connected module 413 to obtain the full-face feature point set. The full-face feature point set includes a set of coordinates representing key points of the face. The concatenation process can be understood as combining corresponding values ​​in the second feature value set into coordinates according to a preset order to obtain the corresponding full-face feature point set.

[0115] The structure of the facial feature point prediction model 410 is illustrated below with a specific example. A specific example of the facial feature point prediction model 410 is shown in Table 1 below:

[0116] Table 1

[0117]

[0118] In Table 1, c represents the number of output channels, s represents the stride of the convolutional layer, t represents the expansion factor, and n represents the number of times the corresponding layer is processed.

[0119] To facilitate understanding of the bottleneck layer, the structures of the bottleneck layers in Table 1 are illustrated below. The bottleneck layer with s=2 in Table 1 has the following structure: (1x1) convolution with (tk) channels - ReLU6 - (tk channels, stride 2, kernel size 3x3) depthwise convolution (each kernel convolves only one feature map) - ReLU6 - (1x1) convolution with k channels. For the bottleneck layer with s=1, its structure is: (1x1) convolution with (tk) channels - ReLU6 - (tk channels, stride 2, kernel size 3x3) depthwise convolution - ReLU6 - (1x1) convolution with k channels. This output is added to the input of the bottleneck layer to achieve residual connections.

[0120] Referring to Table 1, the first encoding module 411 includes a convolutional layer and multiple bottleneck layers (1x1 convolution - ReLU6 - 2-step, 3x3 channel-wise convolution) to obtain a first feature map set C1. The first feature map set C1 is then passed through the second encoding unit 412, as shown in Table 1. The second encoding unit includes a first convolutional layer with a 1x1 kernel and 1280 channels, a first average pooling layer, a second convolutional layer with a 1x1 kernel and 128 channels, a second average pooling layer, a third convolutional layer, and a third average pooling layer. The first feature map set C1 is input into the convolutional layer with a 1x1 kernel and 1280 channels to obtain a feature map set C2. Global average pooling is performed on the feature map set C2 to obtain the feature value set S1. In another branch, the first feature map set C1 is passed through a convolutional layer with a 1x1 kernel and 128 channels, and then subjected to global average pooling to obtain the feature value set S2. In the third branch, the feature map set C2 is passed through a convolutional layer with a 1x1 kernel and 128 channels, and then subjected to global average pooling to obtain the feature value set S3.

[0121] Each feature set (S1, S2, and S3) is input into the fully connected unit 413 (which is the fully connected unit in Table 1). Finally, S1, S2, and S3 are connected and passed through the fully connected layer to obtain 136 values. These 136 values ​​correspond to the x and y coordinates of 68 facial feature points, thus obtaining the final predicted coordinates of the full face feature points, which is the full face feature point set.

[0122] Based on the facial feature point prediction model 410 discussed above, the training process of the facial feature point prediction model 410 will be introduced below, including the part of obtaining the first sample dataset and the part of training the facial feature point prediction model 410 based on the first sample dataset. The two parts will be introduced separately below.

[0123] Obtain the first sample dataset:

[0124] Specifically, the first sample dataset includes a set of sample face images to be processed, and a ground truth full-face feature point set corresponding to each sample face image. Each sample face image includes an undisplayed region. The ground truth full-face feature point set refers to the set of true full-face feature points corresponding to the complete face image corresponding to the sample face image to be processed. The ground truth full-face feature point set can be manually annotated, or it can be data from online resources, or it can be generated by annotating other network models. For example, FAN can be used to annotate the ground truth full-face feature point set corresponding to the complete face image corresponding to the sample face image to be processed.

[0125] As one example, a set of sample full-face images can be input into the FAN network to obtain the set of ground-value sample full-face feature points corresponding to each sample full-face image. Then, some or all of the images in the sample full-face image set can be occluded to obtain more data.

[0126] Specifically, there are several ways to add occlusion, such as adding occlusions at random locations but with the same occlusion area size, or adding occlusions with random masks, where the position and size of the occlusion area are not fixed. The color of the added occlusion can be arbitrary; this application does not restrict the specific color of the occluded area. However, during a single training session, the added occlusions are all the same color, for example, white can be chosen. Different colored occlusions can be used for separate model training.

[0127] For example, please refer to 5. Figure 5 Image (a) represents a sample face image to be processed after occlusion at random locations but with the same occlusion area size. Figure 5 In the middle (b), a sample face image to be processed is shown after occlusion with a random mask added.

[0128] Furthermore, to expand the dataset, the occluded image can be flipped, blurred, and its photometric properties can be altered. Photometric processing could involve multiplying all pixels of the image by a preset factor, for example, ranging from 0.7 to 1.3.

[0129] A facial feature point prediction model 410 was trained based on the first sample dataset:

[0130] The prediction loss of the facial feature point prediction model 410 can be expressed as follows:

[0131]

[0132] Among them, L gt represents the coordinates of the true feature point, and ||·||2 represents the L2 norm.

[0133] Input the sample face image to be processed into the face feature point prediction model to obtain the predicted sample full face feature point set corresponding to the sample face image to be processed.

[0134] Based on the predicted sample full-face feature point set and the ground value sample full-face feature point set, the loss of the facial feature point prediction model 410 is determined. For example, the loss can be determined according to formula (3). The parameters are continuously updated according to the loss. For example, the Adam optimizer can be used to update the parameters, where the initial values ​​of the parameters in the Adam optimizer can be set as: β1 = 0, β2 = 0.9. The initial learning rate of the facial feature point prediction model 410 is set to 10. -4 .

[0135] When the loss of the facial feature point prediction model 410 meets a first preset condition, the trained facial feature point prediction model 410 is obtained. The first preset condition is, for example, that the loss of the facial feature point prediction model 410 is less than or equal to a certain threshold, or that the learning rate of the facial feature point prediction model 410 reaches a preset value.

[0136] In one possible embodiment, the facial feature point prediction model 410 can obtain the full-face feature point set of a species. However, since the distribution of facial feature points may differ among different species, sample datasets corresponding to different species can be sequentially trained through the above-described process to obtain the parameters corresponding to each species. The facial image processing device 100 can pre-store the parameters of various species and their corresponding parameters.

[0137] The following explains the adjustment of the full-face feature point set part 320.

[0138] In one possible embodiment, the face image processing device 100 can adjust the corresponding feature points in the full-face feature point set based on the face feature adjustment information using the face adjustment model 420, thereby obtaining the adjusted full-face feature point set.

[0139] The following example illustrates the face adjustment model 420.

[0140] A1:

[0141] When the facial feature adjustment information is a facial expression target image, the facial adjustment model 420 uses the facial feature point prediction model 410 described above. This model obtains the target full-face feature point set corresponding to the facial expression target image. Based on the relative positions of the feature points in this target full-face feature point set, the coordinate positions of each feature point in the full-face feature point set of the previous face image to be processed are adjusted, thereby obtaining the adjusted full-face feature point set. For example, the adjustment method involves adjusting the coordinate positions of each feature point in the full-face feature point set of the face image to be processed.

[0142] The adjustment method in this application embodiment can not only adjust the facial expression, but also the posture and outline of the facial image.

[0143] For example, please refer to Figure 6 , Figure 6 In the image, A represents the set of full-face feature points corresponding to the facial expression target image. Figure 6 In the image, B represents the set of full-face feature points in the face image to be processed. This is based on the set of full-face feature points corresponding to the target facial expression image (e.g., ...). Figure 6 The coordinates of the full-face feature point set (a2, b2, and c1) in the face image to be processed are adjusted using a1, b1, and c1 to obtain... Figure 6The set of full-face feature points shown in Figure C shows that, compared to Figure B, the facial features in Figure C not only change in expression, but also in facial contour and posture.

[0144] When the face adjustment model 420 adopts the face feature point prediction model 410 described above, the training process of the face adjustment model 420 can be referred to the training process of the face feature point prediction model 410 described above, and will not be repeated here.

[0145] A2:

[0146] By inputting the expression classification labels and the full-face feature point set into the face adjustment model 420, the adjusted full-face feature point set is obtained.

[0147] Specifically, the face adjustment model 420 can adjust the coordinates of feature points in the full-face feature point set based on expression classification labels or facial expression images, thereby obtaining an adjusted full-face feature point set. The structure of the face adjustment model 420 is illustrated below. Expression classification labels can be represented using one-hot encoding or other methods.

[0148] The face reshaping model 420 may include multiple fully connected layers and multiple activation layers, with the same number of fully connected layers and activation layers, and they are spaced apart. The fully connected layers and activation layers perform multi-layer operations to transfer the coordinates of feature points in the full-face feature point set.

[0149] Specifically, the face adjustment model 420 includes a first fully connected layer, a first activation layer, a second fully connected layer, a second activation layer, a third fully connected layer, and a third activation layer connected in sequence.

[0150] The training method for the face adjustment model 420 involved in A2 is explained below.

[0151] Please refer to Figure 7 A set of sample facial expression images is obtained. Each sample facial expression image is labeled with the corresponding expression category label, such as smiling, laughing, crying, etc.

[0152] The loss function for constructing the face reshaping model 420 is shown in the following example:

[0153]

[0154] Among them, L gt Used to represent the set of ground truth feature points corresponding to sample facial expression images. Let 2 be the set of full-face feature points corresponding to the expression classification labels predicted by the face adjustment model, and let ||·|2 represent the L2 norm.

[0155] The trained face adjustment model 420 is obtained when the loss of the face adjustment model 420 meets the preset conditions.

[0156] In this embodiment, the face adjustment model 420 pre-learns the full-face feature point sets corresponding to various expression classification labels. When using the model, the corresponding expression classification label can be selected, and the face adjustment model 420 can adjust the coordinates of the feature points in the input full-face feature point set based on the expression classification label, thereby obtaining the trained face adjustment model 420. Since the full-face feature point sets corresponding to various expression classification labels are pre-trained, the accuracy of the face adjustment model 420 in adjusting the full-face feature point set can be relatively improved.

[0157] A3:

[0158] The facial image processing device 100 can extract facial contour features corresponding to facial feature adjustment information, and adjust the coordinates of corresponding feature points in the full-face feature point set according to the facial contour features to obtain the adjusted full-face feature point set.

[0159] Specifically, the facial feature adjustment information can be a simple line drawing created by the user or an image of a face with facial expressions. Regardless of the type, the facial image processing device 100 can extract the facial contour features from the facial feature adjustment information. Alternatively, the facial image processing device 100 can extract the facial contour features using a pre-trained contour extraction model. These facial contour features are used to represent the contour formed by some or all of the key points of the face.

[0160] After obtaining the facial contour features, the coordinates of the corresponding feature points in the full-face feature point set can be offset based on the facial contour to obtain the adjusted full-face feature point set.

[0161] In this embodiment, it is not necessary to pre-train the corresponding model, which can reduce the processing load of the face image processing device 100. In addition, it can process simple drawings, etc., and improve the accuracy of image processing by users.

[0162] The following section introduces the completion part 330 of the face image to be processed.

[0163] An image completion model can be used to complete the face image by adjusting the full-face feature point set and the face image to be processed, thereby obtaining a full-face image with the missing face areas in the face image to be processed.

[0164] Please continue to refer to Figure 4The image completion model includes a generation sub-model 430, which includes a third encoding module 431, a dilated convolution module 432, and a decoding module 433. The adjusted full-face feature point set and the face image to be processed are input into the generation sub-model 430.

[0165] Specifically, the adjusted full-face feature point set is first converted into a facial feature point map. This conversion process can be understood as representing each feature point in the form of a feature map. For example, the corresponding position of the adjusted full-face feature point set in the face image to be processed can be set to 1, and the rest can be set to 0, thereby obtaining the facial feature point map.

[0166] The third encoding module 431 may include multiple convolutional layers connected in sequence. These convolutional layers perform convolutional processing on the face image to be processed and the face feature point map at different scales, obtaining feature maps output by each convolutional layer, thereby obtaining a third feature map set. The third feature map set includes a first intermediate feature map set, a second intermediate feature map set, and a convolutionally processed feature map set. The first intermediate feature map set consists of feature maps obtained from the initial convolutional layers of the third encoding module 431, the second intermediate feature map set consists of feature maps obtained from the middle convolutional layers of the third encoding module 431, and the convolutionally processed feature map set consists of feature maps output by the final convolutional layer of the third encoding module 431.

[0167] The fourth feature map set is obtained by performing dilated convolution on the convolution-processed feature map through the residual unit in the dilated convolution module 432. The fifth feature map set is obtained by extracting the fourth feature map set and the local features of the convolution-processed feature map through the attention unit in the dilated convolution module 432.

[0168] The processing mechanism of the attention unit will be introduced below.

[0169] Specifically, the attention unit is used to capture the correlations between long-range spatial context features, and its specific calculation process is as follows:

[0170] For the feature f of the fourth feature set d First, Q(f) is obtained through a convolutional layer with a kernel size of 1x1. d ). Then calculate f d The inner product s of component i with respect to component j ij s ij =[Q(f d ) T ] i Q(f d ) j Then, normalization is performed using the softmax function to obtain f. di For f dj Attention value After that, f was obtainedd Other components for f dj The attention-weighted features and c dj The calculation method is as follows The final feature portion result of the fourth feature map set is y. d =γ d c d +f d Among them, γ d These are trainable parameters.

[0171] For the feature map after convolution, the feature f e It uses f d The resulting attention values ​​are weighted and summed to obtain c. ej ,c ej =∑ i β j,i f ei The final output feature map after convolution processing is y. e =γ e (1-M)c e +Mf e Among them, γ e These are trainable parameters.

[0172] The fifth feature map is obtained by concatenating the feature results of the fifth feature map set with the feature results of the convolutional feature map set.

[0173] The decoding module 433 performs upsampling processing on the fifth feature set and the third feature set to obtain a full-face image with the undisplayed areas of the face in the face image to be processed.

[0174] Specifically, the decoding module 433 can decode all features in the fifth feature map set and the third feature map set. Figure 1 Upsampling is performed to obtain the final full-face image.

[0175] The decoding module 433 can also upsample the fifth feature map set to obtain the sixth feature map set. Then, it performs a weighted sum upsampling on the second intermediate feature map set and the sixth feature map set to obtain the seventh feature map set. Finally, it performs a weighted sum upsampling on the seventh feature map set and the first feature map set. set The weighted features are then used to obtain the eighth feature map set. The eighth feature map set is then convolved by the decoding module to obtain the completed full-face image.

[0176] For example, one specific generative sub-model 430 is shown in the table below:

[0177] Table 2

[0178]

[0179] Where k, c, s, and p represent the kernel size, number of output channels, stride, and padding of the convolutional or deconvolutional layer, respectively. Except for the convolutional layers in the first row of Table 1 that use reflection padding, the rest use zero padding. IN represents instance normalization.

[0180] The specific processing procedure for generating sub-model 430 shown in Table 1 is explained below.

[0181] Specifically, the facial feature map and the face image to be processed are first passed through a convolutional layer with a kernel size of 7×7, 3 channels, and 3 padding to obtain the first intermediate feature map set E1. Then, the first intermediate feature map set E1 is input into a convolutional layer with a kernel size of 4×4, 128 channels, a stride of 2, and 1 padding to obtain the second intermediate feature map set E2. Finally, E2 is passed through a convolutional layer with a kernel size of 4x4, 256 channels, a stride of 2, and 1 padding to obtain the convolutionally processed feature map set E3.

[0182] The fourth feature map set R7 is obtained by passing E3 through seven residual blocks with dilated convolutions as the first convolution operation. R7 and E3 are then fed into a Long Short-Term Attention (LSA) module to obtain the fifth feature map set. This fifth feature map set is then upsampled through a transposed convolutional layer with a kernel size of 4x4, 128 channels, a stride of 2, and padding of 1, resulting in the sixth feature map set D1. E2 and D1 are then fed into a 1x1 convolutional layer with 256 channels to obtain a weighted feature map set. This weighted feature map set is then upsampled through a transposed convolutional layer with a kernel size of 4x4, 256 channels, a stride of 2, and padding of 1, resulting in the seventh feature map set D2. D2 and E1 are then fed into a 1x1 convolutional layer with 128 channels to obtain the weighted eighth feature map set. The eighth feature map set is input into a convolutional layer with a kernel size of 7, a channel number of 3, and padding of 3. Finally, the full-face image is obtained by passing the tanh layer.

[0183] In this embodiment, the image completion model further includes a discrimination sub-model 440, used to determine whether the generation result of the generation sub-module 430 originates from the real source or the generation sub-model 420, and to determine whether the generation result matches the adjusted ground truth full-face feature point set.

[0184] The discriminant sub-model 440 can be a simple binary classification model or a discriminator structure of Patch-GAN, such as 70×70 Patch-GAN. The structure of the discriminant sub-model 440 is illustrated below.

[0185] Table 3

[0186]

[0187] Each row represents a sequence of listed layers. K, c, s, and p represent the kernel size, number of output channels, stride, and padding of the convolutional or deconvolutional layer, respectively. SN represents Spectral Normalization, LReLU represents Leaky ReLU, with a slope of 0.2. ReLU means that if x is greater than or equal to 0, the output is x; if x is less than or equal to 0, the output slope is multiplied by x. The discriminant sub-model 440 in Table 1 includes an attention layer for adaptively processing the corresponding features.

[0188] The training process of the image completion model is illustrated below:

[0189] The second sample dataset is obtained, which includes a set of sample face images to be processed, a set of ground truth full-face feature points for each sample face image to be processed, and a sample full-face image corresponding to the sample face image to be processed. The understanding of the ground truth full-face feature point set can be found in the previous discussion and will not be repeated here.

[0190] As one embodiment, a large number of sample full-face images are first obtained, and then some or all of the sample full-face images are occluded. The occlusion method can be referred to the above discussion, and will not be repeated here.

[0191] Based on the second sample dataset, train the image completion model until the loss of the image completion model meets the second preset condition, and obtain the trained image completion model.

[0192] The loss of an image completion model can be represented in various ways. In this embodiment, the loss of the image completion model is obtained by weighting the pixel-wise loss, perceptual loss, style loss, total differential loss, and generation loss.

[0193] Specifically, the pixel-wise loss is used to represent the pixel value error between the completed sample full-face image and the sample face image to be processed; the perceptual loss is used to represent the error between the feature map of the completed sample full-face image and the feature map of the sample face image to be processed; the style loss is used to represent the sum of the Gram matrix differences between the feature maps of the completed sample full-face image with occlusion and the sample face image to be processed with occlusion after vectorization of each channel information; the total differential loss is used to represent the ratio between the sum of the horizontal and vertical derivatives of the completed sample full-face image and the total number of pixels included in the sample face image to be processed; and the generation loss is used to represent the loss between the discrimination sub-model's judgment of the source of the generation result of the generation sub-model and the true source of the generation result.

[0194] The specific formulas for each loss are illustrated below.

[0195] A formula for calculating pixel-by-pixel loss is as follows:

[0196]

[0197] Where ||·||1 represents the L1 norm, and N m This indicates the number of pixels that are obscured.

[0198] One formula for calculating perceived loss is as follows:

[0199]

[0200] Where, φ p (·) represents N from the p-th layer extracted from a pre-trained network model (e.g., VGG19). p Each size is H p ×W p Feature map, φ is used to represent the feature map of the completed sample full-face image. p (I) Feature map used to represent the face image of the sample to be processed.

[0201] One formula for calculating style loss is as follows:

[0202]

[0203] Among them, G p (x)=φ p (x) T φ p (x) represents the expression about φ p The Gramm matrix of (x). This represents the image after the full-face sample image is completed and the occluded area is added. This represents the image of the face image to be processed after the occluded area has been added.

[0204] A formula for calculating total differential loss is as follows:

[0205]

[0206] Wherein, N I This represents the total number of pixels in I. This indicates finding the first derivative, including (horizontal direction) and (Vertical direction)

[0207] One formula for calculating generation loss is as follows:

[0208]

[0209]

[0210] Where G represents the generator sub-model 430, D represents the discriminator sub-model 440, and the generation loss L adv It is the sum of formula (9) and formula (10).

[0211] The formula for calculating the loss of the image completion model is as follows:

[0212] L inp =L pixel +λ perc L perc +λ style L style +λ tv L tv +λ adv L adv (11)

[0213] Using the second sample dataset, the parameters are adjusted based on each training result until the loss of the image completion model meets the second preset condition, thereby obtaining the parameters of the image completion model.

[0214] As one example, λ perc =0.1, λ style =250, λ tv =0.1, λ adv The hyperparameter setting is 0.01.

[0215] As one example, L is used alternately throughout the training process. inp Optimize the generator, use Optimize the discriminant sub-model 440 until the model converges.

[0216] As one example, the Adam optimizer can be used to adjust the parameters, where the Adam optimizer parameter values ​​are set as follows: β1 = 0, β2 = 0.9, and the initial learning rate of the generated sub-model 430 is 10.-4 The initial learning rate of the discriminant sub-model 440 is 10. -5 The batch size is set to 4. The batch size can be understood as the number of samples used in one training session.

[0217] Example process

[0218] based on Figure 2 The application scenarios discussed are described below, along with an introduction to facial image processing methods. Please refer to [link / reference]. Figure 8 The facial image processing method includes:

[0219] S810, client 211 responds to the face image completion input operation on the face image completion interface and generates a completion request.

[0220] Specifically, as discussed above, client 211 supports facial image processing. When a user needs to complete a facial image, they can open client 211 and input information on the facial image completion interface. Client 211 responds to this input by generating a completion request. The user can either customize the facial image to be processed and the facial feature adjustment information, or select a facial image to be processed and the facial feature adjustment information recommended by client 211. For example, client 211 can identify incomplete images from the user's previous photo album and recommend them for completion. After the user inputs or selects the facial image to be processed, client 211 essentially receives the user's completion instruction and generates a completion request. This completion request carries the facial image to be processed and the facial feature adjustment information, or the index of the facial image to be processed and the index of the facial feature adjustment information.

[0221] As one embodiment, the client 211 can receive user-defined facial feature adjustment information, such as user-defined line drawings.

[0222] For example, please refer to Figure 9 This is a schematic diagram of a facial image completion interface. After the user selects an image to be processed, the client 211 can recommend facial feature adjustment information, such as... Figure 9 The app displays options such as "smile," "normal," and "dissatisfied," and users can select facial feature adjustment information recommended by client 211. Clicking the operation control 930 is equivalent to the user performing an input operation. Alternatively, the user can click... Figure 9 The custom operation control 910 shown allows users to select corresponding facial feature adjustment information. When the user clicks the OK control 920, it confirms the input of the facial feature adjustment information.

[0223] Or, for example, please refer to Figure 10This illustrates a user-added occlusion method. After the user selects the image to be processed, client 211 displays... Figure 10 The user can also click on the interface shown in A. Figure 10 In the "Add Occlusion Control 1001" section, you can input your own defined facial feature adjustment information, such as adding... Figure 10 The graffiti shown in B is equivalent to the user performing an input operation when they click on the control.

[0224] S820, client 211 sends a completion request to server 220.

[0225] Specifically, after generating the completion request, client 211 sends the completion request to server 220.

[0226] S830, server 220 obtains the face image to be processed.

[0227] Specifically, after receiving the completion request, server 220 parses the information carried in the completion request to obtain the facial image to be processed. Simultaneously, server 220 can also obtain facial feature adjustment information.

[0228] S840, server 220 inputs the face image to be processed into the trained face feature point prediction model 410 to obtain the full face feature point set of the face image to be processed.

[0229] In one possible embodiment, the facial feature point prediction model 410 includes a first encoding module 411, multiple second encoding modules 412, and a fully connected module 413. A specific example of the process for obtaining the full-face feature point set is as follows:

[0230] The first encoding module 411 performs feature extraction at multiple scales on the face image to be processed to obtain a first feature map set.

[0231] The first feature map set is processed by convolutional pooling through each of the multiple second encoding modules 412 to obtain the second feature value set.

[0232] The full-face feature point set is obtained by concatenating multiple sets of second feature values ​​through the fully connected module 413.

[0233] The facial feature point prediction model 410, the first encoding module 411, the multiple second encoding modules 412 and the fully connected module 413 can be referred to the content discussed above, and will not be repeated here.

[0234] S850 adjusts the corresponding feature points in the full-face feature point set based on facial feature adjustment information to obtain the adjusted full-face feature point set.

[0235] The specific adjustment method is shown in the following example:

[0236] Method 1:

[0237] The facial feature adjustment information is input into the facial feature point prediction model 410 to obtain the full-face feature point set corresponding to the facial feature adjustment information. Then, the full-face feature point set of the face image to be processed is adjusted according to the full-face feature point set corresponding to the facial feature adjustment information to obtain the adjusted full-face feature point set.

[0238] The processing principle of this method can be referred to the previous discussion, and will not be repeated here. This method is suitable for situations where facial feature adjustment information is in the form of an image.

[0239] Method 2:

[0240] The facial feature adjustment information and the set of full-face feature points are input into the facial adjustment model 420 to obtain the adjusted set of full-face feature points.

[0241] Specifically, the facial adjustment model 420 can be referred to the previous discussion, and will not be repeated here. This method is suitable for situations where the facial feature adjustment information is an expression classification label.

[0242] Method 3:

[0243] Extract facial contour features corresponding to facial feature adjustment information; where facial contour features are used to represent the contour formed by facial key points;

[0244] Based on facial contour features, the coordinates of corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set.

[0245] S860, server 220 takes the adjusted set of full-face feature points and the input image completion model of the face image to be processed, and obtains a full-face image that completes the undisplayed areas of the face in the face image to be processed.

[0246] Specifically, the image completion model can be referred to the previous discussion, and will not be repeated here. The image completion model includes a generation sub-model 430, which includes a third encoding module 431, a dilated convolution module 432, and a decoding module 433. The method of obtaining a full-face image through the generation sub-model 430 will be explained below.

[0247] The adjusted set of full-face feature points is converted into a facial feature point map;

[0248] The third encoding module 431 performs convolution processing on the facial feature point map and the face image to be processed at different scales to obtain a third feature map set. The third feature map set includes convolution-processed feature maps. The third encoding module is the final convolution feature map output during the process of performing convolution processing on the facial feature point map and the face image to be processed at different scales.

[0249] The feature map after convolution is dilated by the residual unit in the dilated convolution module 432 to obtain the fourth feature map set.

[0250] The fifth feature map set is obtained by extracting the fourth feature map set and the local features of the feature map after convolution through the attention unit in the dilated convolution module 432.

[0251] The decoding module 433 performs upsampling processing on the fifth feature map set and the third feature map set to obtain a full-face image with the undisplayed areas of the face in the face image to be processed.

[0252] In one possible embodiment, the third feature map set further includes a first intermediate feature map and a second intermediate feature map. The first intermediate feature map is an intermediate convolution result obtained by sequentially performing a preset number of convolution processes on the facial feature point map and the facial image to be processed. The second intermediate feature map is an intermediate convolution result obtained by performing a preset number of convolution processes on the first intermediate feature map.

[0253] The following example illustrates how upsampling the fifth and third feature sets using the decoding module 433 can produce a full-face image that fills in the undisplayed areas of the face in the image to be processed.

[0254] The fifth feature map set is upsampled by the decoding module 433 to obtain the sixth feature map set;

[0255] The second intermediate feature map and the sixth feature map set are weighted and upsampled by the decoding module 433 to obtain the seventh feature map set;

[0256] The seventh feature map set and the first intermediate feature map are weighted by the decoding module 433 to obtain the eighth feature map set;

[0257] The eighth feature map set is convolved by the decoding module 433 to obtain a full-face image with the undisplayed areas of the face in the face image to be processed.

[0258] The generated sub-model 430 includes the third encoding module 431, the dilated convolution module 432, and the decoding module 433. The contents of these modules can be referred to in the previous discussion and will not be repeated here.

[0259] After the server 220 processes the facial image to be processed, it can obtain a full-face image.

[0260] In one possible embodiment, server 220 stores the correspondence between parameters of different species and different models (including the facial feature point prediction model, image completion model and facial adjustment model discussed above). After obtaining the facial image to be processed, server 220 first determines the species classification of the facial image to be processed, and then selects the parameters corresponding to the species to improve the accuracy of facial image processing.

[0261] For example, please refer to Figure 11 , Figure 11 In the image, A, B, C, D, and E represent the ground truth full-face image, the face image to be processed, and the image corresponding to the predicted set of facial feature points, respectively. Figure 11 The C-axis includes each feature point), the generated sub-model 430 directly completes the face image based on the full face feature point set, the face image completed based on the full face feature point set, and the full face image after stitching together the face images to be processed.

[0262] For example, please refer to Figure 12 , Figure 12 In the image, A, B, C, D, and E represent the ground truth full-face image, the occluded face, and the image corresponding to the predicted set of facial feature points, respectively. Figure 13 The C-axis includes various feature points), the generated sub-model 430 directly completes the face image based on the full-face feature point set, and the full-face image is obtained by stitching the face image completed based on the full-face feature point set and the face image to be processed.

[0263] For example, please refer to Figure 13 , Figure 13 In the image, A, B, C, and D represent the occluded face image, the full-face image completed based on A and the expression classification label (normal), the full-face image completed based on A and the expression classification label (dissatisfied), and the full-face image completed based on A and the expression classification label (smiling).

[0264] For example, please refer to Figure 14 , Figure 14 In the image, A, B, C, and D represent the occluded face image, the full-face image completed based on A and the expression classification label (smile), the full-face image completed based on A and the expression classification label (normal), and the full-face image completed based on A and the expression classification label (dissatisfied).

[0265] For example, please refer to Figure 15 , Figure 15In the diagram, A, B, C, D, and E represent the ground truth face image, the occluded face image added manually by the user, the image corresponding to the predicted set of face feature points, the face image completed based on the full face feature point set, and the full face image after stitching together the face image to be processed, respectively.

[0266] For example, please refer to Figure 16 , Figure 16 In the diagram, A, B, C, D, and E represent the ground truth face image, the occluded face image added manually by the user, the image corresponding to the predicted set of face feature points, the face image completed based on the full face feature point set, and the full face image after stitching together the face image to be processed, respectively.

[0267] S870, server 220 sends the full-face image to client 211.

[0268] S880, client 211 acquires and displays the full-face image.

[0269] Specifically, client 211 displays a full-face image for the user to view. Client 211 can also save the face image to be processed; the user can click to cancel the completion, and client 211 will then display the face image to be processed. After obtaining the full-face image, the user can also add occlusions again and continue with completion operations.

[0270] For example, by user selection Figure 9 The image of the face to be processed shown, and taking "smile" as an example, server 220 according to... Figure 9 The image of the face to be processed and the "smile" shown are used to obtain the following: Figure 17 The image shown is a full-face image.

[0271] For example, by user selection Figure 10 The original face image shown is used, and the occlusion of the original face image is adjusted. Taking "smiling" as an example, server 220... Figure 10 The image of the face to be processed and the "smile" shown are used to obtain the following: Figure 18 The image shown is a full-face image.

[0272] In one possible embodiment, the models in server 220 are trained by other devices, and server 220 directly uses the trained models to achieve... Figure 8 Facial image processing methods in [the context of the text].

[0273] It should be noted that, Figure 8 The following explanation uses image processing device 100 as an example of server 220. In reality, there may be many types of image processing device 100, as discussed above. In other words, any type of image processing device 100 can achieve this. Figure 8 The content discussed in the text.

[0274] Based on the same inventive concept, this application provides an image processing apparatus, please refer to... Figure 19 The image processing device 1900 is equivalent to being installed in the image processing device 100 described above, and includes a transceiver module 1901 and a processing module 1902, wherein:

[0275] In one possible embodiment, the facial feature point prediction model includes a first encoding module, multiple second encoding modules, and a fully connected module. The processing module 1902 is specifically used for:

[0276] The first encoding module extracts features from the face image to be processed at multiple scales to obtain the first feature map set.

[0277] The first feature map set is processed by convolutional pooling through each of the multiple second encoding modules to obtain the second feature value set;

[0278] By using a fully connected module, multiple sets of second feature values ​​are concatenated to obtain a full-face feature point set.

[0279] In one possible embodiment, the facial feature adjustment information is an expression classification label, and the processing module 1902 is specifically used for:

[0280] Input the expression classification labels and the full-face feature point set into the face adjustment model to obtain the adjusted full-face feature point set;

[0281] The facial adjustment model is based on a set of sample facial expression images, where each sample facial expression image is labeled with a corresponding expression classification label.

[0282] In one possible embodiment, the processing module 1902 is specifically used for:

[0283] Extract facial contour features corresponding to facial feature adjustment information; where facial contour features are used to represent the contour formed by facial key points;

[0284] Based on facial contour features, the coordinates of corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set.

[0285] In one possible embodiment, the image completion model includes a third encoding module, a dilated convolution module, and a decoding module, and the processing module 1902 is specifically used for:

[0286] The adjusted full-face feature point set is converted into a facial feature point map;

[0287] The third encoding module performs convolution processing on the facial feature point map and the face image to be processed at different scales to obtain the third feature map set. The third feature map set includes the convolution-processed feature map set. The third encoding module is the final convolution feature map output during the process of performing convolution processing on the facial feature point map and the face image to be processed at different scales.

[0288] The fourth feature map is obtained by performing dilated convolution on the convolution-processed feature map using the residual unit in the dilated convolution module.

[0289] The fifth feature map is obtained by extracting the fourth feature map set and the local features of the feature map set after convolution processing through the attention unit in the dilated convolution module.

[0290] The decoding module upsamples the fifth feature set and the third feature set to obtain a full-face image with the missing facial areas in the face image to be processed.

[0291] In one possible embodiment, the third feature set further includes a first intermediate feature set and a second intermediate feature set. The first intermediate feature set is an intermediate convolution result obtained by sequentially performing a preset number of convolution processes on the facial feature point map and the facial image to be processed. The second intermediate feature set is an intermediate convolution result obtained by performing a preset number of convolution processes on the first intermediate feature set. The processing module 1902 is specifically used for:

[0292] The fifth feature map set is upsampled by the decoding module to obtain the sixth feature map set;

[0293] The second intermediate feature map set and the sixth feature map set are weighted by the decoding module to obtain the seventh feature map set;

[0294] The seventh feature map set and the first intermediate feature map set are weighted and upsampled by the decoding module to obtain the eighth feature map set;

[0295] The eighth feature set is convolved by the decoding module to obtain a full-face image with the missing facial areas in the face image to be processed.

[0296] In one possible embodiment, the facial landmark prediction model is trained through the following steps:

[0297] Obtain the first sample dataset; wherein, the first sample dataset includes a set of sample face images to be processed, and a set of ground value sample full-face feature points corresponding to each sample face image to be processed;

[0298] A facial feature point prediction model was trained based on the first sample dataset.

[0299] The trained facial feature point prediction model is obtained when the prediction loss of the facial feature point prediction model meets the first preset condition; wherein, the prediction loss is used to represent the loss between the predicted sample full-face feature point set and the ground value sample full-face feature point set.

[0300] In one possible embodiment, the image completion model includes a generative sub-model and a discriminative sub-model, and the loss of the image completion model is obtained by weighting pixel-wise loss, perceptual loss, style loss, total differential loss and adversarial loss.

[0301] Among them, pixel-wise loss is used to represent the pixel value error between the completed sample full-face image and the sample face image to be processed; perceptual loss is used to represent the error between the feature map of the completed sample full-face image and the feature map of the sample face image to be processed; style loss is used to represent the sum of the Gram matrix differences between the feature maps of the completed sample full-face image with occlusion and the sample face image to be processed with occlusion after vectorization of each channel information; total differential loss is used to represent the ratio between the sum of the horizontal and vertical derivatives of the completed sample full-face image and the total number of pixels included in the sample face image to be processed; and generation loss is used to represent the loss of the discriminant sub-model in distinguishing the source of the generation result from the generation sub-model from the true source of the generation result.

[0302] As one example, Figure 19 The image processing device 1900 in the middle can realize Figure 8 The functions of the server 220 can be implemented, or any of the facial image processing methods discussed above can be used.

[0303] Based on the same inventive concept, this application provides a terminal device, please refer to... Figure 20 The terminal device 210 includes a transceiver module 2001 and a display module 2002, wherein:

[0304] The transceiver module 2001 is used to respond to the face image completion input operation on the face image completion interface, send a completion request to the server, so that the server processes the face image to be processed and the face feature adjustment information according to the face image processing method described above, and obtains the full face image after face image completion; wherein, the completion request includes the face image to be processed and the face feature adjustment information; and obtains the full face image returned by the server;

[0305] Display module 2002 is used to display a full-face image.

[0306] It should be noted that the terminal device 210 also includes some other components to assist in realizing the corresponding functions, but this application does not list these other components.

[0307] Based on the same inventive concept, this application provides a facial image processing device. Please continue to refer to... Figure 1 The facial image processing device 100 includes a processor 102 and a memory 103. The processor 102 is used to call program instructions in the memory 103 to implement any of the facial image processing methods discussed above.

[0308] As one example, Figure 1 The processor 102 in the middle can be used to implement Figure 19 The functions of the transceiver module 1901 and the processing module 1902 in the middle.

[0309] Based on the same inventive concept, this application provides a terminal device, please continue to refer to... Figure 21 The terminal device 210 includes a processor 2101 and a memory 2102. The processor 2101 is used to call program instructions in the memory 2102 to implement any of the facial image processing methods discussed above, or to implement... Figure 8 The functions of client 211.

[0310] As one example, Figure 21 The processor 2101 in the middle can be used to implement Figure 20 The functions of the transceiver module 2001 and the display module 2002.

[0311] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the facial image processing method described above.

[0312] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0313] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A facial image processing method, characterized in that, include: Obtain a face image to be processed, the face image to be processed including a portion of the face that is not displayed; The face image to be processed is input into a trained facial feature point prediction model to obtain the full-face feature point set of the face image to be processed. Based on facial feature adjustment information, the corresponding feature points in the full-face feature point set are adjusted to obtain an adjusted full-face feature point set. The facial feature adjustment information is an expression classification label. The process of adjusting the corresponding feature points in the full-face feature point set based on the facial feature adjustment information to obtain the adjusted full-face feature point set includes: inputting the expression classification label and the full-face feature point set into a facial adjustment model to obtain the adjusted full-face feature point set. The facial adjustment model is based on a sample facial expression image set, where each sample facial expression image in the sample facial expression image set is labeled with a corresponding expression classification label. The adjusted set of full-face feature points and the face image to be processed are input into a trained image completion model to obtain a full-face image with the undisplayed areas of the face in the face image to be processed completed.

2. The method as described in claim 1, characterized in that, The facial feature point prediction model includes a first encoding module, multiple second encoding modules, and a fully connected module. The face image to be processed is input into the trained facial feature point prediction model to obtain a full-face feature point set of the face image to be processed, including: The first encoding module extracts features from the face image to be processed at multiple scales to obtain a first feature map set. The first feature map set is subjected to convolutional pooling processing through each of the plurality of second encoding modules to obtain the second feature value set; The fully connected module is used to concatenate multiple sets of second feature values ​​to obtain a full-face feature point set.

3. The method as described in claim 1, characterized in that, Based on facial feature adjustment information, the full-face feature point set is adjusted to obtain an adjusted full-face feature point set, including: Extract the facial contour features corresponding to the facial feature adjustment information; wherein, the facial contour features are used to represent the contour formed by facial key points; Based on the facial contour features, the coordinates of the corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set.

4. The method as described in claim 1, characterized in that, The image completion model includes a third encoding module, a dilated convolution module, and a decoding module. It inputs the adjusted full-face feature point set and the face image to be processed into the trained image completion model to obtain a full-face image with the missing facial areas in the face image to be processed completed, including: The adjusted full-face feature point set is converted into a facial feature point map; The third encoding module performs convolution processing on the facial feature point map and the face image to be processed at different scales to obtain a third feature map set; wherein, the third feature map set includes a convolution-processed feature map set, and the convolution-processed feature map set is the final convolution feature map output by the third encoding module during the process of performing convolution processing on the facial feature point map and the face image to be processed at different scales. The fourth feature map is obtained by performing dilated convolution on the convolution-processed feature map set through the residual unit in the dilated convolution module. The fifth feature map is obtained by extracting the fourth feature map set and the local features of the convolutional feature map set through the attention unit in the dilated convolution module. The decoding module upsamples the fifth feature set and the third feature set to obtain a full-face image with the undisplayed areas of the face in the face image to be processed.

5. The method as described in claim 4, characterized in that, The third feature map set further includes a first intermediate feature map set and a second intermediate feature map set. The first intermediate feature map set is an intermediate convolution result obtained by sequentially performing a preset number of convolution processes on the facial feature point map and the face image to be processed. The second intermediate feature map set is an intermediate convolution result obtained by performing a preset number of convolution processes on the first intermediate feature map set. The decoding module performs upsampling processing on the fifth feature map set and the third feature map set to obtain a full-face image with the undisplayed facial areas in the face image to be processed filled in, including: The decoding module performs upsampling processing on the fifth feature map set to obtain the sixth feature map set; The decoding module performs weighted processing on the second intermediate feature map set and the sixth feature map set to obtain the seventh feature map set. The decoding module performs weighted and upsampling processing on the seventh feature map set and the first intermediate feature map set to obtain the eighth feature map set. The decoding module performs convolution processing on the eighth feature map set to obtain a full-face image with the undisplayed areas of the face in the face image to be processed completed.

6. The method according to any one of claims 1-5, characterized in that, The facial feature point prediction model is trained through the following steps: Obtain a first sample dataset; wherein the first sample dataset includes a set of sample face images to be processed, and a set of ground value sample full-face feature points corresponding to each sample face image to be processed; Based on the first sample dataset, a facial feature point prediction model is trained; The trained facial feature point prediction model is obtained when the prediction loss of the facial feature point prediction model meets the first preset condition; wherein, the prediction loss is used to represent the loss between the predicted sample full-face feature point set and the ground value sample full-face feature point set.

7. The method according to any one of claims 1-6, characterized in that, The image completion model is trained through the following steps: Obtain a second sample dataset; wherein the second sample dataset includes a set of sample face images to be processed, a set of ground value sample full-face feature points corresponding to each sample face image to be processed, and a sample full-face image corresponding to each sample face image in the set of sample face images to be processed. The image completion module is trained based on the second sample dataset; The trained image completion model is obtained when the loss value corresponding to the image completion model meets the second preset condition.

8. The method as described in claim 7, characterized in that, The image completion model includes a generation sub-model and a discriminator sub-model. The loss of the image completion model is obtained by weighting the pixel-wise loss, perceptual loss, style loss, total differential loss, and generation loss. Wherein, the pixel-wise loss is used to represent the pixel value error between the completed sample full-face image and the sample face image to be processed; the perceptual loss is used to represent the error between the feature map of the completed sample full-face image and the feature map of the sample face image to be processed; the style loss is used to represent the sum of the Gram matrix differences between the feature maps of the completed sample full-face image with the occluded region and the sample face image to be processed with the occluded region, after vectorization of each channel information; the total differential loss is used to represent the ratio between the sum of the horizontal and vertical derivatives of the completed sample full-face image and the total number of pixels included in the sample face image to be processed; and the generation loss is used to represent the loss of the discriminant sub-model on the discrimination result of the generation sub-model and the true source of the generation result.

9. A facial image processing method, characterized in that, include: In response to a face image completion input operation on the face image completion interface, a completion request is sent to the server so that the server processes the face image to be processed and the face feature adjustment information according to any one of claims 1-8 to obtain a full-face image after face image completion; wherein, the completion request includes the face image to be processed and the face feature adjustment information; Obtain and display the full-face image returned by the server.

10. A facial image processing device, characterized in that, include: A transceiver module is used to obtain a face image to be processed, the face image to be processed including a portion of the face that is not displayed; The processing module is used to input the face image to be processed into a trained face feature point prediction model to obtain the full face feature point set of the face image to be processed. Furthermore, based on the facial feature adjustment information, the corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set; Furthermore, the adjusted full-face feature point set and the face image to be processed are input into a trained image completion model to obtain a full-face image with the undisplayed facial areas in the face image to be processed completed, wherein: The facial feature adjustment information is an expression classification label. Based on the facial feature adjustment information, the corresponding feature points in the full-face feature point set are adjusted to obtain the adjusted full-face feature point set. This includes: inputting the expression classification label and the full-face feature point set into a facial adjustment model to obtain the adjusted full-face feature point set. The facial adjustment model is based on a sample facial expression image set, where each sample facial expression image in the sample facial expression image set is labeled with a corresponding expression classification label.

11. The apparatus as claimed in claim 10, characterized in that, The facial feature point prediction model includes a first encoding module, multiple second encoding modules, and a fully connected module. The processing module is used for: The first encoding module extracts features from the face image to be processed at multiple scales to obtain a first feature map set. The first feature map set is subjected to convolutional pooling processing through each of the plurality of second encoding modules to obtain the second feature value set; The fully connected module is used to concatenate multiple sets of second feature values ​​to obtain a full-face feature point set.

12. A terminal device, characterized in that, include: A sending module is configured to respond to a face image completion input operation on the face image completion interface, send a completion request to the server, so that the server processes the face image to be processed and the face feature adjustment information according to any one of claims 1-8 to obtain a full-face image after face image completion; wherein, the completion request includes the face image to be processed and the face feature adjustment information; and obtain the full-face image returned by the server; The display module is used to display the full-face image.

13. A facial image processing device, characterized in that, include: At least one processor, and A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which implements the method as described in any one of claims 1-8 or 9 by executing the instructions stored in the memory.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-8 or 9.

Images