Image beautification processing method, device, storage medium and electronic device

Abstract

The present disclosure provides an image beautification processing method, device, storage medium and electronic device, which relate to the field of image and video processing technology. The image beautification processing method comprises: obtaining a face image to be beautified; performing a first beautification processing on the face image to be beautified by a pre-trained deep neural network to obtain an intermediate face image, and determining a mask of the face image to be beautified by the deep neural network; performing a second beautification processing on the intermediate face image based on the mask of the face image to be beautified to obtain a target image. The present disclosure improves the problem of misbeautification processing and enhances the beautification effect.

Description

Background Art

[0002] Beauty refers to the use of image processing technology to beautify portraits in images or videos to better meet the user's aesthetic needs.

[0003] When performing beautification processing, areas that do not need beautification are often mistakenly processed, which has a negative impact on the beautification effect. For example, when smoothing the facial skin, the bangs may also be smoothed, resulting in a blurred image or an uncoordinated image near the bangs.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides an image beautification processing method, an image beautification processing device, a computer-readable storage medium and an electronic device, thereby improving the problem of image beautification misprocessing in the related art at least to a certain extent.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, there is provided an image beautification processing method, comprising: obtaining a facial image to be beautified; performing a first beautification processing on the facial image to be beautified by a pre-trained deep neural network to obtain an intermediate facial image, and determining a mask of the facial image to be beautified by the deep neural network; performing a second beautification processing on the intermediate facial image based on the mask of the facial image to be beautified to obtain a target image.

[0008] According to a second aspect of the present disclosure, there is provided an image beautification processing device, comprising: an image acquisition module, configured to acquire a facial image to be beautified; a first beautification processing module, configured to perform a first beautification processing on the facial image to be beautified through a pre-trained deep neural network to obtain an intermediate facial image, and determine a mask of the facial image to be beautified through the deep neural network; a second beautification processing module, configured to perform a second beautification processing on the intermediate facial image based on the mask of the facial image to be beautified to obtain a target image.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image beautification processing method of the first aspect and possible implementation methods thereof are implemented.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image beautification processing method of the above-mentioned first aspect and its possible implementation methods by executing the executable instructions.

[0011] The technical solution disclosed in this disclosure has the following beneficial effects:

[0012] Based on the above-mentioned image beautification processing method, on the one hand, the second beautification processing is performed based on the mask of the face image to be beautified, so that the second beautification processing accurately acts on the area that needs beautification and avoids other areas, thereby improving the problem of misbeautification processing and improving the beautification effect. On the other hand, the two tasks of the first beautification processing and determining the mask are implemented through a deep neural network, which is equivalent to merging the two tasks, thereby simplifying the algorithm flow, reducing the amount of calculation, and eliminating the need for users to perform manual operations such as cutting out images. On the other hand, the first beautification processing performed by the deep neural network can replace the original multiple fixed algorithm modules, increase the flexibility of image beautification processing, and be applicable to a variety of lighting conditions or skin conditions.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0015] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;

[0016] Figure 2 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0017] Figure 3 A flowchart of an image beautification processing method according to the exemplary embodiment is shown;

[0018] Figure 4 A flowchart of determining a stable bounding box of a face in this exemplary embodiment is shown;

[0019] Figure 5 A schematic diagram showing a method of combining sub-images of a face to be beautified in this exemplary embodiment;

[0020] Figure 6A schematic diagram showing a deep neural network in this exemplary embodiment;

[0021] Figure 7 A flowchart of performing a first beautification process by a deep neural network in this exemplary embodiment is shown;

[0022] Figure 8 A flowchart of training a deep neural network in this exemplary embodiment is shown;

[0023] Fig. 9 A flowchart of a second beautification process in this exemplary embodiment is shown;

[0024] Fig.10 A flowchart showing another second beautification process in this exemplary embodiment is shown;

[0025] Fig.11 A schematic diagram showing a gradual change process of a boundary area in this exemplary embodiment is shown;

[0026] Fig.12 A schematic flowchart of an image beautification processing method in this exemplary embodiment is shown;

[0027] Fig.13 A schematic structural diagram of an image beautification processing device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0029] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0030] In one solution of the related art, the user is required to manually perform a cutout operation to determine the skin area or other area that needs beautification, and then perform beautification processing on the area. This solution increases the user's manual operation, is inconvenient to use, and affects the user experience.

[0031] In view of one or more of the above problems, an exemplary embodiment of the present disclosure provides an image beautification processing method. Figure 1 The system architecture and application scenarios of the operating environment of this exemplary embodiment are exemplarily described.

[0032] Figure 1 The schematic diagram of the system architecture is shown, and the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides image beautification related services in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to exchange data.

[0033] In one embodiment, the terminal 110 can capture or obtain the image or video to be beautified by other means, and upload it to the server 120. For example, the user opens the beautification App (Application) on the terminal 110, selects the image or video to be beautified from the album, and uploads it to the server 120 for beautification, or the user opens the beautification function in the live broadcast App on the terminal 110, and uploads the real-time collected video to the server 120 for beautification. The server 120 executes the above-mentioned image beautification processing method to obtain the beautified image or video, and returns it to the terminal 110.

[0034] In one implementation, the server 120 may train the deep neural network and send the trained deep neural network to the terminal 110 for deployment. For example, the relevant data of the deep neural network may be packaged in the update package of the beauty app or live broadcast app, so that the terminal 110 can obtain the deep neural network and deploy it locally by updating the app. Furthermore, after the terminal 110 shoots or obtains the image or video to be beautified by other means, it may call the deep neural network to implement the image or video beautification processing by executing the above-mentioned image beautification processing method.

[0035] In one implementation, the terminal 110 may perform training on the deep neural network, for example, obtaining the basic architecture of the deep neural network from the server 120 and training it with a local data set, or obtaining a data set from the server 120 and training a locally constructed deep neural network, or training a deep neural network without relying on the server 120. Furthermore, the terminal 110 may call the deep neural network to implement image or video beautification processing by executing the above-mentioned image beautification processing method.

[0036] As can be seen from the above, the execution subject of the image beautification processing method in this exemplary embodiment may be the terminal 110 or the server 120, and the present disclosure does not limit this.

[0037] The exemplary embodiment of the present disclosure further provides an electronic device for executing the above-mentioned deep neural network training method or image beautification processing method, and the electronic device may be the above-mentioned terminal 110 or server 120. Figure 2 Taking the mobile terminal 200 in the example, the structure of the above electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The construction in can also be applied to fixed type equipment.

[0038] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 201, a memory 202, a bus 203, a mobile communication module 204, an antenna 1, a wireless communication module 205, an antenna 2, a display screen 206, a camera module 207, an audio module 208, a power module 209 and a sensor module 210.

[0039] The processor 201 may include one or more processing units, for example: the processor 201 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The deep neural network in this exemplary embodiment can be run on a GPU, a DSP or an NPU. The DSP and the NPU usually run the deep neural network with int data (integer type), and the GPU usually runs the deep neural network with float data (floating point type). In comparison, the power consumption of the DSP and the NPU is lower, the response speed is faster, and the precision is lower. The power consumption of the GPU is higher, the response speed is slower, and the precision is higher. In actual applications, a suitable processing unit can be selected to run the deep neural network according to the performance and actual needs of the mobile terminal 200. For example, when performing real-time beauty on images in a video, due to the high speed requirement, a DSP or NPU can be selected to run the deep neural network.

[0040] The encoder can encode (i.e. compress) the image or video data to form corresponding bitstream data to reduce the bandwidth occupied by data transmission; the decoder can decode (i.e. decompress) the bitstream data of the image or video to restore the image or video data. The mobile terminal 200 can process images or videos in a variety of encoding formats, such as: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats such as MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, and HEVC (High Efficiency Video Coding).

[0041] The processor 201 may be connected to the memory 202 or other components via a bus 203 .

[0042] The memory 202 may be used to store computer executable program codes, which include instructions. The processor 201 executes various functional applications and data processing of the mobile terminal 200 by running the instructions stored in the memory 202. The memory 202 may also store application data, such as images, videos and other files.

[0043] The communication function of the mobile terminal 200 can be realized by the mobile communication module 204, the antenna 1, the wireless communication module 205, the antenna 2, the modulation and demodulation processor, and the baseband processor. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 204 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 200. The wireless communication module 205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 200.

[0044] The display screen 206 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 207 is used to implement shooting functions, such as shooting images or videos to be beautified, etc. The audio module 208 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 209 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 210 may include a depth sensor 2101, a pressure sensor 2102, a gyroscope sensor 2103, an air pressure sensor 2104, etc., to implement corresponding sensing detection functions.

[0045] Combine the following Figure 3 The image beautification processing method in this exemplary embodiment is described. Figure 3 An exemplary process of the image beautification processing method is shown, which may include:

[0046] Step S310, obtaining a face image to be beautified;

[0047] Step S320, performing a first beautification process on the face image to be beautified by using a pre-trained deep neural network to obtain an intermediate face image, and determining a mask of the face image to be beautified by using the deep neural network;

[0048] Step S330 , performing a second beautification process on the intermediate face image based on the mask of the face image to be beautified to obtain a target image.

[0049] Among them, the first beautification processing and the second beautification processing are two stages in the entire beautification process. This exemplary embodiment can set the specific functions of the first beautification processing and the second beautification processing according to actual needs. For example, the first beautification processing can be a universal beautification processing, which refers to a relatively fixed and unified link in the entire beautification processing process, such as blemish removal, which is necessary for most users and does not require personalized settings. The second beautification processing can be a personalized beauty processing, which refers to a link in the entire beautification processing process that varies from person to person and is difficult to uniformly process, such as skin grinding, whitening and brightening, facial features deformation, makeup, etc. For most users, these links are not necessary and require users to perform personalized settings, such as selecting the degree of skin grinding, deformation strength, etc. For universal beautification processing, since its processing process is relatively fixed and unified, this exemplary embodiment can use a deep neural network to implement this part of the processing.

[0050] In one implementation, the first beautification processing may be blemish removal, and the second beautification processing may be other beautification processing other than blemish removal.

[0051] The mask of the face image to be beautified is used to represent the position information of at least one preset area in the face image to be beautified. The preset areas include but are not limited to: facial skin area, facial occlusion area, portrait area (usually refers to non-skin portrait area), hair area, lip area, beard area, and background area. This exemplary embodiment can determine the preset areas according to actual needs. For example, when it is necessary to beautify the facial skin while avoiding facial occlusions, the facial skin area and the facial occlusion area can be used as preset areas. When it is necessary to beautify the lips, the lips area can be used as the preset area. The masks corresponding to the preset areas are obtained through a deep neural network.

[0052] A deep neural network (DNN) is a neural network with a large number of layers. By increasing the number of network layers (i.e., network depth) to reduce the number of parameters, it can learn the deep features of the image and achieve pixel-level processing. The present disclosure does not limit the number of layers of a deep neural network.

[0053] In this exemplary embodiment, the specific function of the first beautification process can be determined according to actual needs, and the deep neural network can be trained to be a network capable of implementing the first beautification process. For example, the first beautification process can be blemish removal, and by obtaining a data set related to blemish removal and training the deep neural network, the deep neural network can implement blemish removal.

[0054] Based on the above-mentioned image beautification processing method, on the one hand, the second beautification processing is performed based on the mask of the face image to be beautified, so that the second beautification processing accurately acts on the area that needs beautification and avoids other areas, thereby improving the problem of misbeautification processing and improving the beautification effect. On the other hand, the two tasks of the first beautification processing and determining the mask are implemented through a deep neural network, which is equivalent to merging the two tasks, thereby simplifying the algorithm flow, reducing the amount of calculation, and eliminating the need for users to perform manual operations such as cutting out images. On the other hand, the first beautification processing performed by the deep neural network can replace the original multiple fixed algorithm modules, increase the flexibility of image beautification processing, and be applicable to a variety of lighting conditions or skin conditions.

[0055] Below Figure 3 Provide detailed instructions for each step.

[0056] refer to Figure 3 In step S310, a facial image to be beautified is obtained.

[0057] The present disclosure does not limit the source of the facial images to be beautified, such as images stored in the device, images currently captured by the device, images obtained from the client, images obtained from other devices, etc.

[0058] In one embodiment, the facial image to be beautified may be one or more frames of a continuous multi-frame image. The continuous multi-frame image may be a video, or a continuously shot image, etc. The continuous multi-frame image is an object that needs to be beautified. Taking a video as an example, it may be a video stream that is currently being shot or received in real time, or it may be a complete video that has been shot or received, such as a locally stored video. The present disclosure does not limit parameters such as the frame rate and image resolution of the video. For example, the video frame rate may be 30fps (frames per second), 60fps, 120fps, etc., and the image resolution may be 720P, 1080P, 4K, etc. and corresponding different aspect ratios. Each frame of the video may be beautified, or a portion of the images may be selected from the video for beautification, and the images that need to be beautified may be used as the facial image to be beautified. For example, when receiving a video stream in real time, each received frame of the image may be used as a facial image to be beautified.

[0059] In one implementation, the above-mentioned obtaining of the face image to be beautified may include the following steps:

[0060] One or more face sub-images to be beautified are extracted from the global image to be beautified, and the face image to be beautified is determined according to the face sub-images to be beautified.

[0061] The global image to be beautified may be a complete original image to be beautified, and may include other parts besides the face, such as the body of a person. The face sub-image to be beautified is a sub-image obtained by intercepting the face part in the global image to be beautified. This exemplary embodiment does not limit the number of faces in the global image to be beautified. For example, when the global image to be beautified includes multiple faces, multiple face sub-images to be beautified may be extracted, and through the processing in subsequent steps, multiple faces may be beautified simultaneously.

[0062] In one implementation, the extracting of one or more face sub-images to be beautified from the global image to be beautified may include the following steps:

[0063] Generate one or more face bounding boxes in the global image to be beautified according to the facial key points identified in the global image to be beautified;

[0064] The bounding box whose area is greater than or equal to the face area threshold is retained, and the image within the bounding box is captured to obtain one or more face sub-images to be beautified.

[0065] Among them, the key points of the face may include the key parts of the face and the points on the edge of the face. A bounding box refers to an area in an image that surrounds the face and has a certain geometric shape. The present disclosure does not limit the shape of the bounding box, such as a rectangle, a trapezoid, or any other shape. The key points of each face are within the bounding box. In one embodiment, the bounding box may be the smallest rectangle that includes the key points of the face.

[0066] Generally, all faces can be detected in the global image to be beautified by the face detection algorithm, which may include faces that do not need beautification (such as faces of passers-by in the distance). Considering that in the scene of image beautification, it is usually necessary to beautify larger faces (smaller faces have less obvious beautification effects, so they usually do not need beautification), the bounding box can be filtered by the face area threshold. The face area threshold can be set based on experience or the size of the global image to be beautified. For example, the face area threshold can be the size of the global image to be beautified * 0.05; if the area of ​​the bounding box is greater than or equal to the face area threshold, it is a face that needs beautification and the bounding box is retained; if the area of ​​the bounding box is less than the face area threshold, it is a face that does not need beautification and the bounding box is deleted.

[0067] After filtering the bounding boxes, the bounding boxes that are retained are the bounding boxes of valid faces. The images within each bounding box are intercepted to obtain the same number of face sub-images to be beautified as the number of bounding boxes.

[0068] In one embodiment, in order to facilitate the subsequent combination of face sub-images to be beautified, an upper limit on the number of face sub-images to be beautified can be set, that is, an upper limit on the number of bounding boxes can be set. For example, it can be set to 4. If the number of bounding boxes retained after filtering by the above-mentioned face area threshold is greater than 4, then 4 bounding boxes can be selected from them, such as the 4 bounding boxes with the largest area, or the 4 bounding boxes closest to the center of the global image to be beautified, and 4 face sub-images to be beautified are correspondingly intercepted, and the faces in other bounding boxes are not beautified; or multiple beautification processes can be performed, in which 4 bounding boxes are selected in this process and the face sub-images to be beautified are intercepted for beautification, and in the next process, other bounding boxes are selected and the face sub-images to be beautified are intercepted for beautification, thereby completing the beautification of faces in all bounding boxes in the global image to be beautified whose areas are greater than the face area threshold.

[0069] In one embodiment, before capturing the image in the bounding box, the bounding box may be expanded so that the bounding box includes a small amount of area other than the face, so as to facilitate gradient processing during subsequent image fusion. When expanding, the bounding box may be expanded in one or more directions according to a preset ratio. For example, the preset ratio is 1.1, and the bounding box is expanded uniformly in all directions so that the size of the expanded bounding box is 1.1 times the original size. It should be noted that when expanding the bounding box, if one or more boundaries of the bounding box reach the boundary of the global image to be beautified, the boundary of the bounding box is stopped at the boundary of the global image to be beautified.

[0070] In the case where the global image to be beautified is one or more frames in a continuous multi-frame image, the face sub-image to be beautified can be extracted based on the information of other frame images in the continuous multi-frame image. In one embodiment, the face in the global image to be beautified is matched with the face in the reference frame image of the global image to be beautified, and a stable bounding box of the face in the global image to be beautified is determined according to the matching result; based on the stable bounding box of the face in the global image to be beautified, the face sub-image to be beautified is extracted from the global image to be beautified.

[0071] The bounding box of the face detected initially is called the basic bounding box, which may be, for example, the minimum rectangle including the key points of the face, or the face frame obtained by a related algorithm. The basic bounding box is optimized, such as expanded, position corrected, etc., and the optimized bounding box is called the stable bounding box.

[0072] In this exemplary embodiment, face detection can be performed on the global image to be beautified to obtain relevant information about the face. The present disclosure does not limit the face detection algorithm. For example, a specific neural network can be used to detect facial key points, including key points of the facial boundary, and a basic bounding box of the face is generated based on the key points of the facial boundary, and a stable bounding box is obtained through optimization.

[0073] The reference frame image can be any frame image in the continuous multiple frames of images for which a stable bounding box of a face has been determined or for which beautification processing has been completed. For example, when performing frame-by-frame beautification processing on a video, the previous frame image of the global image to be beautified can be used as the reference frame image. By matching the face in the global image to be beautified with the reference frame image, the stable bounding box of the face in the global image to be beautified can be determined based on the stable bounding box of the face in the reference frame image.

[0074] In one embodiment, reference Figure 4 As shown, the matching of the face in the global image to be beautified with the face in the reference frame image of the global image to be beautified and determining the stable bounding box of the face in the global image to be beautified according to the matching result may include the following steps S410 to S430:

[0075] Step S410 , detecting a human face in the global image to be beautified, recording it as a to-be determined human face, and matching the to-be determined human face with a determined human face in a reference frame image of the global image to be beautified.

[0076] Among them, the face to be determined refers to a face that needs to be beautified but has not had a stable bounding box determined, which can be regarded as a face with an unknown identity. The determined face refers to a face with a stable bounding box determined, which can be regarded as a face with a known identity. The faces with determined stable bounding boxes in the reference frame image are all determined faces. Correspondingly, the faces detected in the global image to be beautified are faces with no stable bounding boxes determined, that is, faces to be determined. By matching the faces to be determined in the global image to be beautified with the determined faces in the reference frame image, it can be inferred that there is a correlation between the stable bounding box of the face to be determined and the stable bounding box of the determined face that matches the face to be determined, and the stable bounding box of the face to be determined can be determined accordingly.

[0077] In one implementation, a face area threshold may be set based on experience or the size of the global image to be beautified; if the area of ​​a basic bounding box of the face is greater than or equal to the face area threshold, it is a face that needs beautification, and information such as the basic bounding box of the face may be retained, or the face may be recorded as a face to be determined; if the area of ​​the basic bounding box of the face is less than the face area threshold, it is a face that does not need beautification, and related information such as the basic bounding box of the face may be deleted without subsequent processing.

[0078] In one embodiment, in order to facilitate the subsequent processing of the face sub-images to be beautified, such as combining the face sub-images to be beautified, or considering the limitation of device performance, an upper limit on the number of face sub-images to be beautified can be set, that is, an upper limit on the number of faces to be determined can be set. For example, it can be set to 4. If the number of retained faces is greater than 4 after filtering by the above face area threshold, 4 faces to be determined can be further screened out, such as the 4 faces with the largest area, or the 4 faces closest to the center of the global image to be beautified. In this way, 4 face sub-images to be beautified are intercepted in the subsequent processing, and no subsequent processing is performed on other faces. Alternatively, multiple beautification processes can be performed. In this process, 4 faces are selected as faces to be determined, and their corresponding face sub-images to be beautified are intercepted and beautified. In the next process, other faces are selected as faces to be determined, and their corresponding face sub-images to be beautified are intercepted and beautified. In this way, all faces in the basic bounding box with an area greater than the face area threshold in the global image to be beautified are beautified.

[0079] In one implementation, in order to facilitate tracking and identifying faces in multiple consecutive frames of images, an ID (Identity Document) may be assigned to each face. For example, starting from the first frame, an ID is assigned to each face; after a face is detected in each subsequent frame, each face is matched with the face in the previous frame; if the match is successful, the face ID and other related information in the previous frame are inherited; if the match is unsuccessful, it is treated as a new face and a new ID is assigned.

[0080] The present disclosure does not limit the method of matching the face to be determined with the determined face. For example, a face recognition algorithm can be used to identify and compare each face to be determined with each determined face. If the similarity is higher than a preset similarity threshold, it is determined that the face to be determined and the determined face are matched successfully.

[0081] In one embodiment, whether the face to be determined matches the determined face successfully can be determined based on the overlap (Intersection Over Union, IOU, also known as intersection over union ratio) between the basic bounding box of the face to be determined and the basic bounding box of the determined face. An exemplary method for calculating the overlap is provided below:

[0082] Get the position of the basic bounding box of the face to be determined in the global image to be beautified, and the position of the basic bounding box of the determined face in the reference frame image, count the number of pixels in the two basic bounding boxes that overlap, denoted as k1, and the number of pixels that do not overlap, denoted as k2 (indicating the number of pixels in the basic bounding box of the face to be determined that do not overlap with the basic bounding box of the determined face) and k3 (indicating the number of pixels in the basic bounding box of the determined face that do not overlap with the basic bounding box of the undetermined face), then the overlap degree of the two basic bounding boxes is:

[0083]

[0084] After determining the overlap, if the overlap reaches a preset overlap threshold, it is determined that the to-be-determined face successfully matches the determined face. The overlap threshold can be set based on experience and actual needs, for example, can be set to 0.75.

[0085] In addition, the basic bounding box of the face to be determined and any one of the basic bounding boxes of the determined face can be iteratively transformed using an ICP (Iterative Closest Point) algorithm, and the overlap of the two basic bounding boxes can be calculated based on the number of pixels with the same pixel values ​​and the number of pixels with different pixel values ​​in the basic bounding box of the face to be determined and the basic bounding box of the determined face after the final transformation, thereby determining whether the match is successful.

[0086] It should be noted that since there may be multiple faces to be determined in the global image to be beautified and multiple determined faces in the reference frame image, each face to be determined can be matched with each determined face separately to obtain a similarity matrix or an overlap matrix, and then the Hungarian algorithm can be used to achieve the global maximum match, and then whether the match is successful is determined based on the similarity or overlap between each face to be determined and the determined face.

[0087] Step S420: If the face to be determined does not match the determined face, the basic bounding box of the face to be determined is expanded according to the first preset parameter to obtain a stable bounding box of the face to be determined;

[0088] If the face to be determined is not matched with the determined face, it means that the face to be determined is a new face in the continuous multi-frame image, and reference information cannot be obtained from the reference frame image. Therefore, the basic bounding box of the face to be determined can be appropriately expanded to obtain a stable bounding box. The first preset parameter is the expansion parameter of the basic bounding box for the new face, which can be determined according to experience or actual needs. For example, the width and height of the basic bounding box can be expanded by 1 / 4.

[0089] Suppose the basic bounding box of the face to be determined is represented as [bb0, bb1, bb2, bb3], where bb0 is the abscissa of the upper left point of the basic bounding box, bb1 is the ordinate of the upper left point of the basic bounding box, bb2 is the abscissa of the lower right point of the basic bounding box, and bb3 is the ordinate of the lower right point of the basic bounding box. The width of the basic bounding box is w and the height is h. Note that the pixel coordinates in the image usually have the upper left point of the image as (0, 0) and the lower right point as (Wmax, Hmax), where Wmax and Hmax represent the width and height of the image. Therefore, bb0 < bb2 and bb1 < bb3. Let E1 represent the first preset parameter. When the basic bounding box is centered and expanded (i.e., uniformly expanded up, down, left, and right) according to the first preset parameter, the size of the stable bounding box can be obtained as follows:

[0090]

[0091] where expand_w and expand_h are the width and height of the stable bounding box of the face to be determined, respectively. It should be noted that if the expanded width expand_w exceeds the width Wmax of the global image to be beautified, then expand_w = Wmax; if the expanded height expand_h exceeds the height Hmax of the global image to be beautified, then expand_h = Hmax.

[0092] The center point coordinates of the stable bounding box are equal to the center point coordinates of the basic bounding box, that is:

[0093]

[0094] where center_x represents the x coordinate of the center point of the stable bounding box of the face to be determined, and center_y represents the y coordinate of the center point of the stable bounding box of the face to be determined.

[0095] Then the coordinates of the upper left point and the lower right point of the stable bounding box can be calculated as follows:

[0096]

[0097] where expand_bb0 is the abscissa of the upper left point of the stable bounding box, expand_bb1 is the ordinate of the upper left point of the stable bounding box, expand_bb2 is the abscissa of the lower right point of the stable bounding box, and expand_bb3 is the ordinate of the lower right point of the stable bounding box. Thus, the stable bounding box of the face to be determined is obtained. If the calculated coordinates exceed the boundary of the global image to be beautified, the boundary coordinates of the global image to be beautified are used to replace the coordinates that exceed the boundary. Finally, the expanded bounding box can be represented in the form of [expand_bb0, expand_bb1, expand_bb2, expand_bb3].

[0098] It should be added that the above coordinates are usually pixel coordinates in the image, which are integers. Therefore, when calculating, float data can be used for calculation, and then rounded, and the result is saved as int data. Exemplarily, when division operations are involved, float data is used for calculation and intermediate results are cached. When calculating the final result (including the above expand_w, expand_h, center_x, center_y, expand_bb0, expand_bb1, expand_bb2, expand_bb3), rounding is performed and saved as int data.

[0099] For the center point coordinates, since saving int data will affect the accuracy of subsequent processing of other frames, int and float data can be saved. For example, the result calculated in formula (3) is saved as float data, as shown below:

[0100]

[0101] Among them, center_x_float and center_y_float represent the center point coordinates stored in float type data, center_x and center_y represent the center point coordinates stored in int type data, and int() represents rounding operation.

[0102] Furthermore, to ensure the accuracy of the results, formula (4) can be changed to the following calculation method:

[0103]

[0104] Step S430: If the face to be determined matches the determined face successfully, a stable bounding box of the face to be determined is determined based on the stable bounding box of the determined face.

[0105] Generally, the face to be determined in the global image to be beautified will not change much compared with the determined face in the matching reference frame image, which is reflected in that both the position change and the size change will not be too large. Therefore, based on the stable bounding box of the determined face, appropriate position and size changes can be made to obtain a stable bounding box of the face to be determined.

[0106] In one implementation, the stable bounding box of the determined face can be changed in position and size according to the position change parameters and size change parameters of the basic bounding box of the face to be determined relative to the basic bounding box of the determined face, to obtain the stable bounding box of the face to be determined.

[0107] In one implementation, the above-mentioned determining the stable bounding box of the face to be determined based on the stable bounding box of the determined face may include the following steps:

[0108] Based on a preset stability coefficient, the center point coordinates of the stable bounding box of the determined face and the center point coordinates of the basic bounding box of the face to be determined are weighted to obtain the center point coordinates of the stable bounding box of the face to be determined.

[0109] The above steps represent the fusion of the position of the stable bounding box of the determined face and the position of the basic bounding box of the face to be determined as the position of the stable bounding box of the face to be determined. During the fusion, the center point coordinates of the two are weighted using a preset stability coefficient. The preset stability coefficient can be the weight of the stable bounding box of the determined face, which can be determined based on experience or actual scenarios. In general, the faster the face moves, the smaller the preset stability coefficient. For example, in a live broadcast scenario, the face usually moves very little within a certain range. The preset stability coefficient can be set to 0.9, and the center point coordinates of the stable bounding box of the face to be determined are calculated as follows:

[0110]

[0111] Where pre_center_x represents the x-coordinate of the center point of the stable bounding box of the determined face, and pre_center_y represents the y-coordinate of the center point of the stable bounding box of the determined face. It can be seen that formula (7) represents the weight of the center point coordinate of the stable bounding box of the determined face as 0.9 and the weight of the center point coordinate of the basic bounding box of the face to be determined as 0.1, and the two center point coordinates are weighted to obtain the center point coordinate of the stable bounding box of the face to be determined.

[0112] Similar to the above formula (5), the coordinates of the data center points of int type and float type can be saved, then:

[0113]

[0114] Among them, pre_center_x_float is the saved float data of pre_center_x, and pre_center_y_float is the saved float data of pre_center_y.

[0115] The above-mentioned weighted method for calculating the center point coordinates essentially adopts a mechanism for momentum updating of the center point coordinates, which can avoid excessive movement of the center point coordinates of the stable bounding box of the same face from the reference frame image to the global image to be beautified, so as to cause jitter in the subsequently captured sub-image of the face to be beautified, thereby affecting the beautification effect.

[0116] In one embodiment, determining the stable bounding box of the face to be determined based on the stable bounding box of the determined face may include the following steps:

[0117] If the size of the basic bounding box of the face to be determined is greater than the product of the size of the stable bounding box of the determined face and the first magnification factor, expand the size of the stable bounding box of the determined face according to the second preset parameter to obtain the size of the stable bounding box of the face to be determined;

[0118] If the size of the basic bounding box of the face to be determined is less than the product of the size of the stable bounding box of the determined face and the second magnification factor, reduce the size of the stable bounding box of the determined face according to the third preset parameter to obtain the size of the stable bounding box of the face to be determined; the first magnification factor is greater than the second magnification factor;

[0119] If the size of the basic bounding box of the face to be determined is less than the product of the size of the stable bounding box of the determined face and the first magnification factor and greater than the product of the size of the stable bounding box of the determined face and the second magnification factor, take the size of the stable bounding box of the determined face as the size of the stable bounding box of the face to be determined.

[0120] The above steps represent calculations in three cases respectively according to the comparison result between the size of the basic bounding box of the face to be determined and the size of the stable bounding box of the determined face. The first magnification factor and the second magnification factor can be integer magnification factors or non-integer magnification factors. In one embodiment, the first magnification factor is greater than or equal to 1 and the second magnification factor is less than 1. Exemplarily, the first magnification factor can be 1 and the second magnification factor can be 0.64.

[0121] When performing calculations, the width and height can be compared and calculated separately. For example, if the comparison result of the width belongs to the first case above and the comparison result of the height belongs to the second case, calculate the width and height of the stable bounding box of the face to be determined in the two cases respectively.

[0122] Assume that the first magnification factor is t1 and the second magnification factor is t2. The calculation of the width is described as follows:

[0123] In the first case, if w > pre_expand_w·t1, and E2 represents the second preset parameter, then:

[0124] expand_w = pre_expand_w + pre_expand_w·E2 (9)

[0125] In the second case, if w < pre_expand_w·t2, and E3 represents the third preset parameter, then:

[0126] expand_w = pre_expand_w - pre_expand_w·E3 (10)

[0127] In the third case, if pre_expand_w·t2 < w < pre_expand_w·t1, then:

[0128] expand_w = pre_expand_w (11)

[0129] For the height, it can also be calculated according to the above three cases respectively to obtain expand_h.

[0130] Generally, in a series of consecutive video frames, as long as the face does not approach the camera rapidly, move away from the camera rapidly, or move out of the frame, the size of the face will not change drastically, and thus the third case mentioned above is satisfied. At this time, the size of the stable bounding box of the face to be determined is made equal to the size of the stable bounding box of the face that has been determined, that is, the size of the stable bounding box remains unchanged. The first and second cases mentioned above are both cases where the size of the face changes drastically. In the first case, the face expands drastically. At this time, the size of the stable bounding box of the face that has been determined is appropriately expanded according to the second preset parameter to obtain the size of the stable bounding box of the face to be determined. The second preset parameter can be determined according to experience and the actual scenario. In the second case, the face shrinks drastically. At this time, the size of the stable bounding box of the face that has been determined is appropriately reduced according to the third preset parameter to obtain the size of the stable bounding box of the face to be determined. The third preset parameter can be determined according to experience and the actual scenario.

[0131] If the expanded width expand_w exceeds the width Wmax of the global image to be beautified, then expand_w = Wmax; if the expanded height expand_h exceeds the height Hmax of the global image to be beautified, then expand_h = Hmax.

[0132] Through the calculations of the above three cases, it is possible to avoid excessive changes in the size of the stable bounding box of the same face from the reference frame image to the global image to be beautified, so as to prevent the sub-image of the face to be beautified intercepted subsequently from jittering and affecting the beautification effect.

[0133] After obtaining the center point coordinates and size of the stable bounding box of the face to be determined respectively, the coordinates of the upper left point and the lower right point of the stable bounding box can be calculated. If the calculated coordinates exceed the boundaries of the global image to be beautified, the boundary coordinates of the global image to be beautified are used to replace the coordinates that exceed the boundaries. Finally, the stable bounding box can be represented in the form of [expand_bb0, expand_bb1, expand_bb2, expand_bb3].

[0134] From the above, it can be seen that when the face to be determined is successfully matched with the determined face, the stable bounding box of the face to be determined is determined according to the stable bounding box of the determined face, so that the face to be determined inherits the information of the stable bounding box of the determined face to a certain extent, thereby ensuring that the stable bounding box of the face between different frame images has a certain continuity and stability, and will not undergo drastic changes in position or size, thereby ensuring the consistency of the face beauty effect during subsequent beauty processing, and preventing the face after beauty from flickering due to drastic changes in the face.

[0135] In one implementation, after obtaining the stable bounding box of the face to be determined, the relevant parameters of the stable bounding box can be saved, and the face to be determined can be marked as a determined face for use in matching the face to be determined and determining the stable bounding box in subsequent frames.

[0136] After obtaining the stable bounding box of the face in the global image to be beautified, the image in the stable bounding box can be intercepted to obtain a sub-image of the face to be beautified. When the global image to be beautified includes stable bounding boxes of multiple faces, a sub-image of the face to be beautified corresponding to each face can be intercepted.

[0137] In one embodiment, when extracting one or more face sub-images to be beautified from the global image to be beautified, cropping information may be recorded. The cropping information includes the position information of each face sub-image to be beautified in the global image to be beautified. This facilitates filling the face sub-image back into the global image to be beautified after the face is beautified.

[0138] After obtaining the face sub-images to be beautified, the face image to be beautified can be determined according to the face sub-images to be beautified. For example, each face sub-image to be beautified can be used as the face image to be beautified.

[0139] In one implementation, the determining of the face image to be beautified based on the face sub-image to be beautified may include the following steps:

[0140] The face sub-image to be beautified is adjusted based on the input image size of the deep neural network to generate the face image to be beautified, and the adjustment process includes transformation and / or combination.

[0141] The input image size is the image size that matches the input layer of the deep neural network. Transformation includes processing such as rotation and scaling, and combination refers to combining multiple sub-images of the face to be beautified into one image. Transformation and combination are collectively referred to as adjustment processing.

[0142] In this exemplary embodiment, a single face sub-image to be beautified can be used as the face image to be beautified, or a single face sub-image to be beautified can be transformed and used as the face image to be beautified, or multiple face images to be beautified can be combined into one face image to be beautified, or multiple face images to be beautified can be transformed and combined into one face image to be beautified. Exemplarily, for multiple face sub-images to be beautified extracted from the global image to be beautified, they can be transformed and combined into one face image to be beautified, or the larger face sub-images to be beautified (such as those with an area larger than a set sub-image threshold) can be used as the face image to be beautified, and the smaller face sub-images to be beautified (such as those with an area smaller than a set sub-image threshold) can be combined into one face image to be beautified.

[0143] This exemplary embodiment does not limit the size and aspect ratio of the input image. For example, the ratio of the long side to the short side of the input image can be set to be close to In one embodiment, the deep neural network can be a fully convolutional network, which can process images of different sizes. In this case, the deep neural network has no requirements for the input image size, and the size has an impact on the amount of calculation, memory usage, and beauty refinement. The input image size can be determined according to the beauty refinement set by the user or the performance of the device. As a result, the deep neural network can be deployed on devices with different performances such as high, medium, and low, and has a wide range of applications. There is no need to deploy different deep neural networks for different devices, which reduces the training cost of the network. For example, considering that it is suitable for lightweight computing on a mobile terminal, the input image size can be determined as a smaller value, such as 640 in width and 448 in height.

[0144] When combining the face sub-images to be beautified into the face image to be beautified, the specific combination method is related to the number of the face sub-images to be beautified. In one embodiment, the above-mentioned adjustment of the face sub-images to be beautified based on the input image size of the deep neural network to generate the face image to be beautified may further include the following steps:

[0145] According to the number of face sub-images to be beautified, the input image size is divided into sub-image sizes corresponding to the face sub-images to be beautified;

[0146] Transforming the corresponding face sub-image to be beautified based on the size of each sub-image;

[0147] The transformed face sub-images to be beautified are combined to generate a face image to be beautified.

[0148] Combine the following Figure 5 Give an example. Figure 5 Q in the figure represents the number of face sub-images to be beautified. Figure 5The exemplary modes of input image size segmentation and image combination when Q is 1 to 4 are shown respectively. Assuming that the input image size is 640*448 in width and Q is 1, the sub-image size is also 640*448 in width; when Q is 2, the sub-image size is half of the input image size, that is, 320*448 in width; when Q is 3, the sub-image sizes are 0.5, 0.25, and 0.25 of the input image size, that is, 320*448 in width, 320*224 in height, and 320*224 in width; when Q is 4, the sub-image sizes are all 0.25 of the input image size, that is, 320*224 in width. Each face sub-image to be beautified is transformed to be consistent with the sub-image size. It should be noted that when the sub-image sizes are inconsistent, such as when Q is 3, the face sub-images to be beautified can be matched with the sub-image sizes one by one according to the size order of the face sub-images to be beautified and the size order of the sub-images, that is, the largest face sub-image to be beautified corresponds to the largest sub-image size, and the smallest face sub-image to be beautified corresponds to the smallest sub-image size. After the face sub-images to be beautified are transformed, the transformed face sub-images to be beautified are matched according to Figure 5 The above steps are combined in the manner shown to generate a face image to be beautified.

[0149] In one embodiment, when Q is an even number, the input image size can be divided into Q equal parts to obtain Q sub-images of the same size. Specifically, Q can be decomposed into the product of two factors, that is, Q=q 1 *q 2 , so that q 1 / q 2 The ratio of the input image size to the aspect ratio (such as ) as close as possible, multiply the width of the input image size by q 1 Equally divided, highly q 2 When Q is an odd number, the input image size is divided into Q+1 equal parts to obtain Q+1 sub-image sizes of the same size, two of which are merged into one sub-image size, and the remaining Q-1 sub-image sizes remain unchanged, thereby obtaining Q sub-image sizes.

[0150] In one implementation, the size ratio (or area ratio) of the face sub-image to be beautified may be calculated first, for example, S 1 :S 2 :S 3 :…:S Q , and then divide the input image size into Q sub-image sizes according to this ratio.

[0151] After determining the sub-image size corresponding to each face sub-image to be beautified, the face sub-image to be beautified may be transformed based on the sub-image size. In one embodiment, transforming the corresponding face sub-image to be beautified based on each sub-image size may include any one or more of the following:

[0152] ① When the size relationship between the width and height of the face sub-image to be beautified is different from the size relationship between the width and height of the sub-image size, rotate the face sub-image to be beautified 90 degrees. That is to say, in the face sub-image to be beautified and the sub-image size, the width is greater than the height or the width is less than the height, then the size relationship between the width and height of the face sub-image to be beautified and the sub-image size is the same, and there is no need to rotate the face sub-image to be beautified; otherwise, the size relationship between the width and height of the face sub-image to be beautified and the sub-image size is different, and the face sub-image to be beautified needs to be rotated 90 degrees (either clockwise or counterclockwise). For example, when the sub-image size is 320 in width and 448 in height, that is, the width is less than the height, if the face sub-image to be beautified is wider than the height, then rotate the face sub-image to be beautified 90 degrees.

[0153] In one implementation, in order to keep the angle of the human face in the face sub-image to be beautified, the face sub-image to be beautified may not be rotated.

[0154] ② When the size of the face sub-image to be beautified is larger than the sub-image size, the face sub-image to be beautified is downsampled according to the sub-image size. The size of the face sub-image to be beautified is larger than the sub-image size, which means that the width of the face sub-image to be beautified is larger than the width of the sub-image size, or the height of the face sub-image to be beautified is larger than the height of the sub-image size. In the image beautification scenario, the global image to be beautified is generally a clear image taken by the terminal device, and its size is large. Therefore, it is a common situation that the size of the face sub-image to be beautified is larger than the sub-image size, that is, it is usually necessary to downsample the face sub-image to be beautified.

[0155] Down sampling can be achieved by using methods such as bilinear interpolation and nearest neighbor interpolation, which is not limited in the present disclosure.

[0156] After downsampling, at least one of the width and height of the face sub-image to be beautified is aligned with the sub-image size, including the following situations:

[0157] The width and height of the face sub-image to be beautified are the same as the sub-image size;

[0158] The width of the face sub-image to be beautified is the same as the width of the sub-image size, and the height is smaller than the height of the sub-image size;

[0159] The height of the face sub-image to be beautified is the same as the height of the sub-image size, and the width is smaller than the width of the sub-image size.

[0160] It should be noted that if the above-mentioned rotation has been performed on the sub-image of the face to be beautified to obtain a rotated sub-image of the face to be beautified, then when the size of the sub-image of the face to be beautified is larger than the sub-image size, it is downsampled according to the sub-image size. The specific implementation method is the same as the downsampling method of the sub-image of the face to be beautified, so it will not be repeated here.

[0161] On the contrary, when the size of the face sub-image to be beautified (or the rotated face sub-image to be beautified) is smaller than or equal to the sub-image size, the downsampling processing step may not be performed.

[0162] ③ When the size of the face sub-image to be beautified is smaller than the sub-image size, the face sub-image to be beautified is padded according to the difference between the size of the face sub-image to be beautified and the sub-image size, so that the size of the padded face sub-image to be beautified is equal to the sub-image size. The size of the face sub-image to be beautified is smaller than the sub-image size, which means that at least one of the width and height of the face sub-image to be beautified is smaller than the sub-image size, and the other is not larger than the sub-image size, which specifically includes the following situations:

[0163] The width of the face sub-image to be beautified is smaller than the width of the sub-image size, and the height is also smaller than the height of the sub-image size;

[0164] The width of the face sub-image to be beautified is smaller than the width of the sub-image size, and the height is equal to the height of the sub-image size;

[0165] The height of the face sub-image to be beautified is smaller than the height of the sub-image size, and the width is equal to the height of the sub-image size.

[0166] Preset pixel values ​​can be used for filling, which are usually pixel values ​​that are very different from the color of the face, such as (R0, G0, B0), (R255, G255, B255), etc.

[0167] Generally, the sub-image of the face to be beautified can be filled around. For example, the center of the sub-image of the face to be beautified is overlapped with the center of the sub-image size, and the difference part around the sub-image of the face to be beautified is filled, so that the size of the sub-image of the face to be beautified after filling is consistent with the size of the sub-image. Of course, the sub-image of the face to be beautified can also be aligned with one side edge of the sub-image size, and the other side is filled. This disclosure does not limit this.

[0168] It should be noted that if the sub-image of the face to be beautified has been subjected to at least one of the above-mentioned rotation and downsampling processing to obtain a sub-image of the face to be beautified that has been subjected to at least one of the rotation and downsampling processing, then when the size of the sub-image of the face to be beautified is smaller than the sub-image size, it is filled according to the difference between the size of the sub-image and the sub-image size. The specific implementation method is the same as the filling method of the sub-image of the face to be beautified mentioned above, and therefore it will not be repeated here.

[0169] The above ① to ③ are three commonly used transformation methods, and any one or more of them can be used according to actual needs. For example, ①, ②, and ③ are used to process each face sub-image to be beautified, and the processed face sub-images to be beautified are combined into a face image to be beautified.

[0170] In one embodiment, when the size of the input image to be beautified is adjusted based on the above-mentioned deep neural network, the adjustment processing information can be recorded. The adjustment processing information may include transformation information and / or combination information. The transformation information may include: the direction and angle of rotation of each sub-image of the face to be beautified, the ratio of downsampling, the coordinates of the filled pixels, etc. The combination information may include: the size of each sub-image of the face to be beautified, and the position information in the image of the face to be beautified, the arrangement and order of the sub-images of the face to be beautified, etc. In this way, it is convenient to restore the original direction, size and position of each face after the face image to be beautified is beautified.

[0171] The above explains how to obtain the face image to be beautified. Figure 3 In step S320, the first beautification processing is performed on the face image to be beautified by a pre-trained deep neural network to obtain an intermediate face image, and the mask of the face image to be beautified is determined by the deep neural network.

[0172] The intermediate facial image is an image obtained after the facial image to be beautified has been subjected to the first beautification processing. For example, the intermediate facial image may be a blemish-removed image corresponding to the facial image to be beautified.

[0173] The present disclosure does not limit the specific form of the mask, for example, it can be a single-channel or multi-channel mask image.

[0174] In one embodiment, the deep neural network can identify k preset areas in the face image to be beautified, where k is a positive integer not less than 2, and outputs a mask image of k channels accordingly. Taking k as 4 as an example, the deep neural network can identify 4 preset areas in the face image to be beautified, such as the background area, the portrait area, the facial skin area, and the facial occlusion area, wherein the background area can be other areas except the portrait area, the facial skin area, and the facial occlusion area. The deep neural network can output a 4-channel mask image, which can be recorded as a background mask image, a portrait mask image, a facial skin mask image, and a facial occlusion mask image. Taking the facial skin mask image as an example, its size can be the same as that of the face image to be beautified, where the value of each pixel is used to quantify the probability that the pixel belongs to the facial skin area. For example, the value range of the value can be [0, 255], which is unified with the value range of the pixel value of the face image to be beautified or the intermediate face image. In this case, the value can be regarded as the probability that the pixel belongs to the facial skin area quantized to a value within the interval [0, 255] (for example, the probability is multiplied by 255 to obtain the value), or the value range of the value can be [0, 1], in which case the value is equivalent to the probability. For mask images of other preset areas, the meaning of the pixel values ​​is similar. Generally, the sum of the probabilities that each pixel belongs to the above four preset areas is 1. The pixel in the mask image of which channel has the largest value indicates that the pixel belongs to the preset area category corresponding to the channel.

[0175] In one embodiment, a deep neural network can identify k preset areas in a facial image to be beautified, where k is a positive integer not less than 2, and output a single-channel mask image accordingly, in which the value of each pixel in the mask image indicates which preset area it belongs to. Taking k as 4 as an example, a deep neural network can identify four preset areas in a facial image to be beautified, such as a background area, a portrait area, a facial skin area, and a facial occlusion area, wherein the background area may be an area other than the portrait area, the facial skin area, and the facial occlusion area. A deep neural network can output a single-channel mask image, in which the value of each pixel is 0, 1, 2, or 3, where a value of 0 indicates that the pixel belongs to the background area, a value of 1 indicates that the pixel belongs to the portrait area, a value of 2 indicates that the pixel belongs to the facial skin area, and a value of 3 indicates that the pixel belongs to the facial occlusion area.

[0176] The deep neural network can simultaneously implement the first beautification processing of the beautified face image and determine the mask of the face image to be beautified. When the first beautification processing is performed on the beautified face image, it is necessary to determine the position information of certain preset face areas, so the first beautification processing and the determination of the mask are highly correlated.

[0177] The present disclosure does not limit the specific structure of the deep neural network. Exemplarily, the deep neural network may include a backbone network, and a first branch network and a second branch network respectively connected to the backbone network, the first branch network is used to perform a first beauty processing, the second branch network is used to determine the image mask, and the backbone network can be regarded as a common processing part of the first beauty processing and determining the image mask, such as being used to extract image features, the first branch network outputs the image after the first beauty processing, and the second branch network outputs the mask. Alternatively, the deep neural network may not have multiple branches, and it may have C1 input channels, such as C1 is 3, and the face image to be beautified in RGB format can be input into the deep neural network; the deep neural network may have C1+C2 input channels, where C1 represents the number of channels of the image after the first beauty processing, and C2 represents the number of channels of the mask, such as C1 is 3 and C2 is 4, then the deep neural network can output an intermediate face image in RGB format, and output a 4-channel mask image.

[0178] Figure 6 An exemplary structure of a deep neural network is shown. Figure 6 As shown, the deep neural network may include: a first pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a second pixel rearrangement layer.

[0179] based on Figure 6 The deep neural network shown in Figure 7 As shown, step S320 can be performed by Figure 7 Steps S710 to S740 in the embodiment are implemented as follows:

[0180] Step S710: Perform a first pixel rearrangement process on the face image to be beautified by using a first pixel rearrangement layer to obtain a first feature image.

[0181] The facial image to be beautified may be an image with any number of channels, for example, a single-channel image (such as a grayscale image) or a multi-channel image (such as an RGB image). The first pixel rearrangement layer is used to perform a first pixel rearrangement process, and the first pixel rearrangement process may be a rearrangement that increases the number of channels, for example, each channel of the facial image to be beautified may be rearranged into multiple channels.

[0182] In one implementation, step S710 may include:

[0183] Input the face image to be beautified with the number of channels C1 into the first pixel rearrangement layer;

[0184] The pixels in each m*m neighborhood in each channel of the facial image to be beautified are rearranged to the same positions in the m*m channels, and the first feature image with a channel number of C1*m*m is output.

[0185] Wherein, C1 represents the number of channels of the face image to be beautified (i.e., the number of input channels of the deep neural network), which is a positive integer, and m represents the parameter of the first pixel rearrangement processing, which is a positive integer not less than 2. Taking m=2 as an example, traverse the first channel of the face image to be beautified, usually starting from the upper left corner, extract the pixels of each 2*2 grid, and rearrange them to the same position in the four channels, thereby turning one channel into four channels, and at the same time reducing the width and height of the image to half, and record the rearranged image as the first feature image; the same method is used to process other channels. If the face image to be beautified is a single-channel image, the first feature image of the four channels is obtained after the pixels are rearranged; if the face image to be beautified is a three-channel image, the first feature image of the twelve channels is obtained after the pixels are rearranged.

[0186] The first pixel rearrangement layer can be implemented by using the space_to_depth function in TensorFlow (a machine learning implementation framework) to convert the spatial features in the face image to be beautified into depth features. It can also be implemented by using a convolution operation with a step size of m. In this case, the first pixel rearrangement layer can be regarded as a special convolution layer.

[0187] Step S720: Perform convolution processing on the first feature image using a convolution layer to obtain a second feature image.

[0188] The present disclosure does not limit the number of convolutional layers, the size of convolutional kernels, the specific structure of convolutional layers, etc. The convolutional layer is used to extract image features from different scales and learn depth information. The convolutional layer may include a matching pooling layer for downsampling the convolutional image to achieve information abstraction, increase the receptive field, and reduce parameter complexity.

[0189] When multiple convolutional layers are set, a step-by-step convolution and downsampling method can be used. For example, the image can be reduced by a factor of 2 until the last convolutional layer outputs a second feature image. The second feature image can be the feature image with the smallest size in the deep neural network processing process.

[0190] Step S730: Use a transposed convolution layer to perform a transposed convolution process on the second feature image to obtain a third feature image.

[0191] The present disclosure does not limit the number of transposed convolution layers, the size of the transposed convolution kernel, the specific structure of the transposed convolution layer, etc. The transposed convolution layer is used to upsample the second feature image, which can be regarded as the reverse process of convolution, thereby restoring the size of the image.

[0192] When multiple transposed convolutional layers are set, a stepwise upsampling method may be used, for example, the image may be upsampled by a factor of 2 until the last transposed convolutional layer outputs the third feature image.

[0193] In one implementation, the convolution layer and the transposed convolution layer may be symmetrical structures, and the third feature image has the same size and the same number of channels as the first feature image.

[0194] In one implementation, a direct connection can be established between the convolutional layer and the transposed convolutional layer, such as Figure 6 As shown, a direct connection is established between the convolution layer and the transposed convolution layer corresponding to the feature image of the same size, thereby directly connecting the feature image information of the convolution link to the feature image in the transposed convolution link, which is conducive to obtaining a third feature image with more comprehensive information.

[0195] Step S740, using the second pixel rearrangement layer to perform a second pixel rearrangement process and a third pixel rearrangement process on the third feature image, respectively, to obtain masks of the intermediate face image and the face image to be beautified.

[0196] The second pixel rearrangement layer is used to perform a second pixel rearrangement process and a third pixel rearrangement process. Both the second pixel rearrangement process and the third pixel rearrangement process can be rearrangements that reduce the number of channels. For example, multiple channels of the third feature image can be rearranged into a single channel.

[0197] In one implementation, step S740 includes:

[0198] Input the third feature image with the number of channels C1*n*n into the second pixel rearrangement layer;

[0199] Rearrange the pixels at the same position in each n*n channel of the third feature image to the n*n neighborhood in a single channel, and output an intermediate face image with C1 channels;

[0200] Rearrange the pixels at the same position in each r*r channel of the third feature image to the r*r neighborhood in a single channel, and output a mask with C2 channels;

[0201] Wherein, n represents the parameter of the second pixel rearrangement processing, r represents the parameter of the third pixel rearrangement processing, and both n and r are positive integers not less than 2, and the two can be the same or different. For example, when C1=C2, n=r. The second pixel rearrangement processing and the third pixel rearrangement processing can be opposite to the first pixel rearrangement processing. For example, the second pixel rearrangement processing and the third pixel rearrangement processing can be implemented using the depth_to_space function in TensorFlow to convert the depth features in the third feature image into spatial features, or can be implemented using a transposed convolution operation with a step size of n or r. In this case, the second pixel rearrangement layer can be regarded as a special transposed convolution layer.

[0202] If the convolution layer and the transposed convolution layer are completely symmetrical structures, that is, the size and number of channels of the third feature image are the same as those of the first feature image, then m=n, and the size of the face image to be beautified is the same as that of the intermediate face image. That is, the first beautification process of the deep neural network does not change the image size and number of channels.

[0203] It should be noted that the first beautification process of the deep neural network also does not change the number of faces. For example, the face image to be beautified is composed of 4 face sub-images to be beautified. After the first beautification process of the deep neural network, the middle face image also includes 4 faces, which are the faces after beautification of the faces in the 4 face sub-images to be beautified.

[0204] It can be seen from the above that this exemplary embodiment can achieve the two tasks of the first beauty processing and determining the image mask without making too many modifications to the deep neural network or setting a complex structure, thereby reducing the computing cost.

[0205] This exemplary embodiment trains a deep neural network so that it can simultaneously implement the first beautification process and determine the image mask. In one embodiment, the image beautification processing method also includes a training process for the deep neural network, referring to Figure 8 As shown, the following steps S810 to S830 are included:

[0206] Step S810, obtaining a sample image to be beautified and an annotated beautified image and an annotated mask corresponding to the sample image to be beautified;

[0207] Step S820, inputting the sample image to be beautified into the deep neural network to be trained to output a sample beautified image and a sample mask;

[0208] Step S830: updating the parameters of the deep neural network based on the difference between the sample beauty image and the annotated beauty image, and the difference between the sample mask and the annotated mask.

[0209] Among them, the sample images to be beautified can be various facial images that have not been beautified. For each sample image to be beautified, a corresponding annotated beauty image can be obtained by artificial beauty retouching. The artificial beauty retouching method can correspond to the first beauty processing, including but not limited to removing spots and acne, removing eye bags, even skin tone, smoothing light and shadow, dirty mouth corner processing, lip processing, etc. For example, if the first beauty processing is blemish removal, then the annotated beauty image is obtained by artificially removing blemishes from the sample image to be beautified. In addition, by manually marking the position of a preset area in the sample image to be beautified, a corresponding annotation mask can be obtained. The present disclosure does not limit the specific form of the annotation mask. For example, the positions of four preset areas can be marked, 0 represents the background area, 1 represents the portrait area, 2 represents the facial skin area, and 3 represents the facial occlusion area. In addition, one-hot encoding can be used to convert 0 to 0000, 1 to 0100, 2 to 0010, and 3 to 0001 as a marking mask, or the information of the marked preset area can be converted into the above-mentioned 4-channel or single-channel mask image as a marking mask. In this way, a beauty image dataset formed by a triple of a sample image to be beautified-annotated beauty image-annotated mask is constructed.

[0210] In one implementation, multiple face images may be combined to obtain a sample image to be beautified, and the artificially beautified images corresponding to the multiple face images may be combined, and the artificially annotated masks corresponding to the multiple face images may be combined to obtain an annotated beauty image and annotated mask corresponding to the sample image to be beautified. In other words, the beauty image dataset may include different types such as single-face images, multi-face images, and combined-face images.

[0211] The beautification image dataset can be further divided into a training set and a validation set. The sample images to be beautified in the training set are input into the deep neural network to be trained to obtain sample beautification images and sample masks. Since the deep neural network is not trained, there is usually a large difference between the sample beautification image and the annotated beautification image, and there is usually a large difference between the sample mask and the annotated mask. Based on the difference between the sample beautification image and the annotated beauty image, and the difference between the sample mask and the annotated mask, the loss function value can be calculated. For example, a first loss function for the sample beautification image and a second loss function for the sample mask can be established, the sample beautification image and the annotated beauty image are substituted into the first loss function, the first loss function value is calculated, the sample mask and the annotated mask are substituted into the second loss function, and the second loss function value is calculated. Based on the first loss function value and the second loss function value, the parameters of the deep neural network are back-propagated and updated. When the accuracy of the deep neural network on the validation set reaches the set standard, it is determined that the training is completed.

[0212] Therefore, in step S320, the facial image to be beautified is input into a trained deep neural network, which can simultaneously output the intermediate facial image and the mask of the facial image to be beautified.

[0213] Continue to refer Figure 3 In step S330, a second beautification process is performed on the intermediate face image based on the mask of the face image to be beautified to obtain a target image.

[0214] Among them, the target image can be an image output after beautification, or it can be an image output as a beautification stage and further beautification processing is performed, and the present disclosure does not limit this.

[0215] Since the mask of the facial image to be beautified represents the position information of at least one preset area in the facial image to be beautified, which is basically equivalent to the position information of the preset area in the intermediate facial image, targeted second beautification processing can be performed on different preset areas in the intermediate facial image based on the mask.

[0216] In one implementation, performing the second beautification processing on the intermediate face image based on the mask of the face image to be beautified to obtain the target image may include the following steps:

[0217] At least one to-be processed region in the intermediate face image is determined based on the mask of the face image to be beautified, and a second beautification process is performed on the to-beautify region to obtain a target image.

[0218] Among them, the second beauty processing may include at least one specific beauty function, and each beauty function corresponds to a specific preset area, that is, the area to be processed. The beauty function of the second beauty processing can be selected by the user or automatically determined by the system. For example, in the beauty App, all optional beauty functions can be provided, such as skin grinding, whitening and brightening, facial features deformation, makeup, etc., and the user selects skin grinding and makeup from them. Then the second beauty processing includes these two beauty functions of skin grinding and makeup. Skin grinding and makeup correspond to the facial skin area, and the facial skin area can be determined as the area to be processed. Then, the area to be processed in the middle face image is processed with the corresponding beauty function in the second beauty processing, and the target image is obtained after processing.

[0219] In one embodiment, the mask of the face image to be beautified includes a facial skin mask of the face image to be beautified and a facial occlusion mask of the face image to be beautified. The above-mentioned determining at least one area to be processed in the intermediate face image based on the mask of the face image to be beautified, performing the second beautification processing on the area to be processed, and obtaining the target image may include the following steps:

[0220] Based on the facial skin mask of the face image to be beautified and the facial occluder mask of the face image to be beautified, determining a facial skin region to be processed in the intermediate face image, where the facial skin region to be processed does not include a facial occluder region;

[0221] The second beautification processing is performed on the facial skin area to be processed in the intermediate face image to obtain a target image.

[0222] Among them, the facial skin area to be processed can be the part of the facial skin area that does not include facial obstructions. In some facial images to be beautified, there may be situations where hands or other objects obstruct the face, such as images of people holding their faces or covering their mouths, or images of people wearing glasses, etc. These obstructed parts usually do not require second beautification processing such as skin grinding and makeup. By determining the facial skin mask and the facial obstruction mask of the facial image to be beautified, the facial obstruction area can be excluded from the facial skin area to obtain the facial skin area to be processed. Performing the second beautification processing on the facial skin area to be processed in the intermediate facial image makes the position of the second beautification processing more accurate and more targeted, which is conducive to improving the effect of the second beautification processing and obtaining a high-quality target image.

[0223] In one implementation, if the face image to be beautified is determined based on the face sub-image to be beautified in the global image to be beautified, for example, the face image to be beautified is generated after adjusting the face sub-image to be beautified in the global image to be beautified, then Fig. 9 As shown, the second beautification processing is performed on the intermediate face image based on the mask of the face image to be beautified to obtain the target image, which may include the following steps S910 and S920:

[0224] Step S910 , replacing the face sub-image to be beautified in the global image to be beautified with the intermediate face image, to obtain the intermediate global image corresponding to the global image to be beautified.

[0225] When replacing the face sub-image to be beautified in the global image to be beautified with the intermediate face image, the size, direction, etc. of the intermediate face image may be adaptively adjusted.

[0226] In one embodiment, if the face image to be beautified is determined based on the face sub-image to be beautified, and the face sub-image to be beautified is adjusted, then the intermediate face image can be reversely adjusted based on the adjustment processing information to obtain the face sub-image to be beautified. Reverse adjustment processing refers to processing that is completely opposite to the above adjustment processing. For example, if the face image to be beautified is generated by transforming and combining the face sub-images to be beautified, then after obtaining the intermediate face image, the intermediate face image can be split based on the combination information in the adjustment processing information to obtain an image of a single face. Then, each image is transformed in the opposite direction based on the transformation information in the adjustment processing information, including removing padded pixels, upsampling, rotating 90 degrees in the opposite direction, etc., to obtain a face sub-image to be beautified that corresponds one to one with the face sub-image to be beautified. The face sub-image to be beautified is consistent with the face sub-image to be beautified in direction, size, etc., and is the first image after beauty processing corresponding to the single face. Furthermore, the face sub-image to be beautified in the global image to be beautified can be replaced with the corresponding beautified face sub-image, and a 1:1 original position replacement can be performed to obtain an intermediate global image corresponding to the global image to be beautified, and the intermediate global image is an image of the face in the global image to be beautified after the first beautification processing.

[0227] Step S920: performing a second beautification process on the intermediate global image based on the mask of the face image to be beautified to obtain a target image.

[0228] The mask of the beautified face image includes the position information of the preset area. The preset area that needs to be subjected to the second beautification process is determined as the area to be processed. Combined with the corresponding position of the face image to be beautified in the global image to be beautified, the position of the area to be processed in the global image to be beautified can be determined, which is basically equivalent to the position of the area to be processed in the intermediate global image. Thus, the area to be processed can be determined in the intermediate global image, and then the second beautification process is performed on the area to be processed in the intermediate global image to obtain the target image.

[0229] In one implementation, when the size of the input image to be beautified is adjusted based on the deep neural network, adjustment information can be recorded. Fig.10 As shown, the second beautification processing is performed on the intermediate global image based on the mask of the face image to be beautified to obtain the target image, which may include the following steps S1010 and S1020:

[0230] Step S1010 , performing inverse adjustment processing on the mask of the face image to be beautified according to the adjustment processing information to obtain the mask of the global image to be beautified.

[0231] The reverse adjustment processing of the mask of the face image to be beautified can refer to the reverse adjustment processing of the intermediate face image. The mask of the face image to be beautified is used to represent the position information of the preset area in the face image to be beautified. Through the reverse adjustment processing, the preset area can be mapped to the global image to be beautified, and the position information of the preset area in the global image to be beautified can be obtained, which is represented in the form of the mask of the global image to be beautified.

[0232] In one implementation, the above-mentioned performing a reverse adjustment process on the mask of the face image to be beautified according to the adjustment process information to obtain the mask of the global image to be beautified may include the following steps:

[0233] Obtain an initialization mask corresponding to the global image to be beautified;

[0234] Performing a reverse adjustment process on the mask of the face image to be beautified according to the processed information to obtain a mask to be replaced;

[0235] The initialized mask is numerically replaced according to the mask to be replaced to obtain the mask of the global image to be beautified.

[0236] Among them, the initialization mask is used to initialize the information of different areas of the global image to be beautified. Exemplarily, a mask with the same size as the global image to be beautified can be generated, and filled with numerical values ​​representing blank areas or background areas (such as preset values ​​such as 0, 1, etc.) as an initialization mask. Alternatively, semantic segmentation is performed on the global image to be beautified, and the generated semantic segmentation image is used as the initialization mask. Alternatively, skin color detection is performed on the global image to be beautified. The skin color detection can determine whether it is a skin pixel through the pixel value of each pixel (such as RGB value), thereby detecting the pixels belonging to the skin and obtaining the skin mask of the global image to be beautified. The skin mask can show the skin area and other areas (i.e., areas other than the skin) in the global image to be beautified, and can be used as an initialization mask.

[0237] The inverse adjustment processing of the mask of the face image to be beautified can refer to the inverse adjustment processing of the intermediate face image mentioned above. For example, if the face image to be beautified is generated by transforming and combining the face sub-images to be beautified, then after obtaining the mask of the face image to be beautified, the mask can be split according to the combination information in the adjustment processing information (i.e., the opposite operation to the combination) to obtain the mask of a single face image. Then, according to the transformation information in the adjustment processing information, the mask of each face image is transformed in the opposite direction, including removing the area corresponding to the filling pixels, upsampling, reverse rotation of 90 degrees, etc., to obtain the mask of the face sub-image to be beautified, which is called the mask to be replaced.

[0238] Combined with the position of the face sub-image to be beautified in the global image to be beautified, the mask to be replaced is mapped to the global image to be beautified, and the corresponding position of the mask to be replaced in the global image to be beautified is determined, that is, the corresponding position of the mask to be replaced in the above-mentioned initialization mask, and then the value of the corresponding position in the initialization mask is replaced by the mask to be replaced to obtain the mask of the global image to be beautified.

[0239] In one embodiment, the mask of the face image to be beautified may include a facial skin mask of the face image to be beautified and a facial occlusion mask of the face image to be beautified. Correspondingly, the mask to be replaced may include a facial skin mask to be replaced and a facial occlusion mask to be replaced. The above-mentioned numerical replacement of the initialization mask according to the mask to be replaced to obtain the mask of the global image to be beautified may include the following steps:

[0240] The values ​​corresponding to the facial skin area in the initialization mask are replaced according to the facial skin mask to be replaced, and the values ​​corresponding to the facial occluder area in the initialization mask are replaced according to the facial occluder mask to be replaced, so as to obtain the mask of the global image to be beautified.

[0241] For example, a 4-channel mask image of the face image to be beautified is obtained, including a background mask image, a portrait mask image, a facial skin mask image, and a facial occlusion mask image of the face image to be beautified. The 4-channel mask image is subjected to a reverse adjustment process to obtain a 4-channel mask image to be replaced, including a background mask image to be replaced, a portrait mask image to be replaced, a facial skin mask image to be replaced, and a facial occlusion mask image to be replaced. Detect the pixels that obtain the maximum value and are non-zero in the facial skin mask image to be replaced, and replace the pixels at the corresponding positions in the initialization mask with the values ​​corresponding to the facial skin or the values ​​of the pixels in the facial skin mask image to be replaced, so as to indicate that the pixels belong to the facial skin area. Detect the pixels that obtain the maximum value and are non-zero in the facial occlusion mask image to be replaced, and replace the pixels at the corresponding positions in the initialization mask with the values ​​corresponding to the facial skin or the values ​​of the pixels in the facial occlusion mask image to be replaced, so as to indicate that the pixels belong to the facial occlusion area. Thus, the facial skin area and the facial occluder area are marked in the initialization mask, and the mask after the above replacement is used as the mask of the global image to be beautified.

[0242] By numerically replacing the initialization mask, the preset areas in the global image to be beautified can be further accurately identified on the basis of the initialization mask, such as the facial skin area, the facial occlusion area, etc., so as to facilitate the subsequent second beautification processing. Since the relevant algorithm used in the initialization mask can only roughly determine the different areas in the global image to be beautified, for example, the skin color detection may identify the bangs (such as yellow hair) and the objects worn on the face that are similar to the skin color as skin, after the initialization mask is numerically replaced with the mask to be replaced, the false detection in the initialization mask can be eliminated, and the accuracy of the mask of the global image to be beautified can be improved.

[0243] Step S1020 , performing a second beautification process on the intermediate global image based on the mask of the global image to be beautified to obtain a target image.

[0244] The mask of the global image to be beautified represents the position information of a preset area in the global image to be beautified, which is basically equivalent to the position information of the preset area in the intermediate global image. Therefore, the intermediate global image can be subjected to a second beautification processing based on the mask of the global image to be beautified. Specifically, the second beautification processing is performed on the area to be processed in the intermediate global image to obtain a target image.

[0245] As can be seen from the above, this exemplary embodiment can perform the first beautification processing on the beautified face image, that is, only perform the first beautification processing on the local image of the face, and can perform the second beautification processing on the intermediate global image, that is, perform the second beautification processing on the entire global image. Since the second beautification processing is mostly regional processing, processing on the scale of the global image is conducive to improving the overall visual coordination of the image and reducing the processing amount.

[0246] The middle face image is an image that has been beautified by the first beautification process, and sometimes there is over-processing. In one embodiment, in order to increase the realism of the middle face image, before replacing the face sub-image to be beautified in the global image to be beautified with the middle face image, the face sub-image to be beautified can be used to perform beautification weakening processing on the middle face image. Beautification weakening processing refers to reducing the beautification degree of the middle face image to increase the naturalness and realism. Two exemplary methods of beauty weakening processing are provided below:

[0247] Method 1: According to the set beautification degree parameters, the sub-image of the face to be beautified is merged into the middle face image. The beautification degree parameter may be a beautification intensity parameter under a specific beautification function, such as the degree of blemish removal. In this exemplary embodiment, the beautification degree parameter may be a parameter used for the current setting, a system default parameter, or a parameter used for the last beautification. After determining the beautification degree parameter, the sub-image of the face to be beautified may be merged with the middle face image using the beautification degree parameter as a weight. For example, assuming that the range of the blemish removal degree is 0 to 100 and the currently set value is a, refer to the following formula:

[0248]

[0249] Wherein, image_blend represents the fused image, image_ori represents the face sub-image to be beautified, and image_deblemish represents the intermediate face image. When a is 0, it means no blemish removal is performed, and the face sub-image to be beautified is used completely; when a is 100, it means complete blemish removal, and the intermediate face image is used completely. Therefore, formula (12) indicates that an image between the face sub-image to be beautified and the intermediate face image is obtained through fusion. The larger a is, the closer the obtained image is to the intermediate face image, that is, the higher the degree of beautification is, the more obvious the beautification effect is.

[0250] Method 2: Fusing the high-frequency image of the face sub-image to be beautified into the intermediate face image. The high-frequency image refers to an image containing high-frequency information such as detailed texture in the face sub-image to be beautified.

[0251] In one embodiment, the high-frequency image may be acquired by:

[0252] When determining the face image to be beautified according to the face sub-image to be beautified, if the face sub-image to be beautified is down-sampled, then the down-sampled face sub-image obtained after the down-sampling is up-sampled to obtain the up-sampled face sub-image;

[0253] According to the difference between the face sub-image to be beautified and the upsampled face sub-image, a high-frequency image of the face sub-image to be beautified is obtained.

[0254] Among them, the resolution of the downsampled face sub-image is lower than that of the face sub-image to be beautified. Generally, in the process of downsampling, the high-frequency information of the image will inevitably be lost. The downsampled face sub-image is upsampled so that the obtained upsampled face sub-image has the same resolution as the face sub-image to be beautified. It should be noted that if the face sub-image to be beautified is rotated before downsampling, the downsampled face sub-image can also be rotated in the opposite direction after upsampling so that the obtained upsampled face sub-image has the same direction as the face sub-image to be beautified.

[0255] Upsampling can be performed by bilinear interpolation, nearest neighbor interpolation, and other methods. Although upsampling can restore the resolution, it is difficult to completely restore the lost high-frequency information, that is, the upsampled face sub-image can be regarded as a low-frequency image of the face sub-image to be beautified. Thus, the difference between the face sub-image to be beautified and the upsampled face sub-image is determined. For example, the face sub-image to be beautified can be subtracted from the upsampled face sub-image, and the result is the high-frequency information of the face sub-image to be beautified. The subtracted value is formed into an image, that is, the high-frequency image of the face sub-image to be beautified.

[0256] In another implementation, the beautified face sub-image may be filtered to extract high-frequency information and obtain a high-frequency image.

[0257] When the high-frequency image is fused to the middle face image, the high-frequency image can be superimposed on the middle face image by direct addition, so that high-frequency information such as detail texture is added to the middle face image, making it more realistic.

[0258] Since the face sub-image to be beautified and the upsampled face sub-image are usually very similar, the pixel values ​​in the high-frequency image obtained based on their difference are generally small, such as the RGB channel values ​​not exceeding 4. However, for the mutation positions in the face sub-image to be beautified, such as small black moles on the face, they have strong high-frequency information, so the pixel values ​​of the corresponding positions in the high-frequency image may be relatively large. When the high-frequency image is fused to the face sub-image to be beautified, the pixel values ​​at these positions may have adverse effects, such as sharp edges such as "mole marks", resulting in unnatural visual experience.

[0259] In view of the above problem, in one implementation, the image beautification processing method may further include the following steps:

[0260] Identify defect points in high-frequency images;

[0261] The pixel values ​​in a preset area around the above defect point in the high-frequency image are adjusted to a preset value range.

[0262] Among them, the defect point is a pixel point with strong high-frequency information, and the point with a larger pixel value in the high-frequency image can be determined as a defect point. Alternatively, in an implementation, the defect point can be determined by the following method:

[0263] Unify the size of the intermediate face image (or the beautified face sub-image) and the corresponding face sub-image to be beautified, and then subtract them to obtain the difference value of each pixel;

[0264] When it is determined that the difference value of a certain pixel point meets the preset defect condition, the pixel point corresponding to the pixel point in the high-frequency image is determined as the defect point.

[0265] The preset defect condition is used to measure the difference between the intermediate face image and the face sub-image to be beautified, so as to determine whether each pixel is a defect point to be removed. In the defect removal process, small moles and acne on the face are usually removed and the face skin color is filled. At this position, the difference between the face sub-image to be beautified and the face sub-image to be beautified is very large, so the defect points can be identified by setting the preset defect condition.

[0266] Exemplarily, the preset defect condition may include: the difference of each channel is greater than the first color difference threshold, and at least one of the difference of each channel is greater than the second color difference threshold. The first color difference threshold and the second color difference threshold may be empirical thresholds. For example, when the above-mentioned channels include RGB, the first color difference threshold may be 20, and the second color difference threshold may be 40. Thus, after obtaining the difference between each pixel in the beautified face sub-image and the face sub-image to be beautified, the specific difference of the three RGB channels in the difference is judged to determine whether the difference of each channel is greater than 20, and whether there is at least one channel whose difference is greater than 40. When these two conditions are met, it means that the preset defect condition is met, and the pixel at the corresponding position in the high-frequency image is determined as a defect point.

[0267] After determining the defect point, the preset area around the defect point can be further determined in the high-frequency image. For example, it can be a 5*5 pixel area centered on the defect point. The specific size can be determined according to the size of the high-frequency image, and the present disclosure does not limit it. The pixel values ​​in the preset area are adjusted to a preset numerical range. The preset numerical range is generally a smaller numerical range, which can be determined based on experience and actual needs. The pixel value usually needs to be reduced during adjustment. Exemplarily, the preset numerical range can be -2 to 2, and the pixel values ​​around the defect point may exceed -5 to 5. Adjusting it to -2 to 2 actually performs limit processing. This can weaken sharp edges such as "mole marks" and increase the visual natural feeling.

[0268] The above describes two beautification and weakening processing methods. This exemplary embodiment can use these two beautification and weakening processing methods at the same time. For example, the face sub-image to be beautified is firstly fused with the intermediate face image through method 1, and then the high-frequency image is superimposed thereon through method 2 to obtain the intermediate face image after beautification and weakening processing, which has both good beautification effect and realism.

[0269] It should be understood that before any of the above fusion methods are performed, the sub-image of the face to be beautified or the intermediate face image may be subjected to adaptive processing such as size. Exemplarily, if the sub-image of the face to be beautified is adjusted when determining the face image to be beautified based on the sub-image of the face to be beautified, then the intermediate face image may be subjected to reverse adjustment processing, for example, to obtain a beautified face sub-image that corresponds one to one with the sub-image of the face to be beautified. Then, a beautification weakening process is performed, for example, the sub-image of the face to be beautified is fused to the beautified face sub-image, and image_deblemish in the above formula (12) may represent the beautified face sub-image; or the high-frequency image of the sub-image of the face to be beautified is fused to the beautified face sub-image.

[0270] In one implementation, when the intermediate face image is used to replace the face sub-image to be beautified in the global image to be beautified, a gradient process may be performed on the boundary area between the unreplaced area in the global image to be beautified and the intermediate face image to make a smooth transition in the boundary area.

[0271] The unreplaced area in the global image to be beautified is the area in the global image to be beautified except for the face sub-image to be beautified. The boundary area between the unreplaced area and the middle face image actually includes two parts: the boundary area in the unreplaced area adjacent to the middle face image, and the boundary area in the middle face image adjacent to the unreplaced area. This exemplary embodiment can perform gradient processing on any part of them, or can perform gradient processing on both parts at the same time.

[0272] refer to Fig.11 As shown, a certain proportion (such as 10%) of the boundary area can be determined in the middle face image, which extends inward from the edge of the beautified face sub-image. It should be noted that the boundary area usually needs to avoid the face part to avoid changing the color of the face part during the gradient processing. For example, the face sub-image to be beautified is intercepted by the above-mentioned stable bounding box, so that the face in the face sub-image to be beautified is at a certain distance from the boundary, and the face in the beautified face sub-image is also at a certain distance from the boundary. In this way, when performing the gradient processing, the face part can be better avoided. After determining the boundary area, the inner edge color of the boundary area is obtained, which is recorded as the first color; the inner edge color of the unreplaced area is obtained, which is recorded as the second color; and the boundary area is subjected to the gradient processing of the first color and the second color. As a result, the boundary between the unreplaced area and the middle face image is a gradient color area ( Fig.11 This creates a smooth transition and prevents color mutations that would cause visual disharmony.

[0273] If the face image to be beautified is determined based on the face sub-image to be beautified, and the face sub-image to be beautified is adjusted, then the intermediate face image can be reversely adjusted, for example, to obtain the face sub-images to be beautified that correspond to the face sub-images to be beautified. Then, each face sub-image to be beautified is replaced with the corresponding face sub-image to be beautified in the global image to be beautified, and a gradient process is performed on the boundary area to obtain the intermediate global image.

[0274] Fig.12 The schematic flow of the image beautification processing method is shown, including:

[0275] Step S1201, obtaining a global image to be beautified, for example, the current frame image in the video can be used as the global image to be beautified, and extracting a face sub-image to be beautified from the global image to be beautified.

[0276] Step S1202, transform and combine the face sub-images to be beautified according to the input image size of the deep neural network to obtain the face image to be beautified. During the transformation process, the image after downsampling the face sub-image to be beautified is saved and recorded as the downsampled face sub-image.

[0277] Step S1203, upsampling the downsampled face sub-image. If rotation, padding and other processing are performed when obtaining the downsampled face sub-image, reverse rotation, padding removal and other processing can also be performed to obtain an upsampled face sub-image, which has the same resolution as the corresponding face sub-image to be beautified.

[0278] Step S1204 , subtracting the face sub-image to be beautified from the corresponding upsampled face sub-image to obtain a high-frequency image of the face sub-image to be beautified.

[0279] Step S1205: input the face image to be beautified into a deep neural network, output an intermediate face image after the first beautification process, and output a mask of the face image to be beautified.

[0280] Step S1206, using the transformed and combined adjustment processing information, inverse adjustment processing is performed on the intermediate face image, including image splitting and inverse transformation, to obtain beautified face sub-images corresponding one to one to the face sub-images to be beautified.

[0281] Step S1207 , the beautified face sub-image is fused with the corresponding face sub-image to be beautified according to the beautification degree parameter, and then added to the high-frequency image of the face sub-image to be beautified to obtain the face sub-image to be replaced.

[0282] Step S1208, fusing the face sub-image to be replaced into the global image to be beautified. Specifically, the face sub-image to be replaced can replace part of the face sub-image to be beautified in the global image to be beautified, and performing color gradient processing on the edges, so that the face in the global image to be beautified is replaced by the face after the first beautification processing, to obtain an intermediate global image.

[0283] Step S1209: performing inverse adjustment processing on the mask of the face image to be beautified according to the adjustment processing information to obtain the mask of the global image to be beautified.

[0284] Step S1210: Perform a second beautification process on the intermediate global image based on the mask of the global image to be beautified to obtain a target image. Other beautification processes may also be performed on the target image subsequently.

[0285] The exemplary embodiment of the present disclosure also provides an image beautification processing device. Fig.13 As shown, the image beautification processing device 1300 may include:

[0286] The image acquisition module 1310 is configured to acquire a face image to be beautified;

[0287] The first beautification processing module 1320 is configured to perform a first beautification processing on the face image to be beautified through a pre-trained deep neural network to obtain an intermediate face image, and determine a mask of the face image to be beautified through the deep neural network;

[0288] The second beautification processing module 1330 is configured to perform a second beautification processing on the intermediate facial image based on the mask of the facial image to be beautified to obtain a target image.

[0289] In one embodiment, the deep neural network includes a first pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a second pixel rearrangement layer. The pre-trained deep neural network performs the first beautification processing on the face image to be beautified to obtain an intermediate face image, and determines the mask of the face image to be beautified by the deep neural network, including:

[0290] Using the first pixel rearrangement layer to perform a first pixel rearrangement process on the beautified face image to obtain a first feature image;

[0291] Performing convolution processing on the first feature image using a convolution layer to obtain a second feature image;

[0292] Performing transposed convolution processing on the second feature image using a transposed convolution layer to obtain a third feature image;

[0293] The second pixel rearrangement layer is used to perform a second pixel rearrangement process and a third pixel rearrangement process on the third feature image to obtain masks of an intermediate face image and the face image to be beautified.

[0294] In one implementation, the image beautification processing apparatus 1300 may further include a deep neural network training module configured to:

[0295] Obtaining a sample image to be beautified and an annotated beautified image and an annotated mask corresponding to the sample image to be beautified;

[0296] Inputting the sample image to be beautified into the deep neural network to be trained to output a sample beautified image and a sample mask;

[0297] Based on the difference between the sample beauty image and the annotated beauty image, as well as the difference between the sample mask and the annotated mask, the parameters of the deep neural network are updated.

[0298] In one embodiment, the mask of the face image to be beautified is used to indicate the position information of at least one preset area in the face image to be beautified. The second beautification processing is performed on the intermediate face image based on the mask of the face image to be beautified to obtain the target image, including:

[0299] At least one to-be processed region in the intermediate face image is determined based on the mask of the face image to be beautified, and a second beautification process is performed on the to-beautify region to obtain a target image.

[0300] In one embodiment, the mask of the face image to be beautified includes a facial skin mask of the face image to be beautified and a facial occlusion mask of the face image to be beautified. The above-mentioned determining at least one area to be processed in the intermediate face image based on the mask of the face image to be beautified, performing a second beautification process on the area to be processed, and obtaining a target image includes:

[0301] Based on the facial skin mask of the face image to be beautified and the facial occluder mask of the face image to be beautified, determining a facial skin region to be processed in the intermediate face image, where the facial skin region to be processed does not include a facial occluder region;

[0302] The second beautification processing is performed on the facial skin area to be processed in the intermediate face image to obtain the target image.

[0303] In one implementation, the obtaining of the face image to be beautified includes:

[0304] Extracting one or more face sub-images to be beautified from the global image to be beautified, and determining the face image to be beautified according to the face sub-images to be beautified;

[0305] Performing a second beautification process on the intermediate face image based on the mask of the face image to be beautified to obtain a target image, including:

[0306] The intermediate face image is used to replace the face sub-image to be beautified in the global image to be beautified, so as to obtain the intermediate global image corresponding to the global image to be beautified;

[0307] A second beautification process is performed on the intermediate global image based on the mask of the face image to be beautified to obtain a target image.

[0308] In one implementation, determining the face image to be beautified based on the face sub-image to be beautified includes:

[0309] The face sub-image to be beautified is adjusted based on the input image size of the deep neural network to generate the face image to be beautified, and the adjustment process includes transformation and / or combination.

[0310] In one implementation, the above-mentioned processing of adjusting the face sub-image to be beautified based on the input image size of the deep neural network to generate the face image to be beautified includes:

[0311] According to the number of face sub-images to be beautified, the input image size is divided into sub-image sizes corresponding to the face sub-images to be beautified;

[0312] Transforming the corresponding face sub-image to be beautified based on the size of each sub-image;

[0313] The transformed face sub-images to be beautified are combined to generate a face image to be beautified.

[0314] In one implementation, transforming the corresponding face sub-image to be beautified based on the size of each sub-image includes any one or more of the following:

[0315] When the size relationship between the width and height of the face sub-image to be beautified is different from the size relationship between the width and height of the sub-image size, the face sub-image to be beautified is rotated 90 degrees;

[0316] When the size of the face sub-image to be beautified or the rotated face sub-image to be beautified is larger than the sub-image size, downsampling the face sub-image to be beautified or the rotated face sub-image to be beautified according to the sub-image size;

[0317] When the size of the face sub-image to be beautified or the face sub-image to be beautified after at least one of rotation and downsampling is smaller than the sub-image size, the face sub-image to be beautified is padded according to the difference between the size of the face sub-image to be beautified and the sub-image size, or the face sub-image to be beautified after at least one of rotation and downsampling is padded according to the difference between the size of the face sub-image to be beautified and the sub-image size.

[0318] In one embodiment, when the size of the input image of the face to be beautified is adjusted based on the deep neural network, the adjustment processing information is recorded. The second beautification processing is performed on the intermediate global image based on the mask of the face image to be beautified to obtain the target image, including:

[0319] Performing a reverse adjustment process on the mask of the face image to be beautified according to the adjustment process information to obtain a mask of the global image to be beautified;

[0320] A second beautification process is performed on the intermediate global image based on the mask of the global image to be beautified to obtain a target image.

[0321] In one implementation, the above-mentioned performing a reverse adjustment process on the mask of the face image to be beautified according to the adjustment processing information to obtain the mask of the global image to be beautified includes:

[0322] Obtain an initialization mask corresponding to the global image to be beautified;

[0323] Performing a reverse adjustment process on the mask of the face image to be beautified according to the adjustment process information to obtain a mask to be replaced;

[0324] The initialized mask is numerically replaced according to the mask to be replaced to obtain the mask of the global image to be beautified.

[0325] In one implementation, the obtaining of the initialization mask corresponding to the global image to be beautified includes:

[0326] Skin color detection is performed on the global image to be beautified to obtain a skin mask of the global image to be beautified as an initialization mask.

[0327] In one embodiment, the mask of the face image to be beautified includes a facial skin mask of the face image to be beautified and a facial occlusion mask of the face image to be beautified; the mask to be replaced includes a facial skin mask to be replaced and a facial occlusion mask to be replaced. The above-mentioned numerical replacement of the initialization mask according to the mask to be replaced to obtain the mask of the global image to be beautified includes:

[0328] The values ​​corresponding to the facial skin area in the initialization mask are replaced according to the facial skin mask to be replaced, and the values ​​corresponding to the facial occluder area in the initialization mask are replaced according to the facial occluder mask to be replaced, so as to obtain the mask of the global image to be beautified.

[0329] In one implementation, the second beauty processing module 1330 is further configured to:

[0330] Before the intermediate face image is used to replace the face sub-image to be beautified in the global image to be beautified, the intermediate face image is beautified by using the face sub-image to be beautified.

[0331] In one implementation, the beautification and weakening processing of the intermediate face image using the face sub-image to be beautified includes:

[0332] According to the set beautification degree parameters, the face sub-image to be beautified is fused into the intermediate face image.

[0333] In one implementation, the beautification-weakening processing of the beautification face sub-image using the face sub-image to be beautified includes:

[0334] The high-frequency image of the face sub-image to be beautified is fused into the intermediate face image.

[0335] In one implementation, the image acquisition module 1310 is further configured to:

[0336] When determining the face image to be beautified according to the face sub-image to be beautified, if the face sub-image to be beautified is downsampled, the downsampled face sub-image obtained after the downsampling is upsampled to obtain an upsampled face sub-image, and the upsampled face sub-image has the same resolution as the face sub-image to be beautified;

[0337] According to the difference between the face sub-image to be beautified and the upsampled face sub-image, a high-frequency image of the face sub-image to be beautified is obtained.

[0338] In one implementation, the second beauty processing module 1330 is further configured to:

[0339] Before fusing the high-frequency image of the face sub-image to be beautified into the face sub-image of the beautified person, defect points are determined in the high-frequency image, and pixel values ​​in a preset area around the defect points in the high-frequency image are adjusted to within a preset value range.

[0340] In one implementation, the second beauty processing module 1330 is further configured to:

[0341] When the middle face image is used to replace the face sub-image to be beautified in the global image to be beautified, a gradient process is performed on the boundary area between the non-replaced area in the global image to be beautified and the middle face image, so that the boundary area forms a smooth transition.

[0342] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0343] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In one embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0344] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0345] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0346] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0347] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0348] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0349] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0350] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image beautification processing method, characterized in that: include: Get the face image to be beautified; Performing a first beautification process on the face image to be beautified by using a pre-trained deep neural network to obtain an intermediate face image, and determining a mask of the face image to be beautified by using the deep neural network; Performing a second beautification process on the intermediate face image based on the mask of the face image to be beautified to obtain a target image; The mask of the face image to be beautified is used to indicate position information of at least one preset area in the face image to be beautified; and performing a second beautification process on the intermediate face image based on the mask of the face image to be beautified to obtain a target image includes: At least one to-be processed region in the intermediate facial image is determined based on the mask of the facial image to be beautified, and a second beautification process is performed on the to-beautify region to obtain a target image.

2. The method according to claim 1, characterized in that: The deep neural network includes a first pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a second pixel rearrangement layer; the pre-trained deep neural network is used to perform a first beautification process on the face image to be beautified to obtain an intermediate face image, and the deep neural network is used to determine a mask of the face image to be beautified, including: Using the first pixel rearrangement layer to perform a first pixel rearrangement process on the face image to be beautified to obtain a first feature image; Using the convolution layer to perform convolution processing on the first feature image to obtain a second feature image; Using the transposed convolution layer to perform a transposed convolution process on the second feature image to obtain a third feature image; The second pixel rearrangement layer is used to perform a second pixel rearrangement process and a third pixel rearrangement process on the third feature image to obtain masks of the intermediate face image and the face image to be beautified.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining a sample image to be beautified and a labeled beautified image and a labeled mask corresponding to the sample image to be beautified; Inputting the sample image to be beautified into the deep neural network to be trained to output a sample beautified image and a sample mask; Based on the difference between the sample beauty image and the annotated beauty image, and the difference between the sample mask and the annotated mask, the parameters of the deep neural network are updated.

4. The method according to claim 1, characterized in that: The determining the mask of the face image to be beautified by the deep neural network includes: The deep neural network is used to identify k preset areas in the facial image to be beautified, where k is a positive integer not less than 2, and a k-channel mask image or a single-channel mask image is output accordingly; the k-channel mask images are respectively mask images of the k preset areas, and the value of each pixel in the single-channel mask image indicates which preset area it belongs to.

5. The method according to claim 1, characterized in that The mask of the face image to be beautified includes a facial skin mask of the face image to be beautified and a facial occlusion mask of the face image to be beautified; determining at least one area to be processed in the intermediate face image based on the mask of the face image to be beautified, performing a second beautification process on the area to be processed, and obtaining a target image, comprises: Determine a facial skin region to be processed in the intermediate facial image based on a facial skin mask of the facial image to be beautified and a facial occluder mask of the facial image to be beautified, wherein the facial skin region to be processed does not include a facial occluder region; A second beautification process is performed on the facial skin area to be processed in the intermediate facial image to obtain the target image.

6. The method according to claim 1, characterized in that The step of obtaining the face image to be beautified includes: Extracting one or more face sub-images to be beautified from the global image to be beautified, and determining the face image to be beautified according to the face sub-images to be beautified; The step of performing a second beautification process on the intermediate face image based on the mask of the face image to be beautified to obtain a target image includes: Using the intermediate face image to replace the face sub-image to be beautified in the global image to be beautified, to obtain an intermediate global image corresponding to the global image to be beautified; The second beautification processing is performed on the intermediate global image based on the mask of the facial image to be beautified to obtain the target image.

7. The method according to claim 6, characterized in that The step of determining the face image to be beautified according to the face sub-image to be beautified includes: The face sub-image to be beautified is adjusted based on the input image size of the deep neural network to generate the face image to be beautified, and the adjustment processing includes transformation and / or combination.

8. The method according to claim 7, characterized in that The step of adjusting the face sub-image to be beautified based on the input image size of the deep neural network to generate the face image to be beautified includes: According to the number of the facial sub-images to be beautified, dividing the input image size into sub-image sizes corresponding to the facial sub-images to be beautified; transforming the corresponding face sub-image to be beautified based on the size of each sub-image; The transformed face sub-images to be beautified are combined to generate the face image to be beautified.

9. The method according to claim 8, characterized in that The transforming the corresponding face sub-image to be beautified based on the size of each sub-image respectively includes any one or more of the following: When the size relationship between the width and height of the face sub-image to be beautified is different from the size relationship between the width and height of the sub-image size, rotating the face sub-image to be beautified by 90 degrees; When the size of the face sub-image to be beautified or the rotated face sub-image to be beautified is larger than the sub-image size, downsampling the face sub-image to be beautified or the rotated face sub-image to be beautified according to the sub-image size; When the size of the facial sub-image to be beautified or the facial sub-image to be beautified after at least one of rotation and downsampling is smaller than the sub-image size, the facial sub-image to be beautified is filled according to the difference between the size of the facial sub-image to be beautified and the size of the sub-image, or the facial sub-image to be beautified after at least one of rotation and downsampling is filled according to the difference between the size of the facial sub-image to be beautified and the size of the sub-image.

10. The method according to claim 7, characterized in that When adjusting the face sub-image to be beautified based on the input image size of the deep neural network, recording adjustment processing information; The performing a second beautification process on the intermediate global image based on the mask of the face image to be beautified to obtain the target image includes: Performing a reverse adjustment process on the mask of the face image to be beautified according to the adjustment process information to obtain the mask of the global image to be beautified; A second beautification process is performed on the intermediate global image based on the mask of the global image to be beautified to obtain the target image.

11. The method according to claim 10, characterized in that The step of performing a reverse adjustment process on the mask of the face image to be beautified according to the adjustment process information to obtain the mask of the global image to be beautified comprises: Obtaining an initialization mask corresponding to the global image to be beautified; Performing a reverse adjustment process on the mask of the face image to be beautified according to the adjustment process information to obtain a mask to be replaced; The initialization mask is numerically replaced according to the mask to be replaced to obtain the mask of the global image to be beautified.

12. The method according to claim 11, characterized in that The obtaining of an initialization mask corresponding to the global image to be beautified includes: Skin color detection is performed on the global image to be beautified to obtain a skin mask of the global image to be beautified as the initialization mask.

13. The method according to claim 11, characterized in that The mask of the face image to be beautified includes a facial skin mask of the face image to be beautified and a facial occlusion mask of the face image to be beautified; the mask to be replaced includes a facial skin mask to be replaced and a facial occlusion mask to be replaced; The step of numerically replacing the initialization mask according to the mask to be replaced to obtain the mask of the global image to be beautified includes: The values ​​corresponding to the facial skin area in the initialization mask are replaced according to the facial skin mask to be replaced, and the values ​​corresponding to the facial occluder area in the initialization mask are replaced according to the facial occluder mask to be replaced, so as to obtain the mask of the global image to be beautified.

14. The method according to claim 6, characterized in that Before replacing the face sub-image to be beautified in the global image to be beautified with the intermediate face image, the method further includes: The intermediate face image is beautified and weakened by using the face sub-image to be beautified.

15. The method according to claim 14, characterized in that The step of performing beautification and weakening processing on the intermediate face image by using the face sub-image to be beautified includes: According to the set beautification degree parameter, the face sub-image to be beautified is fused into the intermediate face image.

16. The method according to claim 14, characterized in that The performing beautification and weakening processing on the beautification face sub-image by using the face sub-image to be beautified includes: The high-frequency image of the face sub-image to be beautified is fused to the intermediate face image.

17. The method according to claim 16, characterized in that The method further comprises: When determining the face image to be beautified according to the face sub-image to be beautified, if the face sub-image to be beautified is down-sampled, then the down-sampled face sub-image obtained after the down-sampling is up-sampled to obtain an up-sampled face sub-image, and the up-sampled face sub-image has the same resolution as the face sub-image to be beautified; A high-frequency image of the face sub-image to be beautified is obtained according to a difference between the face sub-image to be beautified and the upsampled face sub-image.

18. The method according to claim 16, characterized in that Before fusing the high-frequency image of the face sub-image to be beautified into the face sub-image to be beautified, the method further includes: determining a defect point in the high frequency image; The pixel values ​​in a preset area around the defect point in the high-frequency image are adjusted to a preset value range.

19. The method according to claim 6, characterized in that When the intermediate face image is used to replace the face sub-image to be beautified in the global image to be beautified, the method further includes: A gradient process is performed on a boundary region between an unreplaced region in the global image to be beautified and the intermediate face image, so that the boundary region forms a smooth transition.

20. An image beautification processing device, characterized in that: include: An image acquisition module is configured to acquire a face image to be beautified; A first beautification processing module is configured to perform a first beautification processing on the face image to be beautified through a pre-trained deep neural network to obtain an intermediate face image, and determine a mask of the face image to be beautified through the deep neural network; A second beautification processing module is configured to perform a second beautification processing on the intermediate face image based on the mask of the face image to be beautified to obtain a target image; The mask of the face image to be beautified is used to indicate position information of at least one preset area in the face image to be beautified; and performing a second beautification process on the intermediate face image based on the mask of the face image to be beautified to obtain a target image includes: At least one to-be processed region in the intermediate facial image is determined based on the mask of the facial image to be beautified, and a second beautification process is performed on the to-beautify region to obtain a target image.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

22. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 19 by executing the executable instructions.

Images