Image beautification processing method, device, storage medium and electronic device

Abstract

The present disclosure provides an image beautification processing method, device, storage medium and electronic device, which relate to the field of image and video processing technology. The image beautification processing method comprises: obtaining an original image to be beautified from a video; matching a face in the original image to be beautified with a face in a reference frame image of the original image to be beautified, and determining a stable bounding box of the face in the original image to be beautified according to the matching result; extracting an original face sub-image from the original image to be beautified based on the stable bounding box of the face in the original image to be beautified; processing the original face sub-image using an image beautification network to obtain a corresponding beautified face sub-image; and generating a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image. The present disclosure improves the problem of inconsistent face beautification effects in multiple consecutive frames of images.

Description

Background Art

[0002] Beauty refers to the use of image processing technology to beautify portraits in images or videos to better meet the user's aesthetic needs.

[0003] When beautifying multiple consecutive frames of images (such as videos), there is often a problem of inconsistent beautification effects for the same face in different frames, resulting in a flickering image and a very discordant visual experience.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides an image beautification processing method, an image beautification processing device, a computer-readable storage medium and an electronic device, thereby improving the problem of inconsistent face beautification effects in multiple consecutive frames of images at least to a certain extent.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, there is provided an image beautification processing method, comprising: acquiring an original image to be beautified from a plurality of consecutive image frames; matching a face in the original image to be beautified with a face in a reference frame image of the original image to be beautified, and determining a stable bounding box of the face in the original image to be beautified according to the matching result; extracting an original face sub-image from the original image to be beautified based on the stable bounding box of the face in the original image to be beautified; processing the original face sub-image by using an image beautification network to obtain a corresponding beautified face sub-image; and generating a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image.

[0008] According to a second aspect of the present disclosure, there is provided an image beautification processing apparatus, comprising: an image acquisition module, configured to acquire an original image to be beautified from a video; a face matching module, configured to match a face in the original image to be beautified with a face in a reference frame image of the original image to be beautified, and determine a stable bounding box of the face in the original image to be beautified according to the matching result; a sub-image extraction module, configured to extract an original face sub-image from the original image to be beautified based on the stable bounding box of the face in the original image to be beautified; a beautification processing module, configured to process the original face sub-image by using an image beautification network to obtain a corresponding beautified face sub-image; and an image generation module, configured to generate a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image beautification processing method of the first aspect and possible implementation methods thereof are implemented.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image beautification processing method of the above-mentioned first aspect and its possible implementation methods by executing the executable instructions.

[0011] The technical solution disclosed in this disclosure has the following beneficial effects:

[0012] Based on the image beautification processing method disclosed in the present invention, on the one hand, by matching the face in the original image to be beautified with the face in the reference frame image, a stable bounding box of the face is determined, and then the original face sub-image is extracted for beautification processing, so that the stable bounding box of the face in the original image to be beautified inherits the relevant information in the reference frame image to a certain extent. Therefore, when processing the original images to be beautified in different frames of multiple continuous images, the faces extracted from the original images to be beautified in different frames have a certain degree of continuity and stability, and will not change drastically, thereby ensuring the consistency of the effect of beautification processing on the face. For example, the faces in the original images to be beautified in different frames have consistency of defects, consistency of facial shape, consistency of skin texture and skin color after beautification processing, which is conducive to improving visual experience. On the other hand, through the image beautification network, blemish removal or other beautification functions can be realized to replace multiple fixed algorithm processes in related technologies, increase the flexibility of image beautification processing, apply to various lighting conditions or skin conditions, improve image beautification effects, and reduce time consumption and memory usage.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0015] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;

[0016] Figure 2A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0017] Figure 3 A flowchart of an image beautification processing method according to the exemplary embodiment is shown;

[0018] Figure 4 A flow chart for determining a stable bounding box in this exemplary embodiment is shown;

[0019] Figure 5 A flowchart of obtaining a beauty face sub-image in this exemplary embodiment is shown;

[0020] Figure 6 A flow chart of combining original face sub-images in this exemplary embodiment is shown;

[0021] Figure 7 A schematic diagram showing a method of combining original face sub-images in this exemplary embodiment;

[0022] Figure 8 A schematic diagram of the structure of an image beautification network in this exemplary embodiment is shown;

[0023] Fig. 9 A schematic diagram showing the structure of another image beautification network in this exemplary embodiment is shown;

[0024] Fig.10 A flowchart of processing a facial image to be beautified by using an image beautification network in this exemplary embodiment is shown;

[0025] Fig.11 A flowchart of training an image beautification network in this exemplary embodiment is shown;

[0026] Fig.12 A schematic diagram showing a training image beautification network in this exemplary embodiment is shown;

[0027] Fig.13 A schematic diagram showing a gradual change process of a boundary area in this exemplary embodiment is shown;

[0028] Fig.14 A schematic flowchart of an image beautification processing method in this exemplary embodiment is shown;

[0029] Fig.15 A schematic structural diagram of an image beautification processing device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0031] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] In the related art, on the one hand, when beautifying multiple consecutive frames of images (such as videos), it is usually necessary to beautify each frame separately. If there is obvious movement of the face between different frames, especially between adjacent frames, it may lead to inconsistent beautification effects on different frames, including but not limited to inconsistencies in blemishes such as eye bags, dark circles, acne, moles, inconsistencies in facial shapes, inconsistencies in skin texture, inconsistencies in noise points, etc., presenting a flickering phenomenon of the picture, affecting the visual experience after beautification. On the other hand, image beautification processing usually includes multiple fixed algorithm processes, such as image feature calculation based on artificial design, spatial filtering processing, layer fusion, etc. However, in actual shooting scenes, complex and diverse lighting conditions may be faced, and the skin conditions of the subjects are diverse. The above method cannot cope with different situations well, resulting in unsatisfactory beautification effects.

[0033] In view of one or more of the above problems, an exemplary embodiment of the present disclosure provides an image beautification processing method. Figure 1 The system architecture and application scenarios of the operating environment of this exemplary embodiment are exemplarily described.

[0034] Figure 1The schematic diagram of the system architecture is shown, and the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides image beautification related services in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to exchange data.

[0035] In one embodiment, the terminal 110 can capture or obtain the image or video to be beautified by other means, and upload it to the server 120. For example, the user opens the beautification App (Application) on the terminal 110, selects the image or video to be beautified from the album, and uploads it to the server 120 for beautification, or the user opens the beautification function in the live broadcast App on the terminal 110, and uploads the real-time collected video to the server 120 for beautification. The server 120 executes the above-mentioned image beautification processing method to obtain the beautified image or video, and returns it to the terminal 110.

[0036] In one implementation, the server 120 may train the image beautification network and send the trained image beautification network to the terminal 110 for deployment. For example, the relevant data of the image beautification network may be packaged in the update package of the above-mentioned beautification App or live broadcast App, so that the terminal 110 may obtain the image beautification network by updating the App and deploy it locally. Furthermore, after the terminal 110 shoots or obtains the image or video to be beautified by other means, it may call the image beautification network to implement the image or video beautification processing by executing the above-mentioned image beautification processing method.

[0037] In one implementation, the terminal 110 may perform training on the image beautification network, for example, by obtaining the basic architecture of the image beautification network from the server 120 and training it with a local data set, or by obtaining a data set from the server 120 and training the locally constructed image beautification network, or by training the image beautification network without relying on the server 120 at all. Furthermore, the terminal 110 may call the image beautification network to implement image or video beautification processing by executing the above-mentioned image beautification processing method.

[0038] As can be seen from the above, the execution subject of the image beautification processing method in this exemplary embodiment may be the terminal 110 or the server 120, and the present disclosure does not limit this.

[0039] The exemplary embodiment of the present disclosure further provides an electronic device for executing the above-mentioned image beautification network training method or image beautification processing method, and the electronic device may be the above-mentioned terminal 110 or server 120. Figure 2 Taking the mobile terminal 200 in the example, the structure of the above electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The construction in can also be applied to fixed type equipment.

[0040] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a USB (Universal Serial Bus) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294 and a SIM (Subscriber Identification Module) card interface 295, etc.

[0041] The processor 210 may include one or more processing units, for example: the processor 210 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image beautification network in this exemplary embodiment may run on a GPU, a DSP or an NPU. The DSP and the NPU usually run the image beautification network with int data (integer type), and the GPU usually runs the image beautification network with float data (floating point type). In comparison, the power consumption of the DSP and the NPU is lower, the response speed is faster, and the precision is lower. The power consumption of the GPU is higher, the response speed is fuller, and the precision is higher. In actual applications, a suitable processing unit can be selected to run the image beautification network according to the device performance and actual needs. For example, when performing real-time beautification on a video, the speed requirement is higher, and a DSP or NPU can be selected to run the image beautification network.

[0042] The encoder can encode (i.e. compress) the image or video data, for example, encode the image or video obtained after the beauty processing to form the corresponding bitstream data to reduce the bandwidth occupied by the data transmission; the decoder can decode (i.e. decompress) the bitstream data of the image or video to restore the image or video data, for example, decode the video to be beautified to obtain the image data of each frame in the video, and extract the original image to be beautified for beauty processing. The mobile terminal 200 can process images or videos in a variety of encoding formats, for example: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding) and other video formats.

[0043] In one implementation, the processor 210 may include one or more interfaces, and may be connected to other components of the mobile terminal 200 via different interfaces.

[0044] The internal memory 221 may be used to store computer executable program codes, which include instructions. The internal memory 221 may include volatile memory and non-volatile memory. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running the instructions stored in the internal memory 221.

[0045] The external memory interface 222 can be used to connect an external memory, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory communicates with the processor 210 through the external memory interface 222 to implement data storage functions, such as storing images, videos and other files.

[0046] The USB interface 230 is an interface that complies with USB standard specifications and can be used to connect a charger to charge the mobile terminal 200, or to connect headphones or other electronic devices.

[0047] The charging management module 240 is used to receive charging input from a charger. While the charging management module 240 is charging the battery 242, it can also power the device through the power management module 241; the power management module 241 can also monitor the status of the battery.

[0048] The wireless communication function of the mobile terminal 200 can be implemented by antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 250 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 200. The wireless communication module 260 can provide WLAN (Wireless Local Area Networks, wireless local area network) (such as Wi-Fi (Wireless Fidelity) network), BT (Bluetooth), GNSS (Global Navigation Satellite System, Global Navigation Satellite System), FM (Frequency Modulation, frequency modulation), NFC (Near Field Communication, short-range wireless communication technology), IR (Infrared, infrared technology) and other wireless communication solutions applied to the mobile terminal 200.

[0049] The mobile terminal 200 can realize the display function and display the user interface through the GPU, the display screen 290 and the AP, etc. For example, when the user performs camera detection, the mobile terminal 200 can display the interface of the camera detection App (Application) on the display screen 290.

[0050] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the encoder, the decoder, the GPU, the display screen 290 and the AP, etc. For example, the user can turn on the image or video shooting function in the hidden camera detection App, and the image of the space to be detected can be collected through the camera module 291.

[0051] The mobile terminal 200 can implement audio functions through the audio module 270, the speaker 271, the receiver 272, the microphone 273, the earphone interface 274 and the AP.

[0052] The sensor module 280 may include a depth sensor 2801 , a pressure sensor 2802 , a gyroscope sensor 2803 , an air pressure sensor 2804 , etc., to implement corresponding sensing detection functions.

[0053] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc. Motor 293 may generate vibration prompts, and may also be used for touch vibration feedback, etc. Buttons 294 include a power button, a volume button, etc.

[0054] The mobile terminal 200 may support one or more SIM card interfaces 295 for connecting to a SIM card to implement functions such as calls and mobile communications.

[0055] Combine the following Figure 3 The image beautification network training method in this exemplary embodiment is described. Figure 3 An exemplary process of the image beautification network training method is shown, which may include:

[0056] Step S310, obtaining the original image to be beautified from a plurality of consecutive frames of images;

[0057] Step S320, matching the human face in the original image to be beautified with the human face in the reference frame image of the original image to be beautified, and determining a stable bounding box of the human face in the original image to be beautified according to the matching result;

[0058] Step S330, extracting an original face sub-image from the original image to be beautified based on a stable bounding box of the face in the original image to be beautified;

[0059] Step S340, using an image beautification network to process the original face sub-image to obtain a corresponding beautified face sub-image;

[0060] Step S350: generating a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image.

[0061] The image beautification network can be trained to implement any combination of one or more beautification functions, including but not limited to blemish removal, deformation, skin color adjustment, skin smoothing, light and shadow adjustment, etc. Figure 3 The image beautification processing method is used as a stage of beautification processing. Figure 3 Before or after the image beautification processing method, other stages of beautification processing are added. For example, the image beautification network is used to remove blemishes from the image. After obtaining the original image to be beautified, Figure 3 The image beautification processing method is used to process the target beautification image, and the obtained target beautification image is the blemish-free beautification image. The blemish-free beautification image can also be subsequently subjected to personalized beautification processing to obtain the final beautification image.

[0062] Generally, blemish removal is necessary for image beautification, and users have a relatively fixed demand for blemish removal. Figure 3The image beautification processing method realizes a universal blemish removal beautification processing flow. In contrast, beauty functions such as skin smoothing, deformation, three-dimensional, skin color adjustment, light and shadow adjustment are not necessary, and the specific needs of users for these beauty functions are also personalized. These beauty functions can be called personalized beauty processing, which usually requires users to make specific settings before processing. For example, the user selects one or more beauty functions and sets parameters such as skin smoothing degree and deformation degree, and then the terminal or server processes according to the user's settings.

[0063] It should be noted that this disclosure is for Figure 3 There is no restriction on the order of image beautification processing and other beautification processing. For example, the image to be processed can be first subjected to personalized beautification processing to obtain an intermediate beautification image, and then the intermediate beautification image is used as the original image to be beautified. Figure 3 The target beautified image is the final output beautified image.

[0064] Based on the above-mentioned image beautification processing method, on the one hand, by matching the face in the original image to be beautified with the face in the reference frame image, a stable bounding box of the face is determined, and then the original face sub-image is extracted for beautification processing, so that the stable bounding box of the face in the original image to be beautified inherits the relevant information in the reference frame image to a certain extent. Therefore, when processing the original images to be beautified in different frames of multiple continuous images, the faces extracted from the original images to be beautified in different frames have a certain degree of continuity and stability, and will not change drastically, thereby ensuring the consistency of the effect of beautification processing on the face. For example, the faces in the original images to be beautified in different frames have consistency of defects, consistency of facial shape, consistency of skin texture and skin color after beautification processing, which is conducive to improving visual experience. On the other hand, the image beautification network can realize defect removal or other beautification functions to replace multiple fixed algorithm processes in related technologies, increase the flexibility of image beautification processing, apply to various lighting conditions or skin conditions, improve image beautification effects, and reduce time consumption and memory usage.

[0065] Below Figure 3 Provide detailed instructions for each step.

[0066] refer to Figure 3 In step S310, the original image to be beautified is obtained from a plurality of consecutive image frames.

[0067] Among them, the continuous multi-frame images can be videos, or they can be images taken continuously, etc. The continuous multi-frame images are objects that need to be beautified. Taking video as an example, it can be a video stream that is currently shot or received in real time, or it can be a complete video that has been shot or received, such as a video stored locally. The present disclosure does not limit parameters such as the frame rate and image resolution of the video. For example, the video frame rate can be 30fps (frames per second), 60fps, 120fps, etc., and the image resolution can be 720P, 1080P, 4K, etc. and corresponding different aspect ratios.

[0068] In this exemplary embodiment, each frame of the original image in the video may be beautified, or a portion of the original images may be selected from the video for beautification. The original images to be beautified are referred to as original images to be beautified.

[0069] In one implementation, at least two frames of original images to be beautified may be obtained from the video. For example, the original image containing the target face may be used as the original image to be beautified, or an alternate frame strategy may be adopted to obtain an original image to be beautified at a certain interval.

[0070] In one implementation, when a video stream is received in real time, each frame of the received original image may be used as the original image to be beautified.

[0071] Continue to refer Figure 3 In step S320, the face in the original image to be beautified is matched with the face in the reference frame image of the original image to be beautified, and a stable bounding box of the face in the original image to be beautified is determined according to the matching result.

[0072] A bounding box refers to an area in an image that surrounds a face and has a certain geometric shape. The present disclosure does not limit the shape of the bounding box, and it can be any shape such as a rectangle or a trapezoid. The bounding box of the face area initially detected is called a basic bounding box, which can be, for example, the smallest bounding box containing the face, or a face frame obtained by a related algorithm. The basic bounding box is optimized, such as expanded, position corrected, etc., and the optimized bounding box is called a stable bounding box.

[0073] In this exemplary embodiment, face detection can be performed on the original image to be beautified to obtain relevant information about the face. The present disclosure does not limit the face detection algorithm. For example, a specific neural network can be used to detect facial key points, including key points of the facial boundary, and a basic bounding box of the face is generated based on the key points of the facial boundary, and a stable bounding box is obtained through optimization.

[0074] The reference frame image can be any frame image in the continuous multiple frames of images for which a stable bounding box of a face has been determined or for which beautification processing has been completed. For example, when performing frame-by-frame beautification processing on a video, the previous frame image of the original image to be beautified can be used as the reference frame image. By matching the original image to be beautified with the face in the reference frame image, the stable bounding box of the face in the original image to be beautified can be determined based on the stable bounding box of the face in the reference frame image.

[0075] In one embodiment, reference Figure 4 As shown, the above-mentioned matching of the face in the original image to be beautified with the face in the reference frame image of the original image to be beautified, and determining the stable bounding box of the face in the original image to be beautified according to the matching result may include the following steps S410 to S430:

[0076] Step S410 , detecting a human face in the original image to be beautified, recording it as a to-be determined human face, and matching the to-be determined human face with a determined human face in a reference frame image of the original image to be beautified.

[0077] Among them, the face to be determined refers to a face that needs to be beautified but has not had a stable bounding box determined, which can be regarded as a face with an unknown identity. The determined face refers to a face with a stable bounding box determined, which can be regarded as a face with a known identity. The faces with determined stable bounding boxes in the reference frame image are all determined faces. Correspondingly, the faces detected in the original image to be beautified are faces with no stable bounding boxes determined, that is, faces to be determined. By matching the faces to be determined in the original image to be beautified with the determined faces in the reference frame image, it can be inferred that there is a correlation between the stable bounding box of the face to be determined and the stable bounding box of the determined face that matches the face to be determined, and the stable bounding box of the face to be determined can be determined accordingly.

[0078] Generally, all faces can be detected in the original image to be beautified by the face detection algorithm, which may include faces that do not need beautification (such as the faces of passers-by in the distance). Considering that in the scene of image beautification, it is usually necessary to beautify larger faces (the effect of beautification on smaller faces is not obvious), the detected faces can be filtered by the face area threshold. In one embodiment, the face area threshold can be set according to experience or the size of the original image to be beautified. Exemplarily, the face area threshold can be the size of the original image to be beautified * 0.05; if the area of ​​the basic bounding box of the face is greater than or equal to the face area threshold, it is a face that needs beautification, and the basic bounding box and other information of the face can be retained, or the face can be recorded as a face to be determined; if the area of ​​the basic bounding box of the face is less than the face area threshold, it is a face that does not need beautification, and the basic bounding box and other related information of the face can be deleted without subsequent processing.

[0079] In one embodiment, in order to facilitate the subsequent processing of the original face sub-images, such as combining the original face sub-images, or considering the limitations of device performance, an upper limit on the number of original face sub-images can be set, that is, an upper limit on the number of faces to be determined can be set. For example, it can be set to 4. If the number of retained faces is greater than 4 after filtering by the above-mentioned face area threshold, 4 faces to be determined can be further screened out, such as the 4 faces with the largest area, or the 4 faces closest to the center of the original image to be beautified. In this way, 4 original face sub-images are correspondingly intercepted in the subsequent processing, and no subsequent processing is performed on other faces. Alternatively, multiple beautification processes can be performed. In this process, 4 faces are selected as faces to be determined, and their corresponding original face sub-images are intercepted and beautified. In the next process, other faces are selected as faces to be determined, and their corresponding original face sub-images are intercepted for beautification processing, thereby completing the beautification processing of all faces in the basic bounding box whose area is greater than the face area threshold in the image to be processed.

[0080] In one implementation, in order to facilitate tracking and identifying faces in multiple consecutive frames of images, an ID (Identity Document) may be assigned to each face. For example, starting from the first frame, an ID is assigned to each face; after a face is detected in each subsequent frame, each face is matched with the face in the previous frame; if the match is successful, the face ID and other related information in the previous frame are inherited; if the match is unsuccessful, it is treated as a new face and a new ID is assigned.

[0081] The present disclosure does not limit the method of matching the face to be determined with the determined face. For example, a face recognition algorithm can be used to identify and compare each face to be determined with each determined face. If the similarity is higher than a preset similarity threshold, it is determined that the face to be determined and the determined face are matched successfully.

[0082] In one embodiment, whether the face to be determined matches the determined face successfully can be determined based on the overlap (Intersection Over Union, IOU, also known as intersection over union ratio) between the basic bounding box of the face to be determined and the basic bounding box of the determined face. An exemplary method for calculating the overlap is provided below:

[0083] Get the position of the basic bounding box of the face to be determined in the original image to be beautified, and the position of the basic bounding box of the determined face in the reference frame image, count the number of pixels in the two basic bounding boxes that overlap, denoted as k1, and the number of pixels that do not overlap, denoted as k2 (indicating the number of pixels in the basic bounding box of the face to be determined that do not overlap with the basic bounding box of the determined face) and k3 (indicating the number of pixels in the basic bounding box of the determined face that do not overlap with the basic bounding box of the undetermined face), then the overlap degree of the two basic bounding boxes is:

[0084]

[0085] After determining the overlap, if the overlap reaches a preset overlap threshold, it is determined that the to-be-determined face successfully matches the determined face. The overlap threshold can be set based on experience and actual needs, for example, can be set to 0.75.

[0086] In addition, the basic bounding box of the face to be determined and any one of the basic bounding boxes of the determined face can be iteratively transformed using an ICP (Iterative Closest Point) algorithm, and the overlap of the two basic bounding boxes can be calculated based on the number of pixels with the same pixel values ​​and the number of pixels with different pixel values ​​in the basic bounding box of the face to be determined and the basic bounding box of the determined face after the final transformation, thereby determining whether the match is successful.

[0087] It should be noted that, since there may be multiple faces to be determined in the original image to be beautified and multiple determined faces in the reference frame image, each face to be determined can be matched with each determined face separately to obtain a similarity matrix or an overlap matrix, and then the Hungarian algorithm can be used to achieve the global maximum match, and then whether the match is successful is determined based on the similarity or overlap between each face to be determined and the determined face.

[0088] Step S420: If the face to be determined does not match the determined face, the basic bounding box of the face to be determined is expanded according to the first preset parameter to obtain a stable bounding box of the face to be determined;

[0089] If the face to be determined is not matched with the determined face, it means that the face to be determined is a new face in the continuous multi-frame image, and reference information cannot be obtained from the reference frame image. Therefore, the basic bounding box of the face to be determined can be appropriately expanded to obtain a stable bounding box. The first preset parameter is the expansion parameter of the basic bounding box for the new face, which can be determined according to experience or actual needs. For example, the width and height of the basic bounding box can be expanded by 1 / 4.

[0090] Suppose the basic bounding box of the face to be determined is represented as [bb0, bb1, bb2, bb3], where bb0 is the abscissa of the upper-left point of the basic bounding box, bb1 is the ordinate of the upper-left point of the basic bounding box, bb2 is the abscissa of the lower-right point of the basic bounding box, and bb3 is the ordinate of the lower-right point of the basic bounding box. The width of this basic bounding box is w, and the height is h. Note that the pixel coordinates in the image usually have the upper-left point of the image as (0, 0) and the lower-right point as (W, H), where W and H represent the width and height of the image. Therefore, bb0 < bb2 and bb1 < bb3. Let E1 represent the first preset parameter. When the basic bounding box is centered and expanded (i.e., evenly expanded in all four directions) according to the first preset parameter, the size of the stable bounding box can be obtained as follows:

[0091]

[0092] where expand_w and expand_h are the width and height of the stable bounding box of the face to be determined, respectively. It should be noted that if the expanded width expand_w exceeds the width W of the original image to be beautified, then expand_w = W; if the expanded height expand_h exceeds the height H of the original image to be beautified, then expand_h = H.

[0093] The center point coordinates of the stable bounding box are equal to the center point coordinates of the basic bounding box, that is:

[0094]

[0095] where center_x represents the x coordinate of the center point of the stable bounding box of the face to be determined, and center_y represents the y coordinate of the center point of the stable bounding box of the face to be determined.

[0096] Then the coordinates of the upper-left and lower-right points of the stable bounding box can be calculated as follows:

[0097]

[0098] where expand_bb0 is the abscissa of the upper-left point of the stable bounding box, expand_bb1 is the ordinate of the upper-left point of the stable bounding box, expand_bb2 is the abscissa of the lower-right point of the stable bounding box, and expand_bb3 is the ordinate of the lower-right point of the stable bounding box. Thus, the stable bounding box of the face to be determined is obtained. If the calculated coordinates exceed the boundaries of the original image to be beautified, the boundary coordinates of the original image to be beautified are used to replace the coordinates that exceed the boundaries. Finally, the expanded bounding box can be represented in the form of [expand_bb0, expand_bb1, expand_bb2, expand_bb3].

[0099] It should be added that the above coordinates are usually pixel coordinates in the image, which are integers. Therefore, when calculating, float data can be used for calculation, and then rounded, and the result is saved as int data. Exemplarily, when division operations are involved, float data is used for calculation and intermediate results are cached. When calculating the final result (including the above expand_w, expand_h, center_x, center_y, expand_bb0, expand_bb1, expand_bb2, expand_bb3), rounding is performed and saved as int data.

[0100] For the center point coordinates, since saving int data will affect the accuracy of subsequent processing of other frames, int and float data can be saved. For example, the result calculated in formula (3) is saved as float data, as shown below:

[0101]

[0102] Among them, center_x_float and center_y_float represent the center point coordinates stored in float type data, center_x and center_y represent the center point coordinates stored in int type data, and int() represents rounding operation.

[0103] Furthermore, to ensure the accuracy of the results, formula (4) can be changed to the following calculation method:

[0104]

[0105] Step S430: If the face to be determined matches the determined face successfully, a stable bounding box of the face to be determined is determined based on the stable bounding box of the determined face.

[0106] Generally, the face to be determined in the original image to be beautified will not change much compared with the determined face in the matching reference frame image, which is reflected in that both the position change and the size change will not be too large. Therefore, based on the stable bounding box of the determined face, appropriate position and size changes can be made to obtain a stable bounding box of the face to be determined.

[0107] In one implementation, the stable bounding box of the determined face can be changed in position and size according to the position change parameters and size change parameters of the basic bounding box of the face to be determined relative to the basic bounding box of the determined face, to obtain the stable bounding box of the face to be determined.

[0108] In one implementation, the above-mentioned determining the stable bounding box of the face to be determined based on the stable bounding box of the determined face may include the following steps:

[0109] Based on a preset stability coefficient, the center point coordinates of the stable bounding box of the determined face and the center point coordinates of the basic bounding box of the face to be determined are weighted to obtain the center point coordinates of the stable bounding box of the face to be determined.

[0110] The above steps represent the fusion of the position of the stable bounding box of the determined face and the position of the basic bounding box of the face to be determined as the position of the stable bounding box of the face to be determined. During the fusion, the center point coordinates of the two are weighted using a preset stability coefficient. The preset stability coefficient can be the weight of the stable bounding box of the determined face, which can be determined based on experience or actual scenarios. In general, the faster the face moves, the smaller the preset stability coefficient. For example, in a live broadcast scenario, the face usually moves very little within a certain range. The preset stability coefficient can be set to 0.9, and the center point coordinates of the stable bounding box of the face to be determined are calculated as follows:

[0111]

[0112] Where pre_center_x represents the x-coordinate of the center point of the stable bounding box of the determined face, and pre_center_y represents the y-coordinate of the center point of the stable bounding box of the determined face. It can be seen that formula (7) represents the weight of the center point coordinate of the stable bounding box of the determined face as 0.9 and the weight of the center point coordinate of the basic bounding box of the face to be determined as 0.1, and the two center point coordinates are weighted to obtain the center point coordinate of the stable bounding box of the face to be determined.

[0113] Similar to the above formula (5), the coordinates of the data center points of int type and float type can be saved, then:

[0114]

[0115] Among them, pre_center_x_float is the saved float data of pre_center_x, and pre_center_y_float is the saved float data of pre_center_y.

[0116] The above-mentioned weighted method for calculating the center point coordinates essentially adopts a mechanism for momentum updating of the center point coordinates, which can avoid excessive movement of the center point coordinates of the stable bounding box of the same face from the reference frame image to the original image to be beautified, so as to cause jitter in the subsequently captured original face sub-image and affect the beautification effect.

[0117] In one implementation, determining the stable bounding box of the face to be determined based on the stable bounding box of the determined face may include the following steps:

[0118] If the size of the basic bounding box of the face to be determined is greater than the product of the size of the stable bounding box of the determined face and the first magnification factor, expand the size of the stable bounding box of the determined face according to the second preset parameter to obtain the size of the stable bounding box of the face to be determined;

[0119] If the size of the basic bounding box of the face to be determined is less than the product of the size of the stable bounding box of the determined face and the second magnification factor, reduce the size of the stable bounding box of the determined face according to the third preset parameter to obtain the size of the stable bounding box of the face to be determined; the first magnification factor is greater than the second magnification factor;

[0120] If the size of the basic bounding box of the face to be determined is less than the product of the size of the stable bounding box of the determined face and the first magnification factor and greater than the product of the size of the stable bounding box of the determined face and the second magnification factor, take the size of the stable bounding box of the determined face as the size of the stable bounding box of the face to be determined.

[0121] The above steps indicate that according to the comparison result between the size of the basic bounding box of the face to be determined and the size of the stable bounding box of the determined face, calculations are performed in three cases respectively. The first magnification factor and the second magnification factor can be integer magnification factors or non-integer magnification factors. In one implementation, the first magnification factor is greater than or equal to 1, and the second magnification factor is less than 1. Exemplarily, the first magnification factor can be 1, and the second magnification factor can be 0.64.

[0122] When performing calculations, the width and height can be compared and calculated separately. For example, if the comparison result of the width belongs to the first case above and the comparison result of the height belongs to the second case, the width and height of the stable bounding box of the face to be determined are calculated in the two cases respectively.

[0123] Assume that the first magnification factor is t1 and the second magnification factor is t2, and the calculation of the width is described as follows:

[0124] First case. If w > pre_expand_w·t1, and E2 represents the second preset parameter, then:

[0125] expand_w = pre_expand_w + pre_expand_w·E2 (9)

[0126] Second case. If w < pre_expand_w·t2, and E3 represents the third preset parameter, then:

[0127] expand_w = pre_expand_w - pre_expand_w·E3 (10)

[0128] For the third case, if pre_expand_w·t2 < w < pre_expand_w·t1, then we have:

[0129] expand_w = pre_expand_w (11)

[0130] For the height, the same three cases can be calculated respectively to obtain expand_h.

[0131] Generally, in consecutive multiple frames of video images, as long as the face does not approach the camera quickly, does not move away from the camera quickly, or does not move out of the frame, the size of the face will not change drastically, then the above third case is satisfied. At this time, the size of the stable bounding box of the face to be determined is equal to the size of the stable bounding box of the face that has been determined, that is, the size of the stable bounding box remains unchanged. The first and second cases above are both cases where the size of the face changes drastically. In the first case, the face becomes much larger. At this time, the size of the stable bounding box of the face that has been determined is appropriately enlarged according to the second preset parameter to obtain the size of the stable bounding box of the face to be determined. The second preset parameter can be determined according to experience and the actual scenario. In the second case, the face becomes much smaller. At this time, the size of the stable bounding box of the face that has been determined is appropriately reduced according to the third preset parameter to obtain the size of the stable bounding box of the face to be determined. The third preset parameter can be determined according to experience and the actual scenario.

[0132] If the expanded width expand_w exceeds the width W of the original image to be beautified, then expand_w = W; if the expanded height expand_h exceeds the height H of the original image to be beautified, then expand_h = H.

[0133] Through the calculations of the above three cases, it is possible to avoid excessive changes in the size of the stable bounding box of the same face from the reference frame image to the original image to be beautified, so as to prevent the original face sub-image intercepted subsequently from jittering and affecting the beautification effect.

[0134] After obtaining the center point coordinates and size of the stable bounding box of the face to be determined respectively, the coordinates of the upper left point and the lower right point of the stable bounding box can be calculated. If the calculated coordinates exceed the boundary of the original image to be beautified, then the boundary coordinates of the original image to be beautified are used to replace the coordinates that exceed the boundary. Finally, the stable bounding box can be expressed in the form of [expand_bb0, expand_bb1, expand_bb2, expand_bb3].

[0135] From the above, it can be seen that when the face to be determined is successfully matched with the determined face, the stable bounding box of the face to be determined is determined according to the stable bounding box of the determined face, so that the face to be determined inherits the information of the stable bounding box of the determined face to a certain extent, thereby ensuring that the stable bounding box of the face between different frame images has a certain continuity and stability, and will not undergo drastic changes in position or size, thereby ensuring the consistency of the face beauty effect during subsequent beauty processing, and preventing the face after beauty from flickering due to drastic changes in the face.

[0136] In one implementation, after obtaining the stable bounding box of the face to be determined, the relevant parameters of the stable bounding box can be saved, and the face to be determined can be marked as a determined face for use in matching the face to be determined and determining the stable bounding box in subsequent frames.

[0137] Continue to refer Figure 3 In step S330, based on the stable bounding box of the face in the original image to be beautified, an original face sub-image is extracted from the original image to be beautified.

[0138] The image of the stable bounding box portion is intercepted from the original image to be beautified to obtain the original face sub-image. When the original image to be beautified includes the stable bounding boxes of multiple faces, the original face sub-image corresponding to each face can be intercepted.

[0139] Continue to refer Figure 3 In step S340, the original face sub-image is processed using the image beautification network to obtain a corresponding beautified face sub-image.

[0140] When extracting multiple original face sub-images from the original image to be beautified, each original face sub-image can be input into the image beautification network to obtain the beautified face sub-image corresponding to each original face sub-image, or multiple original face sub-images can be combined and input into the image beautification network for processing. Figure 5 As shown, the above-mentioned processing of the original face sub-image by using the image beautification network to obtain the corresponding beautified face sub-image may include the following steps S510 to S530:

[0141] Step S510 , combining original face sub-images extracted from the original image to be beautified based on the input image size of the image beautification network to generate a face image to be beautified.

[0142] The input image size is the image size that matches the input layer of the image beautification network. The present disclosure does not limit the size and aspect ratio of the input image size. For example, the ratio of the long side to the short side of the input image size can be set to be close to

[0143] In one embodiment, the image beautification network can be a fully convolutional network, which can process images of different sizes. In this case, the image beautification network has no requirements for the input image size, and the size has an impact on the amount of calculation, memory usage, and beautification precision. The input image size can be determined according to the beautification precision set by the user or the performance of the device. As a result, the image beautification network can be deployed on devices with different performances such as high, medium, and low, and has a wide range of applications. There is no need to deploy different image beautification networks for different devices, which reduces the training cost of the network. For example, considering that it is suitable for lightweight computing on a mobile terminal, the input image size can be determined as a smaller value, such as 640 in width and 448 in height.

[0144] The input image size of the image beautification network determines the clarity of the facial image to be beautified. When the clarity is low, it is not suitable for certain beautification functions. For example, for tiny blemishes such as extremely small moles and dry lip lines, they occupy fewer pixels in the facial image to be beautified. If they are removed, they may be removed inaccurately, affecting the surrounding skin texture or causing flickering. Therefore, according to the actual deployment environment of the image beautification network, the input image size of the image beautification network is determined, and then the beautification function of the image beautification network is determined, and a beauty image dataset is constructed and trained. Exemplarily, when the input image size of the image beautification network is less than 448 in width and 320 in height, the image beautification network can be set to not include the function of removing tiny blemishes.

[0145] In this exemplary embodiment, when multiple frames of original images to be beautified are obtained, the original facial sub-images captured from each frame of the original images to be beautified can be combined respectively, so that one frame of the original image to be beautified corresponds to one frame of the facial image to be beautified. For example, the video can be beautified frame by frame, and the original facial sub-images captured from each frame of the image are combined in turn to obtain each frame of the facial image to be beautified corresponding to each frame of the image. The original facial sub-images captured from multiple frames of original images to be beautified can also be combined to generate one or more frames of facial images to be beautified corresponding to the multiple frames of original images to be beautified. For example, multiple frames in the video can be merged and beautified, and the original facial sub-images captured from multiple consecutive frames of images can be arbitrarily combined to match the above-mentioned input image size, such as capturing two original facial sub-images from two consecutive frames of images, and combining these four original facial sub-images into one frame of the facial image to be beautified.

[0146] After obtaining the input image size, the original face sub-images need to be combined into a face image of the size to be beautified. The specific combination method is related to the number of original face sub-images. Figure 6As shown, the input image size based on the image beautification network combines the one or more original face sub-images to generate the face image to be beautified, which may include the following steps S610 to S630:

[0147] Step S610, dividing the input image size into sub-image sizes corresponding to the original face sub-images in one-to-one correspondence according to the number of original face sub-images;

[0148] Step S620, transforming the corresponding original face sub-image based on the size of each sub-image;

[0149] Step S630: Combining the transformed original face sub-images to generate a face image to be beautified.

[0150] Combine the following Figure 7 Give an example. Figure 7 Q represents the number of original face sub-images. Figure 7 The exemplary modes of input image size segmentation and image combination when Q is 1 to 4 are shown respectively. Assuming that the input image size is 640*448 in width and Q is 1, the sub-image size is also 640*448 in width; when Q is 2, the sub-image size is half of the input image size, that is, 320*448 in width; when Q is 3, the sub-image sizes are 0.5, 0.25, and 0.25 of the input image size, that is, 320*448 in width, 320*224 in height, and 320*224 in width; when Q is 4, the sub-image sizes are all 0.25 of the input image size, that is, 320*224 in width. Each original face sub-image is transformed to be consistent with the sub-image size. It should be noted that when the sub-image sizes are inconsistent, such as when Q is 3, the original face sub-images can be matched with the sub-image sizes one by one according to the size order of the original face sub-images and the size order of the sub-images, that is, the largest original face sub-image corresponds to the largest sub-image size, and the smallest original face sub-image corresponds to the smallest sub-image size. After the original face sub-images are transformed, the transformed original face sub-images are matched according to Fig.10 The above steps are combined in the manner shown to generate a face image to be beautified.

[0151] In one embodiment, when Q is an even number, the input image size can be divided into Q equal parts to obtain Q sub-images of the same size. Specifically, Q can be decomposed into the product of two factors, i.e., Q = q1*q2, so that the ratio of q1 / q2 is the same as the aspect ratio of the input image size (e.g. ) as close as possible, divide the width of the input image size into q1 and the height into q2. When Q is an odd number, divide the input image size into Q+1 equal parts to obtain Q+1 identical sub-image sizes, merge two of the sub-image sizes into one sub-image size, and keep the remaining Q-1 sub-images unchanged, thus obtaining Q sub-image sizes.

[0152] In another embodiment, the size ratio (or area ratio) of the original face sub-image may be calculated first, such as S1:S2:S3:…:S Q , and then divide the input image size into Q sub-image sizes according to this ratio.

[0153] After determining the sub-image size corresponding to each original face sub-image, the original face sub-image may be transformed based on the sub-image size. In one embodiment, the transformation of the original face sub-image may include any one or more of the following:

[0154] ① When the size relationship between the width and height of the original face sub-image is different from the size relationship between the width and height of the sub-image size, rotate the original face sub-image 90 degrees. Specifically, in the original face sub-image and the sub-image size, if the width is greater than the height or the width is less than the height, then the size relationship between the width and height of the original face sub-image and the sub-image size is the same, and there is no need to rotate the original face sub-image; otherwise, the size relationship between the width and height of the original face sub-image and the sub-image size is different, and the original face sub-image needs to be rotated 90 degrees (either clockwise or counterclockwise). For example, when the sub-image size is 320 wide and 448 high, that is, the width is less than the height, if the original face sub-image is wider than the height, then rotate the original face sub-image 90 degrees.

[0155] In one implementation, in order to maintain the angle of the face in the original face sub-image, the original face sub-image may not be rotated.

[0156] ② When the size of the original face sub-image is larger than the sub-image size, the original face sub-image is downsampled according to the sub-image size. The size of the original face sub-image is larger than the sub-image size, which means that the width of the original face sub-image is larger than the width of the sub-image size, or the height of the original face sub-image is larger than the height of the sub-image size. In the image beautification scenario, the image to be processed is generally a clear image taken by the terminal device, and its size is large. Therefore, it is a common situation that the size of the original face sub-image is larger than the sub-image size, that is, the original face sub-image usually needs to be downsampled.

[0157] Down sampling can be achieved by using methods such as bilinear interpolation and nearest neighbor interpolation, which is not limited in the present disclosure.

[0158] After downsampling, at least one of the width and height of the original face sub-image is aligned with the sub-image size, including the following cases:

[0159] The width and height of the original face sub-image are the same as the sub-image size;

[0160] The width of the original face sub-image is the same as the width of the sub-image size, and the height is smaller than the height of the sub-image size;

[0161] The height of the original face sub-image is the same as the height of the sub-image size, and the width is smaller than the width of the sub-image size.

[0162] It should be noted that if the original face sub-image has been rotated as described above to obtain a rotated original face sub-image, then when the size of the original face sub-image is larger than the sub-image size, it is downsampled according to the sub-image size. The specific implementation method is the same as the downsampling method of the original face sub-image described above, and will not be repeated herein.

[0163] On the contrary, when the size of the original face sub-image (or the rotated original face sub-image) is smaller than or equal to the sub-image size, the downsampling processing step may not be performed.

[0164] ③ When the size of the original face sub-image is smaller than the sub-image size, the original face sub-image is padded according to the difference between the original face sub-image and the sub-image size, so that the size of the padded original face sub-image is equal to the sub-image size. The size of the original face sub-image is smaller than the sub-image size, which means that at least one of the width and height of the original face sub-image is smaller than the sub-image size, and the other is not larger than the sub-image size, which specifically includes the following situations:

[0165] The width of the original face sub-image is smaller than the width of the sub-image size, and the height is also smaller than the height of the sub-image size;

[0166] The width of the original face sub-image is smaller than the width of the sub-image size, and the height is equal to the height of the sub-image size;

[0167] The height of the original face sub-image is smaller than the height of the sub-image size, and the width is equal to the height of the sub-image size.

[0168] Preset pixel values ​​can be used for filling, which are usually pixel values ​​that are very different from the color of the face, such as (R0, G0, B0), (R255, G255, B255), etc.

[0169] Generally, the original face sub-image can be filled around, for example, the center of the original face sub-image is overlapped with the center of the sub-image size, and the difference part around the original face sub-image is filled, so that the size of the original face sub-image after filling is consistent with the sub-image size. Of course, the original face sub-image can also be aligned with one side edge of the sub-image size, and the other side is filled. This disclosure does not limit this.

[0170] It should be noted that if the original face sub-image has been subjected to at least one of the above-mentioned rotation and downsampling processing to obtain the original face sub-image that has been subjected to at least one of the rotation and downsampling processing, then when the size of the original face sub-image is smaller than the sub-image size, it is filled according to the difference between the size of the original face sub-image and the sub-image size. The specific implementation method is the same as the filling method of the above-mentioned original face sub-image, and therefore it will not be repeated here.

[0171] The above ① to ③ are three commonly used transformation methods, and any one or more of them can be used according to actual needs. For example, ①, ②, and ③ are used to process each original face sub-image in sequence, and the processed original face sub-images are combined into a face image to be beautified.

[0172] In the above transformation, the direction and size of the original face sub-image are changed to facilitate the unified processing of the image beautification network. The beautified image needs to be inversely transformed to restore it to the same direction and size as the original face sub-image to adapt to the size of the original image to be beautified. Therefore, the corresponding transformation information can be saved, including but not limited to: the direction and angle of rotation of each original face sub-image, the downsampling ratio, and the coordinates of the filled pixels. This facilitates the subsequent inverse transformation based on the transformation information.

[0173] After the transformed original face sub-images are combined, the combination information can be saved, including but not limited to the size of each original face sub-image (i.e., the corresponding sub-image size) and the position in the face image to be beautified, and the arrangement and order of each original face sub-image. The beautified face combination image can be subsequently split according to the combination information to obtain each individual beautified face sub-image.

[0174] Step S520: Process the beautified face image using the image beautification network to obtain a corresponding beautified face image.

[0175] In this exemplary embodiment, an image beautification network of any structure can be set according to actual needs. Generally, the input and output of the image beautification network are both images, so an end-to-end structure can be adopted, such as a fully convolutional network. Considering that image beautification processing requires more in-depth feature mining and learning of images, the image beautification network can adopt a deep neural network (DNN), which can reduce the number of parameters by increasing the number of network layers (i.e., network depth), and can learn the deep features of the image to achieve pixel-level processing.

[0176] Figure 8 FIG. 1 shows a schematic structural diagram of an image beautification network, which may adopt a U-Net structure. For example, after the face image to be beautified is input into the image beautification network, the convolution layer 1 performs one or more convolution operations ( Figure 4 It is shown in the figure that convolution layer 1 performs two convolution operations, and the present disclosure does not limit the specific number of convolution operations in each convolution layer), and then undergoes a pooling operation to obtain a feature image with a reduced size; convolution layer 2 performs another round of convolution and pooling operations to obtain a feature image with a further reduced size; convolution layer 3 performs another round of convolution and pooling operations to obtain a feature image with an even smaller size; convolution operation is performed in convolution layer 4, but no pooling operation is performed; then, transposed convolution layer 1 is entered, a transposed convolution operation is first performed, and then it is spliced ​​with the feature image in convolution layer 3, and then one or more convolution operations are performed to obtain a feature image with an increased size; transposed convolution layer 2 performs another round of transposed convolution operation, splicing with the feature image in convolution layer 2, and convolution operation to obtain a feature image with a further increased size; finally, transposed convolution layer 3 performs another round of the above operations to output a beauty face image. It should be noted that the present disclosure does not limit the number of convolutional layers and transposed convolutional layers in the image beautification network. According to the actual scene requirements, other types of intermediate layers can also be added to the image beautification network, such as pixel rearrangement layers, Dropout layers (discarding layers), fully connected layers, etc.

[0177] In one embodiment, the image beautification network may be a fully convolutional network, including: a first pixel rearrangement layer, at least one convolutional layer, at least one transposed convolutional layer, and a second pixel rearrangement layer. The structure of the network may refer to Fig. 9 As shown. Figure 8 Compared with the network structure of , two pixel rearrangement layers are mainly added. Fig. 9 The image beautification network shown above uses the image beautification network to process the beautified face image to obtain the corresponding beautified face image, which may include Fig.10 Steps S1010 to S1040 in:

[0178] Step S1010: using a first pixel rearrangement layer to perform pixel rearrangement processing on the face image to be beautified from a single channel to multiple channels to obtain a first feature image.

[0179] It should be noted that the facial image to be beautified may be a single-channel image (such as a grayscale image) or a multi-channel image (such as an RGB image). The first pixel rearrangement layer may rearrange each channel of the facial image to be beautified into multiple channels.

[0180] In an optional implementation, step S1010 includes:

[0181] Inputting the face image to be beautified with a channel number a into the first pixel rearrangement layer;

[0182] The pixels in each n*n neighborhood in each channel of the facial image to be beautified are rearranged to the same positions in the n*n channels, and a first feature image with a*n*n channel number is output.

[0183] Where a represents the number of channels of the face image to be beautified, which is a positive integer; n represents the parameter of pixel rearrangement, which is a positive integer not less than 2. Taking n=2 as an example, traverse the first channel of the face image to be beautified, usually starting from the upper left corner, extract the pixels of each 2*2 grid, and rearrange them to the same position in the four channels, thereby turning one channel into four channels, and at the same time reducing the width and height of the image to half, and record the rearranged image as the first feature image; the same method is used to process other channels. If the face image to be beautified is a single-channel image, the first feature image of the four channels is obtained after the pixels are rearranged; if the face image to be beautified is a three-channel image, the first feature image of the twelve channels is obtained after the pixels are rearranged.

[0184] The first pixel rearrangement layer can be implemented by using the space_to_depth function in TensorFlow (a machine learning implementation framework) to convert the spatial features in the face image to be beautified into depth features. It can also be implemented by using a convolution operation with a step size of n. In this case, the first pixel rearrangement layer can be regarded as a special convolution layer.

[0185] Step S1020: Perform convolution processing on the first feature image using a convolution layer to obtain a second feature image.

[0186] The present disclosure does not limit the number of convolutional layers, the size of convolutional kernels, the specific structure of convolutional layers, etc. The convolutional layer is used to extract image features from different scales and learn depth information. The convolutional layer may include a matching pooling layer for downsampling the convolutional image to achieve information abstraction, increase the receptive field, and reduce parameter complexity.

[0187] When multiple convolutional layers are set, a stepwise convolution and downsampling method can be used. For example, the image can be reduced by a factor of 2 until the last convolutional layer outputs a second feature image. The second feature image can be the feature image with the smallest size in the image beautification network processing process.

[0188] Step S1030: Use a transposed convolution layer to perform a transposed convolution process on the second feature image to obtain a third feature image.

[0189] The present disclosure does not limit the number of transposed convolution layers, the size of the transposed convolution kernel, the specific structure of the transposed convolution layer, etc. The transposed convolution layer is used to upsample the second feature image, which can be regarded as the reverse process of convolution, thereby restoring the size of the image.

[0190] When multiple transposed convolutional layers are set, a stepwise upsampling method may be used, for example, the image may be upsampled by a factor of 2 until the last transposed convolutional layer outputs the third feature image.

[0191] In an optional implementation, the convolution layer and the transposed convolution layer are completely symmetrical structures, and the third feature image has the same size and the same number of channels as the first feature image.

[0192] In an optional implementation, a direct connection can be established between the convolutional layer and the transposed convolutional layer, such as Fig.11 As shown, a direct connection is established between the convolution layer and the transposed convolution layer corresponding to the feature image of the same size, thereby directly connecting the feature image information of the convolution link to the feature image in the transposed convolution link, which is conducive to obtaining a third feature image with more comprehensive information.

[0193] Step S1040: Using the second pixel rearrangement layer, the third feature image is subjected to pixel rearrangement processing from multi-channel to single-channel to obtain a beauty face composite image.

[0194] The second pixel rearrangement layer may rearrange multiple channels of the third feature image into a single channel. In an optional implementation, step S1240 includes:

[0195] Input the third feature image with a channel number of b*n*n into the second pixel rearrangement layer;

[0196] Rearrange the pixels at the same position in each n*n channel of the third feature image to the n*n neighborhood in a single channel, and output a beauty face composite image with b channels;

[0197] Wherein, b is a positive integer.

[0198] The second pixel rearrangement layer can be implemented by using the depth_to_space function in TensorFlow to convert the depth features in the third feature image into spatial features. It can also be implemented by using a transposed convolution operation with a step size of n. In this case, the second pixel rearrangement layer can be regarded as a special transposed convolution layer.

[0199] If the convolution layer and the transposed convolution layer are completely symmetrical structures, that is, the size and number of channels of the third feature image are the same as those of the first feature image, then a=b, and step S1240 can be the inverse operation of step S1210. Furthermore, the number of channels of the beautified face image and the face image to be beautified is also the same, that is, the processing of the image beautification network does not change the image size and number of channels.

[0200] It should be noted that the processing of the image beautification network also does not change the number of faces. For example, the face image to be beautified is composed of 4 original face sub-images. After being processed by the image beautification network, the output beautified face image also includes 4 faces, which are the faces in the 4 original face sub-images after beautification.

[0201] If the image beautification network is used for blemish removal, its blemish removal effect depends on the quality and training effect of the beautification image dataset, rather than on the artificially designed image feature calculation. After being fully trained with a relatively comprehensive beautification image dataset, the image beautification network can cope with almost all situations in practical applications, including different lighting conditions, different skin conditions, etc., to accurately and fully detect and remove portrait blemishes.

[0202] In one implementation, in addition to conventional beautification training, anti-flicker training can be added to the training process of the image beautification network, so that the trained image beautification network can further ensure the consistency of face beautification effects. Fig.11 As shown, the image beautification processing method may further include the following steps S1110 to S1140:

[0203] Step S1110: inputting a first sample image to be beautified into an image beautification network to be trained to output a first sample image to be beautified.

[0204] The image beautification network can realize the combination of different beautification functions. The exemplary embodiment can obtain the beautification image data set corresponding to different beautification functions according to actual needs to train the required image beautification network. For example, if it is necessary to train the image beautification network for defect removal, a sample image to be beautified with defects is obtained, and the corresponding annotation image (ground truth) is obtained through artificial defect removal processing, thereby constructing the beautification image data set for defect removal; if it is necessary to train the image beautification network for defect removal + deformation, a sample image to be beautified with defects is obtained, and the corresponding annotation image is obtained through artificial defect removal and deformation processing, thereby constructing the beautification image data set for defect removal + deformation. Of course, the annotation image can also be obtained first, and the sample image to be beautified can be obtained through reverse processing, for example, a flawless face image is obtained, and defects are added, reverse deformation (referring to the opposite processing of deformation in beautification, for example, "slimming face" is often performed in beautification, and the face can be widened here) and other processing are performed to obtain the sample image to be beautified, and the flawless face image is used as its corresponding annotation image to construct the beautification image data set for defect removal + deformation. It can be seen that this exemplary embodiment can train an image beautification network with any one or more combinations of beautification functions by constructing different beautification image datasets.

[0205] In one implementation, multiple face images may be combined to obtain a sample image to be beautified, and the artificially beautified images corresponding to the multiple face images may be combined to obtain a labeled image corresponding to the sample image to be beautified, and then the sample image to be beautified and the labeled image are added to the beautified image dataset. In other words, the beautified image dataset may include different types of images such as single-face images, multi-face images, and combined-face images.

[0206] The first sample image to be beautified is used to provide beautification training for the image beautification network. The beautification training refers to training the image beautification network to beautify high-quality and natural images. The first sample image to be beautified can be any image in the beautification image dataset.

[0207] The structure of the image beautification network can refer to the above Figure 8 and Fig. 9 The first sample image to be beautified is input into the image beautification network, and the corresponding first sample image to be beautified is output. Since the image beautification network is not trained or not fully trained at this time, the first sample image to be beautified should be different from the ideal beautification image, for example, different from the annotated image corresponding to the first sample image to be beautified.

[0208] Step S1120: input the second sample image to be beautified into the image beautification network, and transform the image output by the image beautification network through transformation parameters to obtain a second sample image of beautification.

[0209] The second sample image to be beautified is used to provide anti-flickering training for the image beautification network. Anti-flickering training refers to training the image beautification network to achieve stable, flicker-free beautification processing effects for multiple consecutive frames of images. The second sample image to be beautified can be any image in the beautification image dataset.

[0210] In one implementation, the first sample image to be beautified and the second sample image to be beautified may be obtained from the same beautified image dataset. For example, the first sample image to be beautified and the second sample image to be beautified may be the same image, so that each image in the beautified image dataset may be used as the first sample image to be beautified and the second sample image to be beautified at the same time, thereby improving the utilization rate of the dataset.

[0211] The beautification image dataset constructed by manual processing is a labeled dataset, in which the sample images to be beautified all have corresponding labeled images. In addition, an unlabeled dataset can also be constructed. For example, only the sample images to be beautified are collected without manual processing, and these sample images to be beautified are formed into an unlabeled dataset, in which the sample images to be beautified do not have corresponding labeled images. In one embodiment, the first sample image to be beautified can be obtained in the labeled dataset, and the second sample image to be beautified can be obtained in the unlabeled dataset. The labeled image corresponding to the second sample image to be beautified is not used in the training process. The difficulty of obtaining an unlabeled dataset is much lower than that of a labeled dataset, which is conducive to increasing the number of the second sample images to be beautified, and facilitating more comprehensive anti-flicker training of the image beautification network.

[0212] Between consecutive multiple frames of images, due to the movement of the shooting lens or the movement of the subject itself, the main subject (i.e., the face) in the image changes, including one or more of translation, rotation, and scaling. The transformation parameters in step S1120 are used to simulate such changes between different frames of images, and may include any one or more of translation parameters, rotation parameters, and scaling parameters.

[0213] In one implementation, statistical analysis may be performed on the transformation parameters of the images in the video to obtain the transformation parameters in step S1120.

[0214] In one implementation, the transformation parameters may be obtained by random generation. Specifically, the image beautification processing method may further include the following steps:

[0215] Obtaining a preset first numerical interval, a second numerical interval, and a third numerical interval;

[0216] A translation parameter is randomly generated within a first numerical interval, a rotation parameter is randomly generated within a second numerical interval, and a scaling parameter is randomly generated within a third numerical interval.

[0217] Among them, the first numerical interval is the numerical interval for the translation parameter, the second numerical interval is the numerical interval for the rotation parameter, and the third numerical interval is the numerical interval for the scaling parameter, which respectively represent the numerical ranges of translation, rotation, and scaling that may occur in the image in the video. This exemplary embodiment can determine three numerical intervals based on experience and actual scenarios. Exemplarily, the first numerical interval can be [-3, 3], the unit is pixel, indicating the number of pixels of translation; the second numerical interval can be [-5, 5], the unit is degree, indicating the degree of rotation; the third numerical interval can be [0.97, 1.03], the unit is times, indicating the scaling ratio. Then, random numbers are generated in the three numerical intervals respectively to obtain the translation parameter, rotation parameter, and scaling parameter, that is, the transformation parameters in step S320 are obtained.

[0218] The second sample image to be beautified is input into the image beautification network, and then the image output by the image beautification network is transformed using the transformation parameters obtained above, that is, the second sample image to be beautified is first beautified and then transformed to obtain the second sample image.

[0219] Step S1130: transform the second sample image to be beautified by using the transformation parameters, and input the transformed second sample image to be beautified into an image beautification network to output a third sample image to be beautified.

[0220] Step S1130 is equivalent to exchanging the order of beautification and transformation in step S1120, that is, the second image to be beautified is first transformed and then beautified to obtain the third beautification sample image.

[0221] Step S1140 , based on the difference between the annotated image corresponding to the first sample image to be beautified and the first sample image, and the difference between the second sample image to be beautified and the third sample image, updating the parameters of the image beautification network training.

[0222] The difference between the annotated image and the first beautification sample image reflects the beautification effect of the image beautification network, that is, the closer the first beautification sample image is to the annotated image, the better the beautification effect of the image beautification network. Therefore, based on the difference between the annotated image and the first beautification sample image, the parameters of the image beautification network can be updated to achieve beautification training.

[0223] The difference between the second and third beautification sample images reflects the anti-flickering effect of the image beautification network. k , which is equivalent to the second sample image to be beautified after the above transformation) relative to the k-1th frame image (denoted as I k-1 , which is equivalent to the second sample image to be beautified) in the above-mentioned face being transformed. Assuming that the transformation parameter is P, that is, I k =Ik-1 In actual beautification, it is necessary to perform beautification processing on both the k-1th frame image and the kth frame image. After the k-1th frame image and the kth frame image are input into the image beautification network, the output images are recorded as Y(I k-1 )、Y(I k )=Y(I k-1 ·P), if Y(I k-1 )·P=Y(I k-1 ·P) (equivalent to the second beautification sample image being equal to the third beautification sample image), indicating that there is a transformation relationship between the face in the k-1th frame image and the kth frame image after beautification processing, but the face itself is the same, that is, the two consecutive frames in the video have beauty consistency and there is no flicker. Therefore, based on the difference between the second beautification sample image and the third beautification sample image, the parameters of the image beautification network can be updated to achieve anti-flicker training.

[0224] The parameter update of the image beautification network by the above-mentioned beautification training and the parameter update of the image beautification network by the anti-flicker training can be performed simultaneously or separately, and the present disclosure does not limit this.

[0225] In one implementation, step S1140 may include:

[0226] Determining a first loss function value based on a difference between the annotated image corresponding to the first sample image to be beautified and the first sample image to be beautified;

[0227] Determining a second loss function value based on a difference between the second beautification sample image and the third beautification sample image;

[0228] Update the parameters of the image beautification network according to the first loss function value and the second loss function value.

[0229] Among them, the first loss function is used to reflect the beauty loss of the image beautification network, and the second loss function is used to reflect the anti-flicker loss of the image beautification network. The first loss function and the second loss function can be established in advance, for example, they can be in the form of MAE (Mean Absolute Error, i.e. L1 loss) and MSE (Mean Square Error, i.e. L2 loss). During training, the labeled image and the first beauty sample image are substituted into the first loss function to calculate the first loss function value, and the second beauty sample image and the third beauty sample image are substituted into the second loss function to calculate the second loss function value. Furthermore, the parameters of the image beautification network can be updated by gradient descent according to the first loss function value and the second loss function value, respectively, and the global loss function value can also be further calculated from the first loss function value and the second loss function value. The global loss function can be, for example, the weighted result of the first loss function and the second loss function, and then the parameters of the image beautification network are updated by gradient descent according to the global loss function value.

[0230] Fig.12 The schematic process of training the image beautification network is shown. The first sample image to be beautified is input into the image beautification network, the first sample image to be beautified is output, the annotated image corresponding to the first sample image to be beautified and the first sample image to be beautified are substituted into the first loss function to obtain the first loss function value. The second sample image to be beautified is input into the image beautification network, and the output image is transformed using the pre-generated transformation parameters to obtain the second sample image to be beautified. The second sample image to be beautified is transformed using the transformation parameters, and the transformed image is input into the image beautification network to output the third sample image to be beautified. The second sample image to be beautified and the third sample image to be beautified are substituted into the second loss function to obtain the second loss function value. The first loss function value and the second loss function value are weighted to obtain the global loss function value, and the parameters in the image beautification network are updated according to the global loss function value. After multiple rounds of iterative updates, when the accuracy of the image beautification network on the verification subset in the above-mentioned beauty image data set reaches the preset accuracy threshold or the loss value is lower than the preset loss threshold, it is determined that the training is completed, and an image beautification network that can be used for actual beauty processing is obtained.

[0231] From the above, we can see that while the image beautification network implements conventional beautification processing, it can also show the invariance of the beautification effect to transformations such as translation, rotation, and scaling of the image. For example, when the face in the video is translated, rotated, or scaled due to changes in the distance from the camera, the image beautification network can maintain the beautification effect on the face. This, combined with the above-mentioned technical means of stabilizing the position and size of the face by stabilizing the bounding box, can further ensure the consistency of the face beautification effect and prevent the screen flickering after multiple consecutive frames of images have been beautified.

[0232] Step S530: split the beautified face sub-image corresponding to the original face sub-image from the beautified face image.

[0233] When splitting the beautified face image, the above-mentioned saved combined information can be used to split sub-images of specific positions and specific sizes from the beautified face image, namely, the beautified face sub-images, and the beautified face sub-images correspond one-to-one to the original face sub-images.

[0234] Continue to refer Figure 3 In step S350, a target beautified image corresponding to the original image to be beautified is generated according to the beautified face sub-image.

[0235] The beautified face sub-image is the result of beautification of the face in the original image to be beautified. By replacing the face in the original image to be beautified with the beautified face sub-image, the beautification result of the original image to be beautified, i.e., the target beautified image, can be obtained. Exemplarily, the original face sub-image in the original image to be beautified can be replaced with the corresponding beautified face sub-image to obtain the target beautified image.

[0236] In one embodiment, if the original face sub-images are transformed when they are combined into the face image to be beautified, the beautified face sub-images obtained by splitting can be inversely transformed accordingly, including removing padded pixels, upsampling, rotating 90 degrees in the opposite direction, etc., so that the inversely transformed beautified face sub-images are consistent with the original face sub-images in direction, size, etc., so that a 1:1 replacement can be performed in the original image to be beautified to obtain the target beautified image.

[0237] The beautified face sub-image is a face sub-image that has been beautified by the image beautification network, and is usually a face sub-image with a high degree of beautification. In one embodiment, in order to increase the realism of the beautified face sub-image, before replacing the original face sub-image in the original image to be beautified with the corresponding beautified face sub-image, the original face sub-image can be used to perform beautification weakening processing on the beautified face sub-image. Beautification weakening processing refers to reducing the beautification degree of the beautified face sub-image to increase. Two exemplary methods of beauty weakening processing are provided below:

[0238] Method 1: According to the set beautification degree parameters, the original face sub-image is merged into the beautified face sub-image. The beautification degree parameters may be beautification intensity parameters under a specific beautification function, such as the degree of blemish removal. In this exemplary embodiment, the beautification degree parameters may be parameters used for the current settings, system default parameters, or parameters used for the last beautification. After determining the beautification degree parameters, the original face sub-image and the beautified face sub-image may be merged with the beautification degree parameters as a weight. For example, assuming that the range of the blemish removal degree is 0 to 100, and the currently set value is a, refer to the following formula:

[0239]

[0240] Wherein, image_blend represents the fused image, image_ori represents the original face sub-image, and image_deblemish represents the beautified face sub-image. When a is 0, it means no blemish removal is performed, and the original face sub-image is used completely; when a is 100, it means complete blemish removal, and the beautified face sub-image is used completely. Therefore, formula (12) indicates that through fusion, an image between the original face sub-image and the beautified face sub-image is obtained. The larger a is, the closer the obtained image is to the beautified face sub-image, that is, the higher the degree of beautification, the more obvious the beautification effect.

[0241] It should be noted that if the original face sub-images are transformed when combining the original face sub-images into the face images to be beautified, the beautified face sub-images obtained by splitting can be inversely transformed. The original face sub-images and the beautified face sub-images have the following relationship: the original face sub-images before transformation and the beautified face sub-images after inverse transformation have the same direction, size, etc.; the original face sub-images after transformation and the beautified face sub-images before inverse transformation have the same direction, size, etc. Therefore, when the original face sub-images and the beautified face sub-images are fused using the above formula (12), the original face sub-images before transformation and the beautified face sub-images after inverse transformation can be fused, or the original face sub-images after transformation and the beautified face sub-images before inverse transformation can be fused.

[0242] Method 2: Fusing the high-frequency image of the original face sub-image into the beautified face sub-image, wherein the high-frequency image refers to an image containing high-frequency information such as detailed texture in the original face sub-image.

[0243] In one embodiment, the high-frequency image may be acquired by:

[0244] When combining the one or more original face sub-images based on the input image size of the image beautification network, if the original face sub-image is down-sampled, then up-sampling the down-sampled face sub-image obtained after the down-sampling to obtain an up-sampled face sub-image;

[0245] According to the difference between the original face sub-image and the upsampled face sub-image, a high-frequency image of the original face sub-image is obtained.

[0246] Among them, the resolution of the downsampled face sub-image is lower than that of the original face sub-image. Generally, in the process of downsampling, the high-frequency information of the image will inevitably be lost. The downsampled face sub-image is upsampled so that the resolution of the obtained upsampled face sub-image is the same as that of the original face sub-image. It should be noted that if the original face sub-image is rotated before being downsampled, the downsampled face sub-image can also be rotated in the opposite direction after being upsampled so that the direction of the obtained upsampled face sub-image is the same as that of the original face sub-image.

[0247] Upsampling can be performed by bilinear interpolation, nearest neighbor interpolation, and other methods. Although upsampling can restore the resolution, it is difficult to completely restore the lost high-frequency information, that is, the upsampled face sub-image can be regarded as a low-frequency image of the original face sub-image. Thus, the difference between the original face sub-image and the upsampled face sub-image is determined. For example, the original face sub-image and the upsampled face sub-image can be subtracted, and the result is the high-frequency information of the original face sub-image. The subtracted value is formed into an image, that is, the high-frequency image of the original face sub-image.

[0248] In another implementation, the original face sub-image may be filtered to extract high-frequency information to obtain a high-frequency image.

[0249] When fusing the high-frequency image into the beautified face sub-image, a direct addition method can be adopted to superimpose the high-frequency image onto the beautified face sub-image, so that high-frequency information such as detailed texture is added to the beautified face sub-image, making it more realistic.

[0250] Since the original face sub-image and the upsampled face sub-image are usually very similar, the pixel values ​​in the high-frequency image obtained based on their difference are generally small, such as the RGB channel values ​​not exceeding 4. However, for the mutation position in the original face sub-image, such as a small black mole on the face, it has strong high-frequency information, so the pixel value of the corresponding position in the high-frequency image may be relatively large. When the high-frequency image is fused to the original face sub-image, the pixel values ​​at these positions may have adverse effects, such as sharp edges such as "mole marks", resulting in unnatural visual experience.

[0251] In view of the above problem, in one implementation, the image beautification processing method may further include the following steps:

[0252] Identify defect points in high-frequency images;

[0253] The pixel values ​​in a preset area around the above defect point in the high-frequency image are adjusted to a preset value range.

[0254] Among them, the defect point is a pixel point with strong high-frequency information, and the point with a larger pixel value in the high-frequency image can be determined as a defect point. Alternatively, in an implementation, the defect point can be determined by the following method:

[0255] Subtract the beauty face sub-image from the corresponding original face sub-image to obtain the difference value of each pixel;

[0256] When it is determined that the difference value of a certain pixel point meets the preset defect condition, the pixel point corresponding to the pixel point in the high-frequency image is determined as the defect point.

[0257] The preset defect condition is used to measure the difference between the beautified face sub-image and the original face sub-image to determine whether each pixel is a defect point to be removed. In the defect removal process, small moles and acnes on the face are usually removed and the face skin color is filled. At this position, the difference between the beautified face sub-image and the original face sub-image is very large, so the defect points can be identified by setting the preset defect condition.

[0258] Exemplarily, the preset defect condition may include: the difference of each color channel is greater than the first color difference threshold, and at least one of the difference of each color channel is greater than the second color difference threshold. The first color difference threshold and the second color difference threshold may be empirical thresholds. For example, when the color channel includes RGB, the first color difference threshold may be 20, and the second color difference threshold may be 40. Thus, after obtaining the difference between each pixel in the beautified face sub-image and the original face sub-image, the specific difference of the three RGB color channels in the difference is judged to determine whether the difference of each color channel is greater than 20, and whether the difference of at least one color channel is greater than 40. When these two conditions are met, it means that the preset defect condition is met, and the pixel at the corresponding position in the high-frequency image is determined as a defect point.

[0259] After determining the defect point, the preset area around the defect point can be further determined in the high-frequency image. For example, it can be a 5*5 pixel area centered on the defect point. The specific size can be determined according to the size of the high-frequency image, and the present disclosure does not limit it. The pixel values ​​in the preset area are adjusted to a preset numerical range. The preset numerical range is generally a smaller numerical range, which can be determined based on experience and actual needs. The pixel value usually needs to be reduced during adjustment. Exemplarily, the preset numerical range can be -2 to 2, and the pixel values ​​around the defect point may exceed -5 to 5. Adjusting it to -2 to 2 actually performs limit processing. This can weaken sharp edges such as "mole marks" and increase the visual natural feeling.

[0260] The above describes two beautification and weakening processing methods. This exemplary embodiment can use these two beautification and weakening processing methods at the same time. For example, the original face sub-image and the beautification face sub-image are first fused by method 1, and then the high-frequency image is superimposed thereon by method 2 to obtain the beautification face sub-image after beautification and weakening processing. The beautification face sub-image has both good beautification effect and realism.

[0261] In one implementation, when replacing the original face sub-image in the image to be processed with the corresponding beautified face sub-image, the following steps may also be performed:

[0262] A gradient process is performed on the boundary area between the unreplaced area in the original image to be beautified and the beautified face sub-image, so that the boundary area forms a smooth transition.

[0263] The unreplaced area in the original image to be beautified is the area in the original image to be beautified except the original face sub-image. The boundary area between the unreplaced area and the beautified face sub-image actually includes two parts: the boundary area in the unreplaced area adjacent to the beautified face sub-image, and the boundary area in the beautified face sub-image adjacent to the unreplaced area. In this exemplary embodiment, the gradient processing can be performed on any part, or the gradient processing can be performed on both parts at the same time.

[0264] refer to Fig.13As shown, a certain proportion (such as 10%) of the boundary area can be determined in the beautified face sub-image, which extends inward from the edge of the beautified face sub-image. It should be noted that the boundary area usually needs to avoid the face part to avoid changing the color of the face part during the gradient processing. For example, the original face sub-image is intercepted by the above-mentioned stable bounding box, so that the face in the original face sub-image has a certain distance from the boundary, and the face in the beautified face sub-image also has a certain distance from the boundary. In this way, when performing the gradient processing, the face part can be better avoided. After determining the boundary area, the inner edge color of the boundary area is obtained, which is recorded as the first color; the inner edge color of the unreplaced area is obtained, which is recorded as the second color; and the boundary area is subjected to the gradient processing of the first color and the second color. As a result, the boundary between the unreplaced area and the beautified face sub-image is a gradient color area ( Fig.13 This creates a smooth transition and prevents color mutations that would cause visual disharmony.

[0265] It should be noted that when there are multiple beautified face sub-images, each beautified face sub-image can be used to replace the corresponding original face sub-image in the image to be processed, and gradient processing can be performed on the boundary area to obtain a target beautified image with a natural and harmonious visual experience.

[0266] Fig.14 The schematic flow of the image beautification processing method is shown, including:

[0267] Step S1401: determine an original image to be beautified from a video. For example, the current frame may be used as the original image to be beautified.

[0268] Step S1402 , performing face detection on the original image to be beautified, obtaining basic bounding boxes of multiple faces, filtering out faces whose areas are smaller than a face area threshold, and recording the remaining faces as faces to be determined.

[0269] Step S1403 , tracking the faces to be determined according to the determined faces in the reference frame image, obtaining the ID of each face to be determined, and determining its stable bounding box.

[0270] Step S1404, intercepting the image within the stable bounding box to obtain the original face sub-image.

[0271] Step S1405, dividing the input image size of the image beautification network into multiple sub-image sizes according to the number of original face sub-images, down-sampling the original face sub-images according to the sub-image sizes, and further performing processing such as rotation and padding to obtain down-sampled face sub-images corresponding to each original face sub-image.

[0272] Step S1406, upsampling the downsampled face sub-image. If rotation, padding and other processing are performed when obtaining the downsampled face sub-image, reverse rotation, padding removal and other processing can also be performed to obtain an upsampled face sub-image, which has the same resolution as the corresponding original face sub-image.

[0273] Step S1407 , subtracting the original face sub-image from the corresponding upsampled face sub-image to obtain a high-frequency image of the original face sub-image.

[0274] Step S1408: combining the downsampled face sub-images into a face image to be beautified.

[0275] Step S1409, input the face image to be beautified into the image beautification network, and output the beautified face image after processing.

[0276] Step S1410: split the beautified face image into beautified face sub-images corresponding one-to-one to the original face sub-images.

[0277] Step S1411 , the beautified face sub-image is fused with the corresponding original face sub-image according to the beautification degree parameter, and then added to the high-frequency image of the original face sub-image to obtain the face sub-image to be replaced.

[0278] Step S1412, the face sub-image to be replaced is merged into the original image to be beautified. Specifically, the face sub-image to be replaced can replace the part of the original face sub-image in the original image to be beautified, and the edge color gradient processing is performed, so that the face in the original image to be beautified is replaced by the face after beautification, and finally the target beautified image is obtained. Personalized beautification processing can also be performed later.

[0279] The exemplary embodiment of the present disclosure also provides an image beautification processing device. Fig.15 As shown, the image beautification processing device 1500 may include:

[0280] The image acquisition module 1510 is configured to acquire the original image to be beautified from a plurality of consecutive frames of images;

[0281] The face matching module 1520 is configured to match the face in the original image to be beautified with the face in the reference frame image of the original image to be beautified, and determine a stable bounding box of the face in the original image to be beautified according to the matching result;

[0282] The sub-image extraction module 1530 is configured to extract an original face sub-image from the original image to be beautified based on a stable bounding box of the face in the original image to be beautified;

[0283] The beautification processing module 1540 is configured to process the original face sub-image using the image beautification network to obtain a corresponding beautified face sub-image;

[0284] The image generation module 1550 is configured to generate a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image.

[0285] In one implementation, the face matching module 1520 is configured to:

[0286] Detecting a face in the original image to be beautified, recording it as a face to be determined, and matching the face to be determined with a determined face in a reference frame image of the original image to be beautified;

[0287] If the face to be determined does not match the determined face, the basic bounding box of the face to be determined is expanded according to the first preset parameter to obtain a stable bounding box of the face to be determined;

[0288] If the face to be determined successfully matches the determined face, the stable bounding box of the face to be determined is determined based on the stable bounding box of the determined face.

[0289] In one implementation, matching the to-be-determined face with a determined face in a reference frame image of the original image to be beautified includes:

[0290] Whether the face to be determined matches the determined face successfully is determined based on the degree of overlap between the basic bounding box of the face to be determined and the basic bounding box of the determined face.

[0291] In one implementation, the method of determining a stable bounding box of a face to be determined based on a stable bounding box of a determined face includes:

[0292] Based on a preset stability coefficient, the center point coordinates of the stable bounding box of the determined face and the center point coordinates of the basic bounding box of the face to be determined are weighted to obtain the center point coordinates of the stable bounding box of the face to be determined.

[0293] In one implementation, the method of determining a stable bounding box of a face to be determined based on a stable bounding box of a determined face includes:

[0294] If the size of the basic bounding box of the face to be determined is larger than the product of the size of the stable bounding box of the determined face and the first magnification, the size of the stable bounding box of the determined face is expanded according to the second preset parameter to obtain the size of the stable bounding box of the face to be determined;

[0295] If the size of the basic bounding box of the face to be determined is smaller than the product of the size of the stable bounding box of the determined face and the second magnification, the size of the stable bounding box of the determined face is reduced according to the third preset parameter to obtain the size of the stable bounding box of the face to be determined; the first magnification is greater than the second magnification;

[0296] If the size of the basic bounding box of the face to be determined is smaller than the product of the size of the stable bounding box of the determined face and the first magnification, and larger than the product of the size of the stable bounding box of the determined face and the second magnification, the size of the stable bounding box of the determined face will be used as the size of the stable bounding box of the face to be determined.

[0297] In one implementation, the beauty processing module 1540 is configured to:

[0298] Based on the input image size of the image beautification network, the original face sub-images extracted from the original image to be beautified are combined to generate the face image to be beautified;

[0299] The image beautification network is used to process the beautified face image to obtain the corresponding beautified face image;

[0300] A beautified face sub-image corresponding to the original face sub-image is split from the beautified face image.

[0301] In one embodiment, the input image size based on the image beautification network is combined with the original face sub-images extracted from the original image to be beautified to generate the face image to be beautified, including:

[0302] According to the number of original face sub-images, the input image size is divided into sub-image sizes corresponding to the original face sub-images one by one;

[0303] Transforming the corresponding original face sub-image based on the size of each sub-image;

[0304] The transformed original face sub-images are combined to generate the face image to be beautified.

[0305] In one implementation, transforming the corresponding original face sub-image based on the size of each sub-image includes any one or more of the following:

[0306] When the size relationship between the width and height of the original face sub-image is different from the size relationship between the width and height of the sub-image size, the original face sub-image is rotated 90 degrees;

[0307] When the size of the original face sub-image or the rotated original face sub-image is larger than the sub-image size, downsampling the original face sub-image or the rotated original face sub-image according to the sub-image size;

[0308] When the size of the original face sub-image or the original face sub-image processed by at least one of rotation and downsampling is smaller than the sub-image size, the original face sub-image is padded according to the difference between the size of the original face sub-image and the sub-image size, or the original face sub-image processed by at least one of rotation and downsampling is padded according to the difference between the size of the original face sub-image and the sub-image size.

[0309] In one implementation, the image generation module 1550 is configured to:

[0310] The original face sub-image in the original image to be beautified is replaced with the corresponding beautified face sub-image to obtain a target beautified image.

[0311] In one implementation, the image generation module 1550 is configured to:

[0312] Before replacing the original face sub-image in the original image to be beautified with the corresponding beautified face sub-image, the beautified face sub-image is beautified and weakened using the original face sub-image.

[0313] In one implementation, the above-mentioned beautification-weakened processing of the beautified face sub-image using the original face sub-image includes:

[0314] According to the set beautification degree parameters, the original face sub-image is fused into the beautified face sub-image.

[0315] In one implementation, the above-mentioned beautification-weakened processing of the beautified face sub-image using the original face sub-image includes:

[0316] The high-frequency image of the original face sub-image is fused into the beauty face sub-image.

[0317] In one implementation, the image acquisition module 1510 is configured to:

[0318] When the input image size based on the image beautification network is combined with the original face sub-images extracted from the original image to be beautified, if the original face sub-image is down-sampled, the down-sampled face sub-image obtained after the down-sampling is up-sampled to obtain the up-sampled face sub-image, and the resolution of the up-sampled face sub-image is the same as that of the original face sub-image;

[0319] According to the difference between the original face sub-image and the upsampled face sub-image, a high-frequency image of the original face sub-image is obtained.

[0320] In one implementation, the image generation module 1550 is configured to:

[0321] When replacing the original face sub-image in the original image to be beautified with the corresponding beautified face sub-image, a gradient process is performed on the boundary area between the non-replaced area in the original image to be beautified and the beautified face sub-image, so that the boundary area forms a smooth transition.

[0322] In one embodiment, the image beautification network is a fully convolutional network, including: a first pixel rearrangement layer, at least one convolutional layer, at least one transposed convolutional layer, and a second pixel rearrangement layer.

[0323] The beauty processing module 1530 is configured to:

[0324] Using the first pixel rearrangement layer to perform pixel rearrangement processing on the beautified face image from a single channel to multiple channels, to obtain a first feature image;

[0325] Performing convolution processing on the first feature image using a convolution layer to obtain a second feature image;

[0326] Performing transposed convolution processing on the second feature image using a transposed convolution layer to obtain a third feature image;

[0327] The second pixel rearrangement layer is used to perform pixel rearrangement processing on the third feature image from multi-channel to single-channel to obtain a beautified face image.

[0328] In one implementation, the beauty processing module 1530 is configured to:

[0329] Inputting the face image to be beautified with a channel number a into the first pixel rearrangement layer;

[0330] Rearrange the pixels of each n*n neighborhood in each channel of the facial image to be beautified to the same position in the n*n channels, and output a first feature image with a*n*n channels;

[0331] Wherein, a is a positive integer, and n is a positive integer not less than 2.

[0332] In one implementation, the beauty processing module 1530 is configured to:

[0333] Input the third feature image with a channel number of b*n*n into the second pixel rearrangement layer;

[0334] Rearrange the pixels at the same position in each n*n channel of the third feature image to an n*n neighborhood in a single channel, and output a beauty face image with b channels;

[0335] Wherein, b is a positive integer, and n is a positive integer not less than 2.

[0336] In one implementation, the image beautification processing apparatus 1500 may further include a network training module configured to:

[0337] Inputting the first sample image to be beautified into the image beautification network to be trained to output the first sample image to be beautified;

[0338] Inputting the second sample image to be beautified into the image beautification network, and transforming the image output by the image beautification network through transformation parameters to obtain a second sample image to be beautified;

[0339] Transforming the second sample image to be beautified by using the transformation parameters, and inputting the transformed second sample image to be beautified into the image beautification network to output a third sample image to be beautified;

[0340] Based on the difference between the annotated image corresponding to the first beautified sample image and the first beautified sample image, and the difference between the second beautified sample image and the third beautified sample image, the parameters of the image beautification network training are updated.

[0341] In one implementation, the beautified face image includes a blemish-removed beautified image; the beautification processing module 1530 is configured to:

[0342] After obtaining the blemish-removed and beautified image, personalized beautification processing is performed on the blemish-removed and beautified image to obtain a final beautified image.

[0343] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0344] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In an optional embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0345] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0346] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0347] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0348] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0349] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0350] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0351] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image beautification processing method, characterized in that: include: Obtain the original image to be beautified from multiple consecutive frames of images; Matching a human face in the original image to be beautified with a human face in a reference frame image of the original image to be beautified, and determining a stable bounding box of the human face in the original image to be beautified according to the matching result; Extracting an original face sub-image from the original image to be beautified based on a stable bounding box of the face in the original image to be beautified; Processing the original face sub-image using an image beautification network to obtain a corresponding beautified face sub-image; Generating a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image; The step of matching the face in the original image to be beautified with the face in the reference frame image of the original image to be beautified, and determining a stable bounding box of the face in the original image to be beautified according to the matching result includes: Detecting a human face in the original image to be beautified, recording it as a to-be determined human face, and matching the to-be determined human face with a determined human face in a reference frame image of the original image to be beautified; If the to-be-determined face fails to match the determined face, expanding the basic bounding box of the to-be-determined face according to a first preset parameter to obtain a stable bounding box of the to-be-determined face; If the to-be-determined face matches the determined face successfully, a stable bounding box of the to-be-determined face is determined according to the stable bounding box of the determined face.

2. The method according to claim 1, characterized in that The detecting of a human face in the original image to be beautified, recorded as a human face to be determined, includes: If the area of ​​the basic bounding box of the face in the original image to be beautified is greater than or equal to the face area threshold, the face is recorded as a face to be determined.

3. The method according to claim 1, characterized in that The matching of the to-be-determined human face with a determined human face in a reference frame image of the original image to be beautified comprises: Whether the to-be-determined face successfully matches the determined face is determined according to the degree of overlap between the basic bounding box of the to-be-determined face and the basic bounding box of the determined face.

4. The method according to claim 1, characterized in that: The step of determining the stable bounding box of the face to be determined according to the stable bounding box of the determined face comprises: Based on a preset stability coefficient, the center point coordinates of the stable bounding box of the determined face and the center point coordinates of the basic bounding box of the face to be determined are weighted to obtain the center point coordinates of the stable bounding box of the face to be determined.

5. The method according to claim 1, characterized in that The step of determining the stable bounding box of the face to be determined according to the stable bounding box of the determined face comprises: If the size of the basic bounding box of the face to be determined is larger than the product of the size of the stable bounding box of the determined face and the first magnification, expanding the size of the stable bounding box of the determined face according to a second preset parameter to obtain the size of the stable bounding box of the face to be determined; If the size of the basic bounding box of the face to be determined is smaller than the product of the size of the stable bounding box of the determined face and the second magnification, reducing the size of the stable bounding box of the determined face according to a third preset parameter to obtain the size of the stable bounding box of the face to be determined; the first magnification is greater than the second magnification; If the size of the basic bounding box of the face to be determined is smaller than the product of the size of the stable bounding box of the determined face and the first magnification, and is larger than the product of the size of the stable bounding box of the determined face and the second magnification, the size of the stable bounding box of the determined face will be used as the size of the stable bounding box of the face to be determined.

6. The method according to claim 1, characterized in that The process of processing the original face sub-image by using the image beautification network to obtain a corresponding beautified face sub-image includes: combining the original face sub-images extracted from the original image to be beautified based on the input image size of the image beautification network to generate a face image to be beautified; Processing the facial image to be beautified using the image beautification network to obtain a corresponding facial image; A beautified face sub-image corresponding to the original face sub-image is separated from the beautified face image.

7. The method according to claim 6, characterized in that The input image size based on the image beautification network combines the original face sub-images extracted from the original image to be beautified to generate the face image to be beautified, including: According to the number of the original face sub-images, dividing the input image size into sub-image sizes corresponding to the original face sub-images one by one; transforming the corresponding original face sub-image based on the size of each sub-image; The transformed original face sub-images are combined to generate the face image to be beautified.

8. The method according to claim 7, characterized in that The transforming the corresponding original face sub-image based on the size of each sub-image respectively includes any one or more of the following: When the size relationship between the width and height of the original face sub-image is different from the size relationship between the width and height of the sub-image size, rotating the original face sub-image by 90 degrees; When the size of the original face sub-image or the rotated original face sub-image is larger than the sub-image size, downsampling the original face sub-image or the rotated original face sub-image according to the sub-image size; When the size of the original face sub-image or the original face sub-image processed by at least one of rotation and downsampling is smaller than the sub-image size, the original face sub-image is padded according to the difference between the size of the original face sub-image and the sub-image size, or the original face sub-image processed by at least one of rotation and downsampling is padded according to the difference between the size of the original face sub-image processed by at least one of rotation and downsampling and the sub-image size.

9. The method according to claim 6, characterized in that Generating a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image includes: The original face sub-image in the original image to be beautified is replaced with the corresponding beautified face sub-image to obtain the target beautified image.

10. The method according to claim 9, characterized in that Before replacing the original face sub-image in the original image to be beautified with the corresponding face sub-image to be beautified, the method further includes: The original face sub-image is used to perform beauty-weakening processing on the beauty face sub-image.

11. The method according to claim 10, characterized in that The performing beautification and weakening processing on the beautification face sub-image by using the original face sub-image comprises: According to the set beautification degree parameter, the original face sub-image is fused into the beautified face sub-image.

12. The method according to claim 10, characterized in that The performing beautification and weakening processing on the beautification face sub-image by using the original face sub-image comprises: The high-frequency image of the original face sub-image is fused into the beautified face sub-image.

13. The method according to claim 12, characterized in that The method further comprises: When combining the original face sub-images extracted from the original image to be beautified based on the input image size of the image beautification network, if the original face sub-images are downsampled, upsampling the downsampled face sub-images obtained after the downsampling to obtain upsampled face sub-images, the upsampled face sub-images having the same resolution as the original face sub-images; A high-frequency image of the original face sub-image is obtained according to a difference between the original face sub-image and the up-sampled face sub-image.

14. The method according to claim 9, characterized in that When replacing the original face sub-image in the original image to be beautified with the corresponding face sub-image to be beautified, the method further includes: A gradient process is performed on a boundary region between an unreplaced region in the original image to be beautified and the beautified face sub-image, so that the boundary region forms a smooth transition.

15. The method according to claim 6, characterized in that The image beautification network is a fully convolutional network, comprising: a first pixel rearrangement layer, at least one convolutional layer, at least one transposed convolutional layer, and a second pixel rearrangement layer; The step of processing the face image to be beautified by using the image beautification network to obtain a corresponding face image beautified includes: Using the first pixel rearrangement layer to perform pixel rearrangement processing on the face image to be beautified from a single channel to multiple channels to obtain a first feature image; Using the convolution layer to perform convolution processing on the first feature image to obtain a second feature image; Using the transposed convolution layer to perform a transposed convolution process on the second feature image to obtain a third feature image; The second pixel rearrangement layer is used to perform pixel rearrangement processing on the third feature image from multi-channel to single-channel to obtain the beautified face image.

16. The method according to claim 15, characterized in that The step of performing pixel rearrangement processing on the face image to be beautified from a single channel to multiple channels by using the first pixel rearrangement layer to obtain a first feature image includes: Inputting the face image to be beautified with a channel number a into the first pixel rearrangement layer; Rearrange the pixels of each n*n neighborhood in each channel of the facial image to be beautified to the same position in the n*n channels, and output the first feature image with a*n*n number of channels; Wherein, a is a positive integer, and n is a positive integer not less than 2.

17. The method according to claim 15, characterized in that The step of performing pixel rearrangement processing from multi-channel to single-channel on the third feature image using the second pixel rearrangement layer to obtain an intermediate image includes: Inputting the third feature image with a channel number of b*n*n into the second pixel rearrangement layer; Rearrange the pixels at the same position in each n*n channel of the third feature image into an n*n neighborhood in a single channel, and output the beautified face image with b channels; Wherein, b is a positive integer, and n is a positive integer not less than 2.

18. The method according to claim 1, characterized in that The method further comprises: Inputting the first sample image to be beautified into the image beautification network to be trained to output the first sample image to be beautified; Inputting a second sample image to be beautified into the image beautification network, and transforming the image output by the image beautification network by using transformation parameters to obtain a second sample image to be beautified; transforming the second sample image to be beautified by using the transformation parameters, and inputting the transformed second sample image to be beautified into the image beautification network to output a third sample image to be beautified; Based on the difference between the annotated image corresponding to the first sample image to be beautified and the first sample image, and the difference between the second sample image to be beautified and the third sample image, the parameters of the image beautification network training are updated.

19. The method according to claim 1, characterized in that The target beautification image includes a blemish-free beautification image. After obtaining the blemish-free beautification image, the method further includes: Perform personalized beautification processing on the blemish-removed beautification image to obtain a final beautification image.

20. An image beautification processing device, characterized in that: include: An image acquisition module is configured to acquire the original image to be beautified from a plurality of consecutive frames of images; a face matching module configured to match a face in the original image to be beautified with a face in a reference frame image of the original image to be beautified, and determine a stable bounding box of the face in the original image to be beautified according to a matching result; a sub-image extraction module configured to extract an original face sub-image from the original image to be beautified based on a stable bounding box of the face in the original image to be beautified; A beautification processing module is configured to process the original face sub-image using an image beautification network to obtain a corresponding beautified face sub-image; An image generation module is configured to generate a target beautified image corresponding to the original image to be beautified according to the beautified face sub-image; The step of matching the face in the original image to be beautified with the face in the reference frame image of the original image to be beautified, and determining a stable bounding box of the face in the original image to be beautified according to the matching result includes: Detecting a human face in the original image to be beautified, recording it as a to-be determined human face, and matching the to-be determined human face with a determined human face in a reference frame image of the original image to be beautified; If the to-be-determined face fails to match the determined face, expanding the basic bounding box of the to-be-determined face according to a first preset parameter to obtain a stable bounding box of the to-be-determined face; If the to-be-determined face matches the determined face successfully, a stable bounding box of the to-be-determined face is determined according to the stable bounding box of the determined face.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

22. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 19 by executing the executable instructions.

Images