A facial image enhancement processing method and system based on depth of field reconstruction

Abstract

The embodiment of the present application discloses a facial image enhancement processing method and system based on depth of field reconstruction. The technical solution provided by the embodiment of the present application obtains a live video image, intercepts a face area image from the live video image, inputs the face area image into a pre-constructed multi-layer convolutional super-resolution network for facial feature processing, and outputs a face enhanced image; segments a character background depth map from the live video image based on the picture depth prediction, blurs the live video image based on the character background depth map, and obtains a character background blurred image; and generates a face enhanced video image based on the face enhanced image and the character background blurred image. By adopting the above technical means, accurate enhancement processing of the face part of the live video can be achieved, the face enhancement processing effect can be improved, and the user's presentation of high-quality content in various video application scenarios can be met, thereby improving the user's viewing experience.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method and system for face image enhancement processing based on depth of field reconstruction. Background Art

[0002] At present, with the increasing maturity of live video technology and the popularization of related applications, the presentation quality of live video content has become a core competitiveness that all live applications need to pay attention to. In order to improve the quality of live video, considering that the audience's attention to live video is mainly focused on the foreground part of the picture (i.e., content close to the camera, changes in activities related to the main body of the character, face, central area of ​​the picture, etc.), and the background part of the picture (content far away from the camera, such as the edge outside the central area of ​​the picture, the background environment of the video that is not the main body of the character, etc.) is less noticed and not sensitive to its changes. Therefore, when processing live video, enhancing the face part of the live video that is the core of attention often has the most significant effect on improving the image quality of the audience. One of the related frontier methods of face processing is to use a generative adversarial neural network (GAN) to achieve it. A face enhancement model is constructed through adversarial training to obtain a high-performance image generation network model, so that the face image can be reconstructed through the network model, thereby achieving face detail enhancement.

[0003] However, when using the generative adversarial network model for face enhancement in live video, the face reconstructed by the generative model has some changes to the original image, and sometimes the character or facial features are distorted, resulting in deviations in the face enhancement effect. In addition, the network size of the generative adversarial network model is large, and its processing process is time-consuming and cumbersome, which makes it difficult to meet the timeliness requirements of face enhancement processing in live broadcast scenarios, resulting in a relatively poor user experience. Summary of the invention

[0004] The embodiment of the present application provides a facial image enhancement processing method and system based on depth of field reconstruction, which can enhance the foreground facial image of a live video through attention stratification and blur the background part, thereby achieving accurate enhancement processing of the facial part of the live video and solving the problem of facial processing distortion in the live video.

[0005] In a first aspect, an embodiment of the present application provides a facial image enhancement processing method based on depth of field reconstruction, comprising:

[0006] Obtain live video images, capture face area images from the live video images, input the face area images into a pre-built multi-layer convolutional super-resolution network for face feature processing, and output face enhanced images;

[0007] Based on the picture depth prediction, a character background depth map is segmented from the live video image, and the live video image is blurred based on the character background depth map to obtain a character background blurred image;

[0008] A face-enhanced video image is generated by image fusion based on the face-enhanced image and the person's background blurred image.

[0009] In a second aspect, an embodiment of the present application provides a facial image enhancement processing system based on depth of field reconstruction, comprising:

[0010] A face enhancement module is configured to obtain a live video image, intercept a face region image from the live video image, input the face region image into a pre-built multi-layer convolutional super-resolution network to process face features, and output a face enhanced image;

[0011] A background blur module is configured to segment a character background depth map from a live video image based on a picture depth prediction, and blur the live video image based on the character background depth map to obtain a character background blur image;

[0012] The fusion module is configured to generate a face-enhanced video image by performing image fusion based on the face-enhanced image and the person background blurred image.

[0013] In a third aspect, an embodiment of the present application provides a facial image enhancement processing device based on depth of field reconstruction, comprising:

[0014] memory and one or more processors;

[0015] The memory is configured to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the facial image enhancement processing method based on depth of field reconstruction as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a computer processor, are configured to execute the facial image enhancement processing method based on depth of field reconstruction as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the facial image enhancement processing method based on depth of field reconstruction as described in the first aspect.

[0019] The embodiment of the present application obtains a live video image, captures a face area image from the live video image, inputs the face area image into a pre-constructed multi-layer convolutional super-resolution network to process face features, and outputs a face enhanced image; based on the picture depth prediction, a character background depth map is segmented from the live video image, and the live video image is blurred based on the character background depth map to obtain a character background blurred image; and image fusion is performed based on the face enhanced image and the character background blurred image to generate a face enhanced video image. The above technical means are adopted to perform layered processing of the face area and the character background area of ​​the live video image, enhance the foreground face image, and blur the background part, so as to achieve accurate enhancement of the face part of the live video, avoid distortion caused by enhancement of the entire live video image, improve the face enhancement processing effect, meet the user's demand for high-quality content presentation in various video application scenarios, and thus improve the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of a facial image enhancement processing method based on depth of field reconstruction provided in an embodiment of the present application;

[0021] Figure 2 is a flow chart of capturing a face area image in an embodiment of the present application;

[0022] Figure 3 This is a flow chart of face enhanced image generation in an embodiment of the present application;

[0023] Figure 4 Schematic diagram of face enhancement processing of a multi-layer convolutional super-resolution network in an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of live video image segmentation in an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of the fusion of a face enhancement image and a person background blur image in an embodiment of the present application;

[0026] Figure 7 is a structural schematic diagram of a facial image enhancement processing system based on depth of field reconstruction provided in an embodiment of the present application;

[0027] Figure 8 It is a structural schematic diagram of a facial image enhancement processing device based on depth of field reconstruction provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0029] The facial image enhancement processing method based on depth of field reconstruction provided in this application aims to enhance the foreground facial image of the live video through attention layering, blur the background part, and achieve accurate enhancement processing of the facial part of the live video.

[0030] As live video technology matures and related applications become more popular, viewers are more adaptable to this new form of entertainment interaction. At the same time, the quality of the presentation of picture content has become more and more the core competitiveness of various live applications. In order to improve the quality of live video, previous solutions have focused more on adjusting the color of the picture and enriching the details of the picture (such as super-resolution, color enhancement, etc.). When watching live broadcasts, the audience's attention is not consistent and has obvious stratification, that is, the audience's attention to different parts of the entire picture content is different. Among them, the audience's main attention is mainly focused on the foreground part of the picture (content close to the camera, changes in activities related to the main body of the character, human face, the center circle area of ​​the picture, etc.), and has a more obvious preference and stay for the content with clear details in the foreground area. For the part of the picture that is far away from the camera, such as the edge outside the center area of ​​the picture, the video background environment of the non-human subject, etc., pay less attention to it and are not sensitive to its changes.

[0031] For this purpose, the details of the foreground of the picture, especially the face, are usually enhanced or repaired to improve the quality of this part of the picture content. At present, the more mainstream technical idea of ​​face detail enhancement is mainly achieved through image super resolution (ISR) technology. Image super resolution (ISR) technology mainly enlarges the size of low-resolution images in different proportions while ensuring that no detailed features of the image are lost. The image is downsampled (Downsampling) and then upsampled (Upsampling) the same number of times using a convolutional deep neural network (CNN) multiple times, so as to use different convolutional network layers to learn features of different granularities, supplement image details in the upsampling process, and obtain high-definition images that are enlarged several times without losing details.

[0032] However, super-resolution technology generally requires a large amount of computation, and the super-resolution effect is limited. It is generally difficult to perform real-time calculations. With the further exploration of the CNN model structure and the innovation of the network structure, a super-resolution network has emerged to improve computational efficiency and performance. For example, the RLFN network uses residual connections to preserve detail information for multi-scale image features, and the SPAN network uses the attention mechanism (Attention) to highlight the weight of texture details in the image, achieving good results in computational efficiency and super-resolution effect. However, as a universal image lossless magnification technology, image super-resolution is not specifically designed for processing face images, and in actual application scenarios, unknown image problems (such as noise, blur, etc.) are often encountered, and super-resolution technology rarely involves these problems.

[0033] In addition, another cutting-edge facial detail enhancement method is based on the Generative Adversarial Network (GAN). The GAN achieves a balance through adversarial training to obtain a high-performance image generation network model. The face image can be reconstructed through the image generation network model to achieve facial detail enhancement. However, the image generation network model still takes into account some extreme image degradation methods, so that the face restored by the generation model has a certain degree of change to the original image, and sometimes the character or facial features are distorted. In addition, the size of such networks is very large, so the image generation network model is mostly based on the stored face image for repair, and it is difficult to process video images in scenes such as live video that have timeliness requirements. Its application in continuous character-related videos may also have time discontinuity.

[0034] Based on this, a facial image enhancement processing method based on depth of field reconstruction is provided in an embodiment of the present application to solve the problem of face processing distortion in live video. By constructing a depth of field reconstruction technology solution based on attention perception layering for actual live broadcast and video streaming scenarios, the image content is layered by attention using depth estimation, and while retaining the complete details of the foreground, the core foreground area of ​​the audience's attention, i.e., the face, is enhanced in detail and the unknown degradation is restored. At the same time, the background is blurred with bokeh effect, so as to highlight the core area of ​​the audience's attention and improve the overall subjective experience.

[0035] Example:

[0036] Figure 1 A flowchart of a facial image enhancement processing method based on depth of field reconstruction provided in an embodiment of the present application is given. The facial image enhancement processing method based on depth of field reconstruction provided in this embodiment can be executed by a facial image enhancement processing device based on depth of field reconstruction. The facial image enhancement processing device based on depth of field reconstruction can be implemented by software and / or hardware. The facial image enhancement processing device based on depth of field reconstruction can be composed of two or more physical entities, or it can be composed of one physical entity. Generally speaking, the facial image enhancement processing device based on depth of field reconstruction can be a processing device such as a live video server, a live server image processing device, etc.

[0037] The following description is made by taking the face image enhancement processing device based on depth of field reconstruction as an example of the main body of the face image enhancement processing method based on depth of field reconstruction. Figure 1 The facial image enhancement processing method based on depth of field reconstruction specifically includes:

[0038] S110, obtaining a live video image, capturing a face region image from the live video image, inputting the face region image into a pre-built multi-layer convolutional super-resolution network for face feature processing, and outputting a face enhanced image.

[0039] In the live broadcast scene, the present application processes the live video images sent to the audience end to achieve facial image enhancement processing based on depth of field reconstruction. Since it is generally believed that there are different priorities of attention based on the audience's attention in the live broadcast scene, by stratifying the content of the picture, focusing on enhancing the details of the foreground part, and properly reconstructing the details of the picture content, further highlighting the main foreground that the audience is more concerned about, the benefits of improving the subjective experience of the video are obvious. Therefore, the present application constructs an attention stratification strategy based on depth estimation (Depth Estimation), which fully retains the content details of the foreground whose depth of field is close to the lens, and performs additional foreground enhancement on the facial details of the face that the audience mainly watches in the live broadcast scene, and blurs the background part that is divided into the edge of the depth of field attention, which in turn further enhances the detail perception of the main part.

[0040] For example, refer to Figure 2 In the live broadcast scenario, the video stream recorded or uploaded by the anchor end is then connected to the face image enhancement processing device based on depth of field reconstruction after a series of encoding processing and transmission links. The face image enhancement processing device based on depth of field reconstruction of this application is connected to extract frames of the video and judge whether there is a face. If there is no face, the process ends and the face enhancement processing of the frame is not performed. If there is a face, the face image enhancement processing process based on depth of field reconstruction is triggered.

[0041] It is understandable that the presentation of video content is not clear enough in some cases due to the quality of the video shooter's equipment, shooting techniques, lighting and other factors, and is accompanied by one or more quality problems such as noise, blur, breathing effect, etc. In order to improve the user's viewing experience and such unclear face videos, the live video image is obtained by extracting frames from the live video stream, and it is determined whether face enhancement processing needs to be started. The method of triggering face enhancement processing can be determined based on parameters such as the clarity of the face image, and this application does not make any fixed restrictions here.

[0042] Specifically, the live video image is pre-processed to determine whether face enhancement is required. If face enhancement is required, the live video image to be enhanced is obtained and the face area image f_crop is defined according to the face detection frame BBOX. First, a live video image origin_img with a size of H*W*3 (3 represents the three color channels of the image) and a pixel value range of [0, 255] is read in. The live video image is sent to a face detector algorithm to determine whether there is a face in the current video. Since the face is identified as the core of the foreground audience's attention, non-face videos are skipped and not processed considering their enhancement benefits. For the picture frame containing the face, the face detector can obtain the detection rectangle BBOX containing the face. According to the coordinates of the face detection rectangle, the length and width of the face detection area contained in it are further calculated, which are recorded as face_W and face_H respectively. The face in the BBOX rectangle is cut out to become the f_crop image block. The f_crop image block is numerically normalized before being input into the enhancement network, and the f_crop value range is constrained to the range of [0, 1] to obtain a normalized face area image.

[0043] Furthermore, based on the face area image captured from the live video image, the present application processes the face features by inputting the face area image into a pre-built multi-layer convolutional super-resolution network to output a face enhanced image.

[0044] Reference Figure 3 , input the face area image into the pre-built multi-layer convolutional super-resolution network for face feature processing, and output the face enhanced image, including:

[0045] S1101, inputting the face region image into a pre-built multi-layer convolutional super-resolution network, and extracting shallow features of the face region image based on the convolution module of the multi-layer convolutional super-resolution network;

[0046] S1102, inputting shallow features into a multi-layer attention module of a multi-layer convolutional super-resolution network, and outputting corresponding multiple levels of high-level features and feature weight information;

[0047] S1103: Connect high-level features at each level according to corresponding feature weight information, and output a face enhanced image.

[0048] This application designs a multi-layer convolutional super-resolution network based on the attention mechanism. Figure 4As shown in the figure, the face region image f_crop of the original face with a size of face_H*face_W*3 and pixel value range normalized to the interval [0, 1] is first input into the network and then passed through a convolutional module for feature extraction to collect shallow features of the image. The shallow features of the preliminary image collected are sequentially input into multiple attention modules, which gradually extract high-level features and focus on the parts with rich high-frequency details, and configure them with higher prominence weights. After the six attention modules refine and extract the features, the residual connection layer combines the sequentially calculated feature maps of different granularities to form a comprehensive final feature. The pixelShuffle network layer is used to complete the image scaling in a small range and the final result output, that is, the enhanced face enhanced image enhanced_f_crop. It can be observed that the input image has obtained significantly richer details and a clearer look and feel after enhancement.

[0049] Optionally, the multi-layer convolutional super-resolution network can also construct a complete set of image degradation processes close to the application scenario, involving multiple degradation types such as noise, blur, and coding effects, in order to establish paired low-definition and high-definition images for supervised learning. On the other hand, the adversarial generative network (GAN) loss can also be used. During the training process, in addition to the super-resolution network as a generator to construct as rich details as possible, an additional discriminator is added to strive to distinguish the generated face details from the real face details as much as possible, so as to enable the super-resolution network to construct more realistic facial details during the adversarial learning training process.

[0050] Based on this, for unknown multiple degraded low-resolution images LR (i.e., face area images), feature extraction and detail weight focusing are performed through convolution modules (Conv) and multiple attention modules (Attention Modules), and then features at different levels are connected together (Concat), and the final high-definition effect image HR (i.e., face enhanced image) is output through the final PixelShuffle network layer.

[0051] S120, segmenting a character background depth map from the live video image based on the picture depth prediction, and blurring the live video image based on the character background depth map to obtain a character background blurred image.

[0052] In addition to enhancing the face part of the foreground of the live video image, this application also blurs the background part to highlight the foreground content by reducing the background clarity. Among them, a video content segmentation algorithm is used to read in a single-frame BGR color channel live video image origin_img with a size of H*W*3 and a pixel value range of [0, 255] for inference, and the character background area is segmented from it by predicting the depth of the picture, that is, the character background depth map.

[0053] The method of segmenting a person background depth map from a live video image based on picture depth prediction includes:

[0054] Based on the picture depth prediction, the character foreground area is determined from the live video image, and the character background depth map is segmented from the live video image based on the character foreground area.

[0055] like Figure 5 As shown, by processing the live video image origin_img, a semantic segmentation map H*W*1 of the same size as the input image frame is obtained, including a depth map depth_mask corresponding to the foreground area of ​​the person in the live video image. After obtaining the depth map depth_mask of the same size as the live video image, the portrait foreground can be segmented out by the depth map approximation, and the other area is the background area, so that the depth map of the person background can be obtained.

[0056] Optionally, according to actual needs, different depth areas of the live video image can be determined based on the picture depth prediction, and then according to the division of the depth area, some depth areas away from the camera are selected as the character background area, so as to segment the character background depth map. This application does not impose fixed restrictions on the segmentation method of the character background depth map, and will not be elaborated here.

[0057] Furthermore, based on the character background depth map, blur processing similar to SLR bokeh blur can be performed on the non-foreground area of ​​the live video image, and the blur degree can be adjusted through different kernel functions to obtain a video frame blur_img with prominent foreground, blurred background and appropriate visual effect, that is, a character background blurred image.

[0058] S130: Generate a face-enhanced video image by performing image fusion based on the face-enhanced image and the person background blurred image.

[0059] After completing the network inference for the face f_crop to determine the face enhanced image and the person background area inference to obtain the person background blurred image, the face enhanced image enhance_f_crop with detail enhancement and size face_H*face_W is obtained, and the person background blurred image blur_img with the same size H*W as the original video frame after the background is blurred is also obtained. The final face enhanced video image is obtained by fusing the two images.

[0060] Among them, refer to Figure 6 , based on the face enhancement image and the person background blur image, the image fusion is performed to generate the face enhancement video image, including:

[0061] Paste the face enhancement image to the corresponding coordinate position of the corresponding live video image according to the coordinates of the face area image to generate a face enhancement frame;

[0062] The corresponding pixel coordinates of the live video image are assigned according to the face enhancement frame and the person background blur image to generate a face enhancement video image.

[0063] First, the enhanced face image is pasted to the corresponding coordinate position of a blank image of the same size as the original video frame according to the same position of the face area image (corresponding to the face detection frame) to obtain the face enhanced frame enhanced_img. Then, according to the values ​​of each pixel coordinate of the face enhanced frame and the person background blurred image, the corresponding pixel coordinates of the live video image are assigned to obtain the face enhanced video image.

[0064] The corresponding pixel coordinates of the live video image are assigned according to the face enhancement frame and the person background blur image, including:

[0065] Obtain a binary face mask image of the face region image, and determine a target region assigned a value of 1 in the binary face mask image;

[0066] Determine the pixel values ​​of the pixel coordinates of the target area of ​​the live video image based on the face enhancement frame;

[0067] The pixel values ​​of the pixel coordinates of other areas of the live video image are determined based on the blurred background image of the person.

[0068] When performing depth prediction on live video images, the coordinates of the face area image (corresponding to the face detection frame) are used to segment the binary face mask face_mask corresponding to the face area, and the pixel coordinate area assigned a value of 1 in the mask is determined, which is defined as the target area. The target area is filled with the value of the face enhancement frame, and the rest of the area is filled with the pixel values ​​corresponding to the blur background image blur_img of the person. The calculation formula is as follows:

[0069] final_img=blur_img*(1-face_mask)+enhanced_img*face_mask

[0070] At this time, the face enhancement video image with the face part enhanced and the background blurred is output as the final output final_img, and the entire face enhancement processing flow is completed.

[0071] When the live video image meets the enhancement conditions, the live video image is sent frame by frame as input to the face image enhancement processing device for depth estimation, and the foreground and background segmentation based on attention stratification is completed. The segmented foreground face is enhanced for facial details, and the background is blurred by bokeh defocusing. The processed foreground and background are fused to obtain the enhanced result of foreground highlighting and background blurring, and then transmitted to the background for corresponding distribution and processing, and the result is presented to the client for the corresponding viewing user. In this way, the reconstruction of the picture based on depth of field through attention stratification can significantly improve the subjective feeling of users when watching videos. It can be clearly felt that the overall perception of the face that may be blurred or otherwise degraded in the original video has been improved by comparing the images before and after enhancement. The whole picture has a clear subjectivity highlighting of the characters, and the facial details are richer, which overall enhances the picture quality and viewing experience of the video when the anchor starts the live broadcast. On the other hand, from a technical point of view, it can effectively reduce the video bit rate and save computing costs while ensuring the improvement of the audience experience.

[0072] Optionally, after determining the person foreground area from the live video image based on the picture depth prediction, the method further includes:

[0073] Determine the target area after filtering out the face area image from the foreground area of ​​the person, determine the depth areas at different levels according to the different depths of each pixel coordinate in the target area, perform feature processing at the corresponding level on each depth area, and generate the target image;

[0074] The face-enhanced video image is generated by fusing the face-enhanced image and the person's background blur image, and further includes:

[0075] The target image is fused with the face enhanced image to generate a person foreground enhanced image;

[0076] The person foreground enhanced image and the person background blurred image are fused to generate a face enhanced video image.

[0077] Considering that the audience's attention to the non-face part of the foreground area of ​​the person is between the background area and the face, and according to the distance from the camera, the audience's attention may gradually decrease from near to far. Based on this, the present application refers to the above-mentioned enhancement processing method of the face area image, and adaptively performs enhancement processing on this part of the area. By determining the target area after the face area image is filtered out from the foreground area of ​​the person, the areas of different depths of the target area are layered according to the depth prediction method, and then the image of the target area is input into the super-resolution network, and the feature processing is performed on each depth area to obtain an enhanced target image. Then, when the face-enhanced image and the person's background blurred image are used to generate the face-enhanced video image, the value of the target image is assigned to the target area according to the coordinates of the target area, and the pixel value of the pixel coordinates of the live video image corresponding to the target area is determined based on the face enhancement frame, so that the target image and the face-enhanced image are fused to generate the person's foreground enhanced image. Similarly, the pixel values ​​of the pixel coordinates of other areas of the live video image are determined based on the person's background blurred image, so that the person's foreground enhanced image and the person's background blurred image are fused to generate the face-enhanced video image.

[0078] Optionally, feature processing of corresponding levels is performed on each depth region, including:

[0079] Feature weights are configured according to the depth of each depth region from high to low, and feature processing is performed on each depth region based on the configured feature weights.

[0080] By determining the target area after filtering out the face area image from the foreground area of ​​the person, the target area is layered according to the depth prediction method for each area with different depths, and the feature weight of each layer is gradually reduced from shallow to deep. The area closer to the face has a stronger enhancement effect, but the enhancement weight is smaller than the face part. Based on the feature weights configured above, the image of the target area can be input into the super-resolution network, and feature processing corresponding to the feature weights is performed on each area to obtain an enhanced target image. A more detailed multi-layer attention face enhancement process is achieved.

[0081] In the above, by acquiring a live video image, capturing a face area image from the live video image, inputting the face area image into a pre-built multi-layer convolutional super-resolution network for face feature processing, and outputting a face enhanced image; segmenting the character background depth map from the live video image based on the picture depth prediction, blurring the live video image based on the character background depth map, and obtaining a character background blurred image; and generating a face enhanced video image by image fusion based on the face enhanced image and the character background blurred image. The above technical means are adopted to perform layered processing of the face area and the character background area of ​​the live video image, enhance the foreground face image, and blur the background part, so as to achieve accurate enhancement of the face part of the live video, avoid distortion caused by enhancement of the entire live video image, improve the face enhancement processing effect, meet the user's demand for high-quality content presentation in various video application scenarios, and thus improve the user's viewing experience.

[0082] Based on the above embodiments, Figure 7 This is a schematic diagram of the structure of a facial image enhancement processing system based on depth of field reconstruction provided by this application. Figure 7 The face image enhancement processing system based on depth of field reconstruction provided in this embodiment specifically includes: a face enhancement module 21, a background blur module 22 and a fusion module 23.

[0083] The face enhancement module 21 is configured to obtain a live video image, capture a face region image from the live video image, input the face region image into a pre-built multi-layer convolutional super-resolution network for face feature processing, and output a face enhanced image;

[0084] Background blur module 22, configured to segment a character background depth map from a live video image based on picture depth prediction, and blur the live video image based on the character background depth map to obtain a character background blur image;

[0085] The fusion module 23 is configured to generate a face-enhanced video image by performing image fusion based on the face-enhanced image and the person background blurred image.

[0086] Specifically, the face region image is input into the pre-built multi-layer convolutional super-resolution network for face feature processing, and the face enhanced image is output, including:

[0087] The face area image is input into a pre-built multi-layer convolutional super-resolution network, and the shallow features of the face area image are extracted based on the convolution module of the multi-layer convolutional super-resolution network;

[0088] Input shallow features into the multi-layer attention module of the multi-layer convolutional super-resolution network, and output the corresponding multi-level high-level features and feature weight information;

[0089] The high-level features at each level are connected according to the corresponding feature weight information, and the face enhanced image is output.

[0090] Specifically, segmenting a person background depth map from a live video image based on picture depth prediction includes:

[0091] Based on the picture depth prediction, the character foreground area is determined from the live video image, and the character background depth map is segmented from the live video image based on the character foreground area.

[0092] After determining the person foreground area from the live video image based on the picture depth prediction, the method further includes:

[0093] Determine the target area after filtering out the face area image from the foreground area of ​​the person, determine the depth areas at different levels according to the different depths of each pixel coordinate in the target area, perform feature processing at the corresponding level on each depth area, and generate the target image;

[0094] The face-enhanced video image is generated by fusing the face-enhanced image and the person's background blur image, and further includes:

[0095] The target image is fused with the face enhanced image to generate a person foreground enhanced image;

[0096] The person foreground enhanced image and the person background blurred image are fused to generate a face enhanced video image.

[0097] Perform feature processing at the corresponding level for each depth region, including:

[0098] Feature weights are configured according to the depth of each depth region from high to low, and feature processing is performed on each depth region based on the configured feature weights.

[0099] The face-enhanced video image is generated by performing image fusion based on the face-enhanced image and the person background blur image, including:

[0100] Paste the face enhancement image to the corresponding coordinate position of the corresponding live video image according to the coordinates of the face area image to generate a face enhancement frame;

[0101] The corresponding pixel coordinates of the live video image are assigned according to the face enhancement frame and the person background blur image to generate a face enhancement video image.

[0102] Specifically, the corresponding pixel coordinates of the live video image are assigned according to the face enhancement frame and the person background blur image, including:

[0103] Obtain a binary face mask image of the face region image, and determine a target region assigned a value of 1 in the binary face mask image;

[0104] Determine the pixel values ​​of the pixel coordinates of the target area of ​​the live video image based on the face enhancement frame;

[0105] The pixel values ​​of the pixel coordinates of other areas of the live video image are determined based on the blurred background image of the person.

[0106] In the above, by acquiring a live video image, capturing a face area image from the live video image, inputting the face area image into a pre-built multi-layer convolutional super-resolution network for face feature processing, and outputting a face enhanced image; segmenting the character background depth map from the live video image based on the picture depth prediction, blurring the live video image based on the character background depth map, and obtaining a character background blurred image; and generating a face enhanced video image by image fusion based on the face enhanced image and the character background blurred image. The above technical means are adopted to perform layered processing of the face area and the character background area of ​​the live video image, enhance the foreground face image, and blur the background part, so as to achieve accurate enhancement of the face part of the live video, avoid distortion caused by enhancement of the entire live video image, improve the face enhancement processing effect, meet the user's demand for high-quality content presentation in various video application scenarios, and thus improve the user's viewing experience.

[0107] The facial image enhancement processing system based on depth of field reconstruction provided in the embodiment of the present application can be configured to execute the facial image enhancement processing method based on depth of field reconstruction provided in the above embodiment, and has corresponding functions and beneficial effects.

[0108] Based on the above practical example, the embodiment of the present application also provides a face image enhancement processing device based on depth of field reconstruction, referring to Figure 8, the facial image enhancement processing device based on depth of field reconstruction includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the facial image enhancement processing method based on depth of field reconstruction described in any embodiment of the present application (for example, a face enhancement module, a background blur module and a fusion module in a facial image enhancement processing system based on depth of field reconstruction). The communication module is configured to perform data transmission. The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, realizing the above-mentioned facial image enhancement processing method based on depth of field reconstruction. The input device can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device can include display devices such as display screens. The facial image enhancement processing device based on depth of field reconstruction provided above can be configured to execute the facial image enhancement processing method based on depth of field reconstruction provided in the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0109] On the basis of the above embodiments, the embodiments of the present application further provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are configured to execute a facial image enhancement processing method based on depth of field reconstruction when executed by a computer processor, and the storage medium may be any of various types of memory devices or storage devices. Of course, the non-volatile computer-readable storage medium provided in the embodiments of the present application, whose computer-executable instructions are not limited to the facial image enhancement processing method based on depth of field reconstruction as described above, may also execute related operations in the facial image enhancement processing method based on depth of field reconstruction provided in any embodiment of the present application.

[0110] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the facial image enhancement processing method based on depth of field reconstruction described in each embodiment of the present application.

Claims

1. A facial image enhancement processing method based on depth of field reconstruction, characterized in that: include: Acquire a live video image, capture a face region image from the live video image, input the face region image into a pre-built multi-layer convolutional super-resolution network to process face features, and output a face enhanced image; Segmenting a character background depth map from the live video image based on picture depth prediction, and blurring the live video image based on the character background depth map to obtain a character background blurred image; The face-enhanced image and the person background blurred image are fused to generate a face-enhanced video image.

2. The facial image enhancement processing method based on depth of field reconstruction according to claim 1 is characterized in that: Inputting the face region image into a pre-built multi-layer convolutional super-resolution network to process face features and outputting a face enhanced image, including: Inputting the face region image into a pre-constructed multi-layer convolutional super-resolution network, and extracting shallow features of the face region image based on the convolution module of the multi-layer convolutional super-resolution network; Input the shallow features into the multi-layer attention module of the multi-layer convolutional super-resolution network, and output the corresponding multiple levels of high-level features and feature weight information; The high-level features at each level are connected according to the corresponding feature weight information, and a face enhanced image is output.

3. The facial image enhancement processing method based on depth of field reconstruction according to claim 1 is characterized in that: The method of segmenting a character background depth map from the live video image based on picture depth prediction includes: A character foreground area is determined from the live video image based on picture depth prediction, and a character background depth map is segmented from the live video image based on the character foreground area.

4. The facial image enhancement processing method based on depth of field reconstruction according to claim 3 is characterized in that: After determining the person foreground area from the live video image based on the picture depth prediction, the method further includes: Determine a target area after filtering out the face area image from the person foreground area, determine depth areas of different levels according to different depths of each pixel coordinate of the target area, perform feature processing of a corresponding level on each of the depth areas, and generate a target image; The generating of the face enhanced video image by performing image fusion based on the face enhanced image and the person background blurred image also includes: Fusion of the target image with the face-enhanced image to generate a person foreground-enhanced image; The person foreground enhanced image and the person background blurred image are fused to generate a face enhanced video image.

5. The facial image enhancement processing method based on depth of field reconstruction according to claim 4 is characterized in that: The performing feature processing of the corresponding level on each of the depth regions includes: Feature weights are configured corresponding to the depths of the depth regions from high to low, and feature processing is performed on the depth regions based on the configured feature weights.

6. The facial image enhancement processing method based on depth of field reconstruction according to any one of claims 1 to 5, characterized in that: The generating of the face enhanced video image by performing image fusion based on the face enhanced image and the person background blurred image includes: Pasting the face-enhanced image to the corresponding coordinate position of the live video image according to the coordinates of the face region image to generate a face-enhanced frame; The corresponding pixel coordinates of the live video image are assigned according to the face enhancement frame and the person background blurred image to generate a face enhancement video image.

7. The facial image enhancement processing method based on depth of field reconstruction according to claim 6 is characterized in that: The assigning of corresponding pixel coordinates of the live video image according to the face enhancement frame and the person background blur image includes: Obtain a binary face mask image of the face region image, and determine a target region assigned a value of 1 in the binary face mask image; Determine the pixel value of the pixel coordinate of the live video image corresponding to the target area based on the face enhancement frame; The pixel values ​​of the pixel coordinates of other areas of the live video image are determined based on the blurred background image of the person.

8. A facial image enhancement processing system based on depth of field reconstruction, characterized in that: include: A face enhancement module is configured to obtain a live video image, intercept a face region image from the live video image, input the face region image into a pre-built multi-layer convolutional super-resolution network to perform face feature processing, and output a face enhanced image; A background blur module is configured to segment a character background depth map from the live video image based on picture depth prediction, and blur the live video image based on the character background depth map to obtain a character background blur image; The fusion module is configured to generate a face-enhanced video image by performing image fusion based on the face-enhanced image and the person background blurred image.

9. A facial image enhancement processing device based on depth of field reconstruction, characterized in that: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the facial image enhancement processing method based on depth of field reconstruction as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by a computer processor, are configured to execute the facial image enhancement processing method based on depth of field reconstruction as described in any one of claims 1-7.

11. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the facial image enhancement processing method based on depth of field reconstruction as described in any one of claims 1 to 7.

Images