Method, device and computer equipment for face video quality enhancement

Abstract

The present application relates to a kind of for face video picture quality enhancement method, device and computer equipment, method includes obtaining the video frame image of video stream, according to frame image obtains frame to be enhanced face map, obtains frame enhanced face after enhancement processing, to obtain enhanced frame;When processing subsequent frame image, the first optical flow between last frame to be enhanced face map and current frame to be enhanced face map and first optical flow occlusion area mask are calculated, obtain current frame optical flow enhanced face;Current frame enhanced face is obtained by mask enhancement model, after all video frame image is enhanced and is handled, it is combined as enhanced video.The present application uses low-resolution original frame as guide, carries out detail enhancement to the occlusion area in optical flow result, carries out detail fine adjustment to non-occlusion area.Meanwhile, the present application only carries out the calculation and optical flow mapping of face part optical flow, greatly improves the processing performance of algorithm, to realize the purpose that face video obtains stable and efficient picture quality enhancement.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a method and device for face video quality enhancement and a computer device. BACKGROUND

[0002] With the gradual improvement of mobile phone hardware performance and software function richness, users have higher requirements for the definition and resolution of videos that they shoot or import from other devices, thus giving rise to the demand for video quality enhancement on mobile terminals, especially for videos containing people or faces.

[0003] In related technologies, in order to improve the visual experience of users watching videos, video enhancement technology is often used to improve the definition of videos watched by users. In existing mobile terminal tool applications that mainly improve video definition, a common practice is to crop the part of the face in the video, because users' perception of faces is stronger than that of backgrounds, so the face is cropped out, and then different enhancement technologies are used to process the face and the background. In this process, the model used to enhance the face consumes a large amount of computing resources; and when repairing low-resolution faces, the above face enhancement model needs to perform a detail supplement operation different from processing non-low-resolution faces, which may cause the actual enhancement effect of the same face to be inconsistent between different frames, resulting in flickering of the overall quality enhancement result. In order to solve the flickering problem, optical flow is generally used to increase the continuity between frames. However, since there is movement between frames, some pixels are blocked, affecting the enhancement effect. At the same time, the result of the optical flow model is also relied on to a large extent, and when the optical flow model is inaccurate, it will have a negative impact on the enhancement result. SUMMARY

[0004] Therefore, the present application aims to overcome the shortcomings of the prior art and provide a method and device for face video quality enhancement and a computer device to solve the problem that the optical flow model is inaccurate in the prior art, which has a negative impact on the enhancement result.

[0005] To achieve the above object, the present application adopts the following technical solution: a method for face video quality enhancement, comprising:

[0006] obtaining a video frame image of a video stream, identifying a face in a first frame image to obtain a first frame face frame, cropping the face in the first frame face frame to obtain a first frame to-be-enhanced face image;

[0007] performing enhancement processing on the first frame to-be-enhanced face image to obtain a first frame enhanced face, and placing the first frame enhanced face in a corresponding position in the first frame face frame to obtain a first enhanced frame.

[0008] When processing the subsequent frame image, a current frame face frame is obtained according to the current frame image, and a current frame face to be enhanced is obtained through the current frame face frame;

[0009] A first optical flow and a first optical flow occlusion area mask between the last frame face to be enhanced and the current frame face to be enhanced are calculated, and the first optical flow is mapped to the last frame enhanced face to obtain a current frame optical flow enhanced face;

[0010] The current frame optical flow enhanced face, the first optical flow occlusion area mask and the current frame face to be enhanced are input into a pre-constructed mask enhancement model to obtain a current frame enhanced face;

[0011] After the enhancement processing of all video frame images in the video stream is completed, the enhanced video is combined.

[0012] Further, if the face in the current frame image is multiple, the face in the current frame image is matched with the face in the last enhanced frame, including:

[0013] The pixel distance value of each face to be enhanced in the last frame and each face to be enhanced in the current frame image is calculated;

[0014] All distances are traversed from small to large, if the distance is less than a preset threshold, and the face to be enhanced in the previous frame corresponding to the distance and the face to be enhanced in the current frame are not matched before, it is considered that the matching is successful; after all the distances are traversed, the face to be enhanced in the current frame which is not matched is processed as a new face.

[0015] Further, when a new face appears in the current frame image, the new face in the current frame image is recognized to obtain a new face frame, and the face in the new face frame is cropped to obtain a face to be enhanced;

[0016] The face to be enhanced is enhanced to obtain a frame enhanced face, and the frame enhanced face is placed in the corresponding position of the new face frame to obtain a new face enhanced frame.

[0017] Further, the pre-constructed mask enhancement model includes:

[0018] Extracting a first high-quality frame image and a second high-quality frame image in a continuous high-definition video stream;

[0019] The first high-quality frame image and the second high-quality frame image are degraded to obtain a first low-quality frame image and a second low-quality frame image;

[0020] perform face recognition on the first low-quality frame image and the second low-quality frame image respectively to obtain a first low-quality frame face frame and a second low-quality frame face frame, and perform face cropping on the first low-quality frame image and the second low-quality frame image based on the first low-quality frame face frame and the second low-quality frame face frame to obtain a first low-quality frame face and a second low-quality frame face;

[0021] perform face recognition on the first high-quality frame image and the second high-quality frame image respectively to obtain a first high-quality frame face frame and a second high-quality frame face frame, and perform face cropping on the first high-quality frame image and the second high-quality frame image based on the first high-quality frame face frame and the second high-quality frame face frame to obtain a first high-quality frame face and a second high-quality frame face;

[0022] calculate a second optical flow and a second optical flow occlusion area mask between the first low-quality frame face and the second low-quality frame face by using an optical flow algorithm;

[0023] map the second optical flow to the first high-quality frame face to obtain a second frame optical flow enhanced face;

[0024] train a deep neural network based on a target loss function and the second frame optical flow enhanced face, the second optical flow occlusion area mask and the second low-quality frame face to obtain a mask enhancement model, wherein the mask enhancement model is configured to output a second frame enhanced face.

[0025] Further, the first high-quality frame image and the second high-quality frame image are subjected to degradation processing, including:

[0026] the first high-quality frame image and the second high-quality frame image are subjected to noise addition, blurring and format change processing respectively.

[0027] Further, the loss function of the mask enhancement model is

[0028] L = l_pix (G (x) * m, gt * m) + 0.1 * l_pix (G (x) * (1. - m), gt * (1. - m) )

[0029] wherein m is the normalized "second optical flow occlusion area mask"; x is the input of the mask enhancement model; G(x) is the output result of the mask enhancement model, i.e. the second frame optical flow enhanced face, and gt is the target enhancement result of the mask enhancement model, i.e. the second high-quality frame face; l_pix(A,B) is a loss function for calculating the pixel-by-pixel difference between images A and B; l_pix(G(x)*m,gt*m) is a loss for supervising the enhancement result of the occluded part; 0.1*l_pix(G(x)*(1.-m),gt*(1.-m)) is a loss for supervising the enhancement result of the non-occluded part.

[0030] Further, the face in the first frame face frame is cropped to obtain a first frame face to be enhanced.

[0031] After recognizing the face in the first frame image, the face is scaled to a preset size.

[0032] The face of the preset size is taken as the first frame face to be enhanced.

[0033] The embodiment of the application provides a device for face video quality enhancement, comprising:

[0034] The acquisition module is configured to acquire a video frame image of a video stream, recognize a face in a first frame image, obtain a first frame face frame, crop the face in the first frame face frame, and obtain a first frame face to be enhanced.

[0035] The first processing module is configured to perform enhancement processing on the first frame face to be enhanced to obtain a first frame enhanced face, place the first frame enhanced face in a corresponding position in the first frame face frame, and obtain a first enhanced frame.

[0036] The second processing module is configured to, when processing a subsequent frame image, obtain a current frame face frame according to a current frame image, and obtain a current frame face to be enhanced through the current frame face frame.

[0037] The calculation module is configured to calculate a first optical flow and a first optical flow occlusion area mask between a last frame face to be enhanced and the current frame face to be enhanced, map the first optical flow to a last frame enhanced face, and obtain a current frame optical flow enhanced face.

[0038] The output module is configured to input the current frame optical flow enhanced face, the first optical flow occlusion area mask, and the current frame face to be enhanced into a pre-constructed mask enhancement model to obtain a current frame enhanced face.

[0039] The enhancement processing module is configured to, after performing enhancement processing on all video frame images in the video stream, combine the video frame images into an enhanced video.

[0040] The embodiment of the application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of any one of the above-mentioned methods for face video quality enhancement.

[0041] The above technical scheme can achieve the following beneficial effects.

[0042] The application provides a method, device and computer equipment for face video quality enhancement, which utilizes optical flow to ensure the continuity of face video enhancement results, and uses a low-resolution original frame image as a guide to enhance the details of the occluded area in the optical flow results and to fine-tune the details of the non-occluded area through a mask enhancement model. In addition, the application only calculates and maps the optical flow of the face part, greatly improving the processing performance of the algorithm, thereby realizing the purpose of stably and efficiently enhancing the quality of the face video. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0044] Figure 1 The steps of the method for face video quality enhancement of the present application are shown in the figure.

[0045] Figure 2 The structure of the device for face video quality enhancement of the present application is shown in the figure. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be described in detail below. Obviously, the described embodiments only constitute some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0047] The following will introduce a specific method, device and computer equipment for face video quality enhancement provided in the embodiments of the present application in combination with the drawings.

[0048] As shown in the figure, the method for face video quality enhancement provided in the embodiments of the present application comprises: Figure 1

[0049] S101, obtaining a video frame image of a video stream, identifying a face in a first frame image to obtain a first frame face frame, cropping the face in the first frame face frame to obtain a first frame face image to be enhanced;

[0050] In some embodiments, cropping the face in the first frame face frame to obtain a first frame face image to be enhanced comprises:

[0051] ​After recognizing the face in the first frame image, the face is scaled to a preset size.

[0052] The face of the preset size is taken as the first frame face image to be enhanced.

[0053] The present application can obtain a video stream by mobile terminal shooting or obtain a video stream through the album of the mobile terminal, and then decodes the video stream to obtain video frame images. First, a face frame algorithm is used to recognize the face in the video frame images to obtain a first frame face frame of the first frame image. The face is cropped from the face frame, and then the face is scaled to a preset size to obtain a first frame face image to be enhanced.

[0054] S102, the first frame face image to be enhanced is processed to obtain a first frame enhanced face. The first frame enhanced face is placed in the corresponding position of the first frame face frame to obtain a first enhanced frame.

[0055] In the present application, a face enhancement model in the prior art is used to process the first frame face image to be enhanced to obtain a first frame enhanced face. Then, the first frame enhanced face is pasted back to the corresponding position of the first frame face frame as a first enhanced frame.

[0056] S103, when processing the subsequent frame images, a current frame face frame is obtained from the current frame image, and a current frame face image to be enhanced is obtained through the current frame face frame.

[0057] It should be noted that when processing the subsequent frame images, for example, when processing the second frame image, the first frame image is taken as the last frame image, and the second frame image is taken as the current frame image, that is, the frame image to be processed. That is, the first frame face frame is the last frame face frame, and the first frame face image to be enhanced is the last frame face image to be enhanced. Then, a face frame in the current frame image is obtained through a face frame algorithm to obtain a "current frame face frame". The face in the current frame face frame is cropped and scaled to a preset fixed size to obtain a "current frame face image to be enhanced".

[0058] S104, a first optical flow and a first optical flow occlusion area mask between the last frame face image to be enhanced and the current frame face image to be enhanced are calculated. The first optical flow is mapped to the last frame enhanced face to obtain a current frame optical flow enhanced face.

[0059] The first optical flow between the "last frame face image to be enhanced" and the "current frame face image to be enhanced" is calculated by using the optical flow method, and the "first optical flow occlusion area mask" is calculated. Then, the first optical flow is mapped on the "last frame enhanced face" to obtain the "current frame optical flow enhanced face".

[0060] It should be noted that the relationship between the "last frame to be enhanced face" and the "current frame to be enhanced face" is similar to the relationship between the "last frame enhanced face" and the "current frame enhanced face", so the same optical flow can be applied to the "last frame enhanced face".

[0061] S105, input the current frame optical flow enhanced face, the first optical flow occlusion area mask and the current frame to be enhanced face into the pre-constructed mask enhancement model, to obtain the current frame enhanced face;

[0062] In some embodiments, the mask enhancement model is constructed, comprising:

[0063] Extracting a first high-quality frame image and a second high-quality frame image in a high-definition video stream in succession;

[0064] Degradation processing is performed on the first high-quality frame image and the second high-quality frame image to obtain a first low-quality frame image and a second low-quality frame image;

[0065] Face recognition is performed on the first low-quality frame image and the second low-quality frame image respectively to obtain a first low-quality frame face frame and a second low-quality frame face frame, and face cropping is performed on the first low-quality frame image and the second low-quality frame image based on the first low-quality frame face frame and the second low-quality frame face frame to obtain a first low-quality frame face and a second low-quality frame face;

[0066] Face recognition is performed on the first high-quality frame image and the second high-quality frame image respectively to obtain a first high-quality frame face frame and a second high-quality frame face frame, and face cropping is performed on the first high-quality frame image and the second high-quality frame image based on the first high-quality frame face frame and the second high-quality frame face frame to obtain a first high-quality frame face and a second high-quality frame face;

[0067] A second optical flow and a second optical flow occlusion area mask between the first low-quality frame face and the second low-quality frame face are calculated by an optical flow algorithm;

[0068] The second optical flow is mapped to the first high-quality frame face to obtain a second frame optical flow enhanced face;

[0069] A second frame optical flow enhanced face, a second optical flow occlusion area mask and a second low-quality frame face are trained based on a target loss function to obtain a mask enhancement model; wherein the mask enhancement model is used to output a second frame enhanced face.

[0070] Wherein, the degradation processing on the first high-quality frame image and the second high-quality frame image comprises:

[0071] Respectively adding noise, blurring and format changing processing is performed on the first high-quality frame image and the second high-quality frame image.

[0072] In some embodiments, the loss function of the mask enhancement model is

[0073] L = l_pix(G(x) * m, gt * m) + 0.1 * l_pix(G(x) * (1. - m), gt * (1. - m))

[0074] wherein m is the normalized "second optical flow occlusion area mask"; x is the input of the mask enhancement model; G(x) is the output result of the mask enhancement model, i.e., the second frame optical flow enhanced face; gt is the target enhancement result of the mask enhancement model, i.e., the second high-quality frame face; l_pix(A, B) is a loss function for calculating the pixel-by-pixel difference between images A and B; l_pix(G(x) * m, gt * m) is a loss for supervising the enhancement result of the occluded part; and 0.1 * l_pix(G(x) * (1. - m), gt * (1. - m)) is a loss for supervising the enhancement result of the non-occluded part.

[0075] Specifically, the "current frame optical flow enhanced face", "optical flow occlusion area mask", and "current frame to be enhanced face, i.e., second low-quality frame face" are input into the "mask enhancement model", and the "mask enhancement model" can enhance the occluded area and also fine-tune the non-occluded area to some extent, and finally obtain the "current frame enhanced face, i.e., second frame enhanced face".

[0076] S106, after enhancing all video frame images in the video stream, combining them into an enhanced video.

[0077] Specifically, all video frame images in the video stream are enhanced by steps S103-S105, and then all enhanced frame images are combined into a video, i.e., an enhanced video.

[0078] The working principle of the method for face video quality enhancement is: a video frame image of a video stream is acquired, a face in a first frame image is recognized to obtain a first frame face frame, the face in the first frame face frame is cropped to obtain a first frame to-be-enhanced face image; the first frame to-be-enhanced face image is subjected to enhancement processing to obtain a first frame enhanced face, the first frame enhanced face is placed in a corresponding position in the first frame face frame to obtain a first enhanced frame; when processing a subsequent frame image, a current frame face frame is obtained according to a current frame image, and a current frame to-be-enhanced face image is obtained through the current frame face frame; a first optical flow and a first optical flow occlusion area mask between a last frame to-be-enhanced face image and the current frame to-be-enhanced face image are calculated, the first optical flow is mapped to a last frame enhanced face to obtain a current frame optical flow enhanced face; the current frame optical flow enhanced face, the first optical flow occlusion area mask and the current frame to-be-enhanced face image are input into a pre-constructed mask enhancement model to obtain a current frame enhanced face; after all the video frame images in the video stream are subjected to enhancement processing, the enhanced video is obtained.

[0079] The technical scheme provided in the application considers improving the definition of the face video while ensuring the continuity of the video result and processing efficiency, and the continuity of the face video enhancement result is ensured by using the optical flow. Secondly, the mask enhancement model and the corresponding training method are used to enhance the details of the occlusion area in the optical flow result and finely adjust the details of the non-occlusion area by using the low-resolution original frame as a guide. Meanwhile, the optical flow calculation and optical flow mapping are only performed on the face part, which greatly improves the processing performance of the algorithm, thereby achieving the purpose of stably and efficiently enhancing the quality of the face video.

[0080] In some embodiments, if the face in the current frame image is multiple, the face in the current frame image is matched with the face in the last enhanced frame, including:

[0081] The pixel distance value of each to-be-enhanced face image in the last frame and each to-be-enhanced face image in the current frame image is calculated.

[0082] All the distances are traversed from small to large, if the distance is less than a preset threshold value, and the to-be-enhanced face image of the previous frame corresponding to the distance and the to-be-enhanced face image of the current frame are not matched before, it is considered that the matching is successful; after the traversal of all the distances is completed, the to-be-enhanced face image of the current frame which is not matched is processed as a new face image.

[0083] Specifically, because multiple faces may appear in the video, the faces need to be matched.

[0084] The matching rule is specifically that a distance is calculated for each face in the previous frame and each face in the current frame. Specifically, the distance between each face in the previous frame and each face in the current frame is calculated, and all distances are traversed from small to large. If the distance is less than a preset threshold and the face in the previous frame corresponding to the distance and the face in the current frame have not been matched before, it is considered that the matching is successful. After the distance traversal is completed, the face in the current frame that is not matched is treated as a new face.

[0085] The distance is directly calculated according to the sum of absolute values of pixel differences.

[0086]

[0087] wherein n is the number of pixels in the face image, R1, G1 and B1 are respectively the red, green and blue color values of a certain pixel in one face in the previous frame image, and R2, G2 and B2 are respectively the red, green and blue color values of a pixel at a corresponding position in one face in the current frame image.

[0088] In some embodiments, if a new face appears in the current frame image,

[0089] identifying a new face in the current frame image to obtain a new face frame, cropping the face in the new face frame to obtain a to-be-enhanced face image;

[0090] performing enhancement processing on the to-be-enhanced face image to obtain a frame-enhanced face, and placing the frame-enhanced face in a corresponding position in the new face frame to obtain a new face-enhanced frame.

[0091] Specifically, when a new face appears in the current frame image in the video frame image, the processing is performed according to steps S101-S102, and then the subsequent processing is performed according to S103-S105.

[0092] As shown in Figure 2 The embodiment of the present application provides a device for face video quality enhancement, which comprises:

[0093] The acquisition module 201 is configured to acquire a video frame image of a video stream, identify a face in a first frame image to obtain a first frame face frame, and crop the face in the first frame face frame to obtain a first frame to-be-enhanced face image.

[0094] The first processing module 202 is configured to perform enhancement processing on the first frame to-be-enhanced face image to obtain a first frame enhanced face, and place the first frame enhanced face in a corresponding position in the first frame face frame to obtain a first enhanced frame.

[0095] The second processing module 203 is configured to, when processing subsequent frame images, obtain a current frame face frame according to a current frame image, and obtain a current frame face image to be enhanced through the current frame face frame.

[0096] The calculation module 204 is configured to calculate a first optical flow and a first optical flow occlusion area mask between a last frame face image to be enhanced and the current frame face image to be enhanced, and map the first optical flow to a last frame enhanced face to obtain a current frame optical flow enhanced face.

[0097] The output module 205 is configured to input the current frame optical flow enhanced face, the first optical flow occlusion area mask and the current frame face image to be enhanced into a pre-constructed mask enhancement model to obtain a current frame enhanced face.

[0098] The enhancement processing module 206 is configured to, after performing enhancement processing on all video frame images in the video stream, combine the video frame images into an enhanced video.

[0099] The working principle of the device for face video quality enhancement provided in the embodiments of the present application is that the acquisition module 201 acquires video frame images of a video stream, recognizes a face in a first frame image to obtain a first frame face frame, crops the face in the first frame face frame to obtain a first frame face image to be enhanced, the first processing module 202 performs enhancement processing on the first frame face image to be enhanced to obtain a first frame enhanced face, the first frame enhanced face is placed in a corresponding position in the first frame face frame to obtain a first enhanced frame, the second processing module 203, when processing subsequent frame images, obtains a current frame face frame according to a current frame image, and obtains a current frame face image to be enhanced through the current frame face frame, the calculation module 204 calculates a first optical flow and a first optical flow occlusion area mask between a last frame face image to be enhanced and the current frame face image to be enhanced, and maps the first optical flow to a last frame enhanced face to obtain a current frame optical flow enhanced face, the output module 205 inputs the current frame optical flow enhanced face, the first optical flow occlusion area mask and the current frame face image to be enhanced into a pre-constructed mask enhancement model to obtain a current frame enhanced face, and the enhancement processing module 206, after performing enhancement processing on all video frame images in the video stream, combines the video frame images into an enhanced video.

[0100] The present application provides a computer device, comprising: a memory and a processor, the memory stores a computer program, when the computer program is executed by the processor, the processor executes

[0101] Acquire video frame images of a video stream, recognize a face in a first frame image to obtain a first frame face frame, crop the face in the first frame face frame to obtain a first frame face image to be enhanced;

[0102] performing enhancement processing on the first frame face-to-be-enhanced image to obtain a first frame enhanced face, and placing the first frame enhanced face into a corresponding position in the first frame face frame to obtain a first enhanced frame;

[0103] When processing a subsequent frame image, a current frame face frame is obtained according to a current frame image, and a current frame face-to-be-enhanced image is obtained through the current frame face frame;

[0104] calculating a first optical flow and a first optical flow occlusion area mask between a last frame face-to-be-enhanced image and the current frame face-to-be-enhanced image, and mapping the first optical flow to a last frame enhanced face to obtain a current frame optical flow enhanced face;

[0105] inputting the current frame optical flow enhanced face, the first optical flow occlusion area mask, and the current frame face-to-be-enhanced image into a pre-constructed mask enhancement model to obtain a current frame enhanced face;

[0106] After performing enhancement processing on all video frame images in the video stream, the enhanced video is obtained.

[0107] It can be understood that the method embodiments provided in the foregoing and the device embodiments correspond to each other, and the specific content can be mutually referred to, which will not be described herein again.

[0108] Those skilled in the art should understand that embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program codes.

[0109] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system), and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one block or multiple blocks.

[0110] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0112] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for face video quality enhancement, characterized in that, The method comprises the following steps: obtaining a video frame image of a video stream, recognizing a face in a first frame image to obtain a first frame face frame, cropping the face in the first frame face frame to obtain a first frame face to be enhanced; performing enhancement processing on the first frame face to be enhanced to obtain a first frame enhanced face, and placing the first frame enhanced face in a corresponding position in the first frame face frame to obtain a first enhanced frame; when processing a subsequent frame image, obtaining a current frame face frame according to a current frame image, and obtaining a current frame face to be enhanced through the current frame face frame; calculating a first optical flow and a first optical flow occlusion area mask between a previous frame face to be enhanced and the current frame face to be enhanced, and mapping the first optical flow to a previous frame enhanced face to obtain a current frame optical flow enhanced face; inputting the current frame optical flow enhanced face, the first optical flow occlusion area mask and the current frame face to be enhanced into a pre-constructed mask enhancement model to obtain a current frame enhanced face; after performing enhancement processing on all video frame images in the video stream, combining the video frame images into an enhanced video; wherein the mask enhancement model is constructed by the following steps: extracting a first high-quality frame image and a second high-quality frame image in a high-definition video stream; performing degradation processing on the first high-quality frame image and the second high-quality frame image to obtain a first low-quality frame image and a second low-quality frame image; performing face recognition on the first low-quality frame image and the second low-quality frame image respectively to obtain a first low-quality frame face frame and a second low-quality frame face frame, and performing face cropping on the first low-quality frame image and the second low-quality frame image based on the first low-quality frame face frame and the second low-quality frame face frame to obtain a first low-quality frame face and a second low-quality frame face; performing face recognition on the first high-quality frame image and the second high-quality frame image respectively to obtain a first high-quality frame face frame and a second high-quality frame face frame, and performing face cropping on the first high-quality frame image and the second high-quality frame image based on the first high-quality frame face frame and the second high-quality frame face frame to obtain a first high-quality frame face and a second high-quality frame face; calculating a second optical flow and a second optical flow occlusion area mask between the first low-quality frame face and the second low-quality frame face through an optical flow algorithm; mapping the second optical flow to the first high-quality frame face to obtain a second frame optical flow enhanced face; training a deep neural network based on the second frame optical flow enhanced face, the second optical flow occlusion area mask and the second low-quality frame face according to a target loss function to obtain a mask enhancement model; wherein the mask enhancement model is used to output a second frame enhanced face.

2. The method of claim 1, wherein, If there are multiple faces in the current frame image, the faces in the current frame image are matched with the faces in the previous frame, comprising: calculating a pixel distance value of each face to be enhanced in the previous frame and each face to be enhanced in the current frame image; Traverse all distances from small to large, if the distance is less than the preset threshold, and the previous frame of the to-be-enhanced face image and the current frame of the to-be-enhanced face image are not matched before, it is considered that the matching is successful; after all distances are traversed, the current frame of the to-be-enhanced face image that is not matched is taken as a new face image for processing.

3. The method of claim 2, wherein, If a new face appears in the current frame image, a new face in the current frame image is recognized to obtain a new face frame, and a face in the new face frame is cropped to obtain a to-be-enhanced face image; The to-be-enhanced face image is subjected to enhancement processing to obtain a frame enhanced face, and the frame enhanced face is placed in a corresponding position in the new face frame to obtain a new face enhanced frame.

4. The method of claim 1, wherein, The first high-quality frame image and the second high-quality frame image are subjected to degradation processing, including: The first high-quality frame image and the second high-quality frame image are respectively subjected to noise adding, blurring and format changing processing.

5. The method of claim 1, wherein, The loss function of the mask enhancement model is: L = l_pix(G(x) * m, gt * m) + 0.1*l_pix(G(x) * (1.-m), gt * (1.-m)) Wherein, m is the normalized "second optical flow occlusion area mask"; x is the input of the mask enhancement model; G(x) is the output result of the mask enhancement model, that is, the second frame optical flow enhanced face, and gt is the target enhancement result of the mask enhancement model, that is, the second high-quality frame face; l_pix(A, B) is a loss function for calculating the pixel-by-pixel difference between images A and B; l_pix(G(x) * m, gt * m) is a loss for supervising the enhancement result of the occluded part; 0.1*l_pix(G(x) * (1.-m), gt *(1.-m)) is a loss for supervising the enhancement result of the non-occluded part.

6. The method of claim 1, wherein, The face in the first frame face frame is cropped to obtain a first frame to-be-enhanced face image, including: After recognizing the face in the first frame image, the face is scaled to a preset size; The face of the preset size is taken as a first frame to-be-enhanced face image.

7. An apparatus for face video quality enhancement, the apparatus comprising: Including: An acquisition module is configured to acquire a video frame image of a video stream, recognize a face in a first frame image, obtain a first frame face frame, and crop the face in the first frame face frame to obtain a first frame to-be-enhanced face image; A first processing module is configured to perform enhancement processing on the first frame to-be-enhanced face image to obtain a first frame enhanced face, and place the first frame enhanced face in a corresponding position in the first frame face frame to obtain a first enhanced frame; A second processing module is configured to, when processing a subsequent frame image, obtain a current frame face frame according to a current frame image, and obtain a current frame to-be-enhanced face image through the current frame face frame; A calculation module is configured to calculate a first optical flow and a first optical flow occlusion area mask between a previous frame to-be-enhanced face image and a current frame to-be-enhanced face image, map the first optical flow to a previous frame enhanced face to obtain a current frame optical flow enhanced face; an output module configured to input the current frame optical flow enhanced face, the first optical flow occlusion area mask, and the current frame face image to be enhanced into a pre-constructed mask enhancement model to obtain a current frame enhanced face; constructing the mask enhancement model comprises: extracting a first high-quality frame image and a second high-quality frame image in a high-definition video stream; performing degradation processing on the first high-quality frame image and the second high-quality frame image to obtain a first low-quality frame image and a second low-quality frame image; performing face recognition on the first low-quality frame image and the second low-quality frame image respectively to obtain a first low-quality frame face frame and a second low-quality frame face frame, and performing face cropping on the first low-quality frame image and the second low-quality frame image based on the first low-quality frame face frame and the second low-quality frame face frame to obtain a first low-quality frame face and a second low-quality frame face; performing face recognition on the first high-quality frame image and the second high-quality frame image respectively to obtain a first high-quality frame face frame and a second high-quality frame face frame, and performing face cropping on the first high-quality frame image and the second high-quality frame image based on the first high-quality frame face frame and the second high-quality frame face frame to obtain a first high-quality frame face and a second high-quality frame face; calculating a second optical flow and a second optical flow occlusion area mask between the first low-quality frame face and the second low-quality frame face through an optical flow algorithm; mapping the second optical flow to the first high-quality frame face to obtain a second frame optical flow enhanced face; training a deep neural network based on a target loss function with the second frame optical flow enhanced face, the second optical flow occlusion area mask, and the second low-quality frame face to obtain a mask enhancement model; wherein the mask enhancement model is configured to output a second frame enhanced face; an enhancement processing module configured to perform enhancement processing on all video frame images in the video stream and combine the video frame images into an enhanced video.

8. A computer device, comprising: comprise: a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to execute the method for face video quality enhancement according to any one of claims 1 to 6.

Images