Method, device, medium and electronic device for evaluating face blur

Abstract

The present disclosure provides a method for evaluating the blurriness of a face, an evaluation device for the blurriness of a face, a computer-readable medium, and an electronic device, and relates to the field of image processing technology. The method comprises: extracting a target video frame containing a face in video data, and obtaining a reference video frame corresponding to the target video frame in the video data; calculating the facial posture features, facial similarity features, and facial gradient features corresponding to each face contained in the target video frame based on the target video frame and the reference video frame; fusing the facial posture features, facial similarity features, and facial gradient features corresponding to each face to obtain a fusion feature, and determining the facial blurriness corresponding to each face based on the fusion feature corresponding to each face. The present disclosure can simultaneously consider the blurriness of the facial region caused by motion or blurred image quality through multiple-frame images and multiple-frame image semantic information, thereby obtaining a more accurate facial blurriness.

Description

Background Art

[0002] As a basic and important task of image classification and understanding, image scene understanding has attracted great research interest in recent years and has been widely used in various fields. In addition to high-level semantics, there is also a demand for scene understanding in aspects related to image quality. For example, in the task of image deblurring, which is related to image quality.

[0003] Image deblurring is a task related to image quality, and the causes are complex. For example, due to exposure time and other reasons, the collected images often appear blurred. Figure 1 The figure shows face images with different blur levels. In this case, it is necessary to first determine the blur level of the image in order to select different deblurring algorithms for deblurring according to the blur level of the image. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method for evaluating the blurriness of a face, an apparatus for evaluating the blurriness of a face, a computer-readable medium and an electronic device, thereby improving the accuracy of blurriness estimation at least to a certain extent.

[0005] According to a first aspect of the present disclosure, a method for evaluating face blurriness is provided, comprising: extracting a target video frame containing a face in video data, and obtaining a reference video frame corresponding to the target video frame in the video data; calculating face posture features, face similarity features, and face gradient features corresponding to each face contained in the target video frame based on the target video frame and the reference video frame; respectively fusing the face posture features, face similarity features, and face gradient features corresponding to each face to obtain a fused feature, and determining the face blurriness corresponding to each face based on the fused feature corresponding to each face.

[0006] According to a second aspect of the present disclosure, a device for evaluating the blurriness of a face is provided, comprising: a video frame acquisition module, used to extract a target video frame containing a face in video data, and to acquire a reference video frame corresponding to the target video frame in the video data; a feature calculation module, used to calculate the facial posture features, facial similarity features and facial gradient features corresponding to each face contained in the target video frame based on the target video frame and the reference video frame; a blurriness evaluation module, used to fuse the facial posture features, facial similarity features and facial gradient features corresponding to each face to obtain a fused feature, and determine the facial blurriness corresponding to each face based on the fused feature corresponding to each face.

[0007] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0009] An embodiment of the present disclosure provides a method for evaluating the blurriness of a face, which obtains a target video frame in video data and performs face detection on the target video frame. Then, when the target video frame includes at least one face, obtain a reference video frame corresponding to the target video frame containing the obtained face, and then calculate the face posture features, face similarity features and face gradient features corresponding to each face based on multiple video frames, and then determine the fusion features of each face based on the face posture features, face similarity features and face gradient features corresponding to each face, and determine the face blurriness corresponding to the face based on any feature. Through multiple frames of images and multi-frame image semantic information, the blurriness of the face area caused by motion or blurred image quality can be considered at the same time, thereby obtaining a more accurate face blurriness.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 Face images with different blur levels are shown;

[0013] Figure 2 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0014] Figure 3 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0015] Figure 4 A flow chart schematically shows a method for determining the degree of fuzziness by using a secondary fuzzy algorithm;

[0016] Figure 5 A flowchart schematically illustrates a method for evaluating the blurriness of a human face in an exemplary embodiment of the present disclosure;

[0017] Figure 6A schematic diagram of extracting a reference video frame in an exemplary embodiment of the present disclosure is schematically shown;

[0018] Figure 7 A flowchart schematically illustrates a regression model training method in an exemplary embodiment of the present disclosure;

[0019] Figure 8 A flowchart schematically illustrates another method for evaluating the blurriness of a human face in an exemplary embodiment of the present disclosure;

[0020] Fig. 9 The following is a schematic diagram schematically showing the composition of a device for evaluating the blurriness of a human face in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0022] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0023] Figure 2 A schematic diagram of a system architecture of an exemplary application environment in which a method and device for evaluating the blurriness of a human face according to an embodiment of the present disclosure can be applied is shown.

[0024] like Figure 2 As shown, the system architecture 200 may include one or more of terminal devices 201, 202, 203, a network 204, and a server 205. The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 201, 202, 203 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 2The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 205 may be a server cluster composed of multiple servers.

[0025] The face blur evaluation method provided in the embodiment of the present disclosure is generally executed by the terminal devices 201, 202, and 203, and accordingly, the face blur evaluation device is generally set in the terminal devices 201, 202, and 203. However, it is easy for those skilled in the art to understand that the face blur evaluation method provided in the embodiment of the present disclosure can also be executed by the server 205, and accordingly, the face blur evaluation device can also be set in the server 205, which is not particularly limited in this exemplary embodiment. For example, in an exemplary embodiment, a user may collect video data for preview in real time through a camera module for collecting images included in terminal devices 201, 202, and 203, and then determine the video frame corresponding to the moment when the user presses the camera button as the target video frame, and obtain a reference video frame corresponding to the target video frame in the video data. Then, the target video frame and the reference video frame corresponding to the target video frame are sent to server 205 through network 204, and then the facial posture features, facial similarity features, and facial gradient features corresponding to each face are calculated by server 205, and feature fusion is performed to obtain the fusion features corresponding to each face. Then, the facial blurriness corresponding to each face is calculated based on the fusion features, and the facial blurriness is returned to the terminal devices 201, 202, 203, etc.

[0026] An exemplary embodiment of the present disclosure provides an electronic device for implementing a method for evaluating the blurriness of a human face, which may be Figure 2 The terminal device 201, 202, 203 or the server 205 in the embodiment of the present invention comprises at least a processor and a memory, wherein the memory is used to store executable instructions of the processor, and the processor is configured to execute the method for evaluating the blurriness of a human face by executing the executable instructions.

[0027] Below Figure 3 Taking the mobile terminal 300 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 3 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 300. In other embodiments, the mobile terminal 300 may also adopt the same Figure 3Different interface connection methods, or a combination of multiple interface connection methods.

[0028] like Figure 3 As shown, the mobile terminal 300 may specifically include: a processor 310, an internal memory 321, an external memory interface 322, a Universal Serial Bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 371, a receiver 372, a microphone 373, an earphone interface 374, a sensor module 380, a display screen 390, a camera module 391, an indicator 392, a motor 393, a button 394, and a subscriber identification module (SIM) card interface 395, etc. The sensor module 380 may include a depth sensor 3801, a pressure sensor 3802, a gyroscope sensor 3803, etc.

[0029] The processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. In an exemplary embodiment, the steps of obtaining a target video frame included in the video data and obtaining a reference video frame corresponding to the target video frame in the video data can be implemented by the processor.

[0030] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of the mobile terminal 300, such as image recognition, face recognition, voice recognition, text understanding, etc. In an exemplary embodiment, NPU can be used to detect faces, extract face features, and determine the blurriness of faces based on features.

[0031] The mobile terminal 300 implements the display function through a GPU, a display screen 390, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 390 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs that execute program instructions to generate or change display information.

[0032] The mobile terminal 300 can realize the shooting function through the ISP, the camera module 391, the video codec, the GPU, the display screen 390 and the application processor. Among them, the ISP is used to process the data fed back by the camera module 391; the camera module 391 is used to capture static images or videos; the digital signal processor is used to process digital signals, and in addition to processing digital image signals, it can also process other digital signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 300 can also support one or more video codecs. In an exemplary embodiment, the process of obtaining video data can be realized through the ISP, the camera module 391, the video codec, the GPU, the display screen 390 and the application processor.

[0033] In related technologies, a single traditional feature is usually used for fuzzy judgment, which can be divided into the following categories:

[0034] The first is to judge by the feature of image gradient, such as Tenengrad gradient function, Laplacian gradient function, SMD (grayscale variance) function, SMD2 (grayscale variance product) function, Brenner gradient function, etc.

[0035] The second is to judge through the secondary blur algorithm. Specifically, if an image is already blurred, then blurring it again will not change the high-frequency component much; but if the original image is clear, blurring it once will change the high-frequency component greatly. For example, Figure 4 As shown, in step S410, a degraded image of the image can be obtained by performing a Gaussian blur process on the image to be evaluated; in step S420, the change of the adjacent pixel values ​​of the original image and the degraded image is compared, and the clarity value is determined according to the size of the change. The smaller the calculation result, the clearer the image, and vice versa. This processing method can be called a judgment method based on a secondary blur algorithm.

[0036] The third is to judge by gradient structural similarity. For example, the image I to be evaluated can be low-pass filtered to obtain the reference image Ir, and then the gradient information of images I and Ir can be extracted to obtain gradient images G and Gr, and then the N image blocks with the richest gradient information in the gradient image G and the corresponding first N blocks of the gradient image Gr are found, and finally the reference-free structural clarity of the image I is calculated based on the N blocks with the richest gradient information.

[0037] The fourth is to make judgments through machine learning methods. For example, using a Laplace filter to find edges in the input image, calculate the variance and the maximum value of the pixel values ​​of the filtered image. A high variance (and a high maximum value) indicates that the edge is clearly visible, that is, the image is clear, and a low variance indicates that the image is blurred.

[0038] However, the above-mentioned image blur detection and prevention has the following main disadvantages: due to the rich application scenarios of blur detection, it is difficult to determine whether the image is blurry. Specifically, it may be necessary to manually set different thresholds as blur boundaries to distinguish blurry and non-blurry according to different application scenarios. In this case, both blurry and non-blurry require manual setting of thresholds. If it is necessary to define each blur degree in the blurry area, it will also be subject to greater human subjective influence. Correspondingly, the selection of a deblurring algorithm also becomes very difficult.

[0039] In addition, there are some methods for blur judgment based on multiple frames of images. For example, the patent application with publication number CN1992813A first finds a reference image with the highest clarity during the focusing stage of the electronic photographic device, and compares the reference image with the captured image to make a judgment based on the difference in clarity between the comparison image and the reference image. If the difference is too large, the captured image is blurred. On the contrary, if the difference is very small, the captured image is clear. For another example, the patent application with publication number CN106296688A obtains a set of points with the same name between two images in an image set through feature point detection, and then obtains regions with the same name. The variance of the regions with the same name is calculated after Laplace convolution is performed on the regions with the same name, and the mean of the variance of all regions with the same name in the two images is taken as the characterization of the image set. The relationship between the blur levels of the two images is reflected in the form of a ratio, and the order of the blur levels of the image set is obtained.

[0040] However, of these two methods of judging multiple frames of images, one requires manual determination of a reference image, while the other can only rank the relative blurriness of images in an image set, and an accurate evaluation result cannot be obtained.

[0041] Based on one or more of the above problems, this exemplary embodiment provides a method for evaluating the blurriness of a face. The method for evaluating the blurriness of a face can be applied to the server 105, or to one or more of the terminal devices 201, 202, and 203, which is not particularly limited in this exemplary embodiment. Figure 5 As shown, the method for evaluating the blurriness of a face may include the following steps S510 to S530:

[0042] In step S510, a target video frame containing a face is extracted from the video data, and a reference video frame corresponding to the target video frame is obtained from the video data.

[0043] The target video frame may include at least one human face.

[0044] In an exemplary embodiment, when extracting a target video frame containing a human face from the video data, a video frame to be tested may be first extracted from the video data, and face detection may be performed on the video frame to be tested. When the video frame to be tested is detected, if at least one human face is detected in the video frame to be tested, the video frame to be tested may be determined as the target video frame.

[0045] In an exemplary embodiment, after the user turns on the camera of the terminal device, the camera can be used to collect video data in real time, and the preview screen can be displayed on the screen of the terminal device. When the user takes a picture (presses a shutter button or other shooting buttons) or performs a specific operation, the video frame corresponding to the moment in the video data is determined as the video frame to be tested. For example, assuming that during the camera shooting process, the video data corresponding to the preview screen is as follows Figure 6 As shown, if the user presses the shutter button at the 6th frame, it can be determined that the 6th frame is the video frame to be tested.

[0046] It should be noted that, in an exemplary embodiment, in addition to determining the video frame to be tested by the above-mentioned shooting or performing a specific operation, the video frame to be tested can also be determined by other methods, and the present disclosure does not specifically limit this. For example, any frame in the collected video data can be designated as the video frame to be tested, or a video frame that meets the requirements can be selected from the video data as the video frame to be tested according to user requirements.

[0047] In an exemplary embodiment, when obtaining a reference video frame corresponding to a target video frame in video data, the reference video frame may be extracted from the video data based on a preset time condition and a time point of the target video frame in the video data. Specifically, the preset time condition may include a time range corresponding to the reference video frame. Figure 6 As shown, when the time point at which the target video frame is the 6th frame in the video data, the preset time condition may include the 4 frames before the target video frame in the video data, that is, the 2nd to 5th frames are the reference video frames corresponding to the target video frame; for another example, the preset time condition may include the 3 frames before and after the target video frame in the video data, that is, the 3rd to 5th frames and the 7th to 9th frames are the reference video frames corresponding to the target video frame; for another example, the preset time condition may also include the 5 frames after the target video frame in the video data, that is, the 7th to 11th frames are the reference video frames corresponding to the target video frame.

[0048] It should be noted that the preset time condition can be set according to specific needs. Specifically, with the target video frame as a reference, reference video frames can be taken forward, backward, or both in the video data, and the number of reference video frames extracted can also be set as needed. In addition, since previous video frames can usually be used to predict future video frames, when obtaining reference video frames, the video frame before the target video frame is usually used as the reference video frame, so that the face blur corresponding to the face included in the target video frame can be more accurately evaluated.

[0049] In step S520, facial posture features, facial similarity features, and facial gradient features corresponding to each face included in the target video frame are calculated based on the target video frame and the reference video frame.

[0050] Among them, the facial posture feature may include a combination of one or more features for characterizing the facial posture in multiple frame images, for example, the facial posture feature may include the change in the facial posture of each frame image in the multiple frame images relative to the previous frame image; the facial similarity feature may include a feature for characterizing the facial similarity in the multiple frame images, for example, the facial similarity feature may include the average value of the similarity of the facial posture of each frame image in the multiple frame images relative to the previous frame image; the facial gradient feature may include a feature for characterizing the facial gradient in the multiple frame images, for example, the facial gradient feature may include the average value of the gradient of the facial posture of each frame image in the multiple frame images relative to the previous frame image.

[0051] In an exemplary embodiment, after the target video frame and the reference video frame are determined, the facial posture features corresponding to each face in the target video frame can be calculated based on the target video frame and the reference video frame. Specifically, the target video frame and the reference video frame are sorted according to their order in the video data to obtain a video frame sequence, and for each face, the head posture corresponding to the face in the target video frame and the reference video frame can be calculated respectively, and then the posture change amount of each video frame relative to the previous video frame is calculated respectively according to the order in the video frame sequence, and the facial posture features corresponding to each face are determined based on the posture change amount.

[0052] Among them, the head posture may include one or more combinations of the following feature data used to characterize the head posture in each video frame: pitch angle, yaw angle, roll angle, two-dimensional coordinates of facial key points in the target video frame and the reference video frame, etc.

[0053] It should be noted that when the number of reference video frames is greater than 1, that is, when multiple video frames are extracted as reference video frames when extracting reference video frames, when determining facial posture features based on posture variation, the posture variation corresponding to the multiple video frames can be directly reduced in dimension to obtain facial posture features. For example, a dimensionality reduction algorithm such as PCA (Principal Component Analysis) can be used to reduce the obtained posture variation in dimension to obtain facial posture features.

[0054] In an exemplary embodiment, after the target video frame and the reference video frame are determined, the facial similarity features corresponding to each face can be calculated based on the target video frame and the reference video frame. Specifically, for each face, the target video frame and the reference video frame can be sorted according to their order in the video data to obtain a video frame sequence, and then the similarity values ​​of the image area corresponding to the face in each video frame relative to the image area corresponding to the face in the previous video frame are calculated according to the order in the video frame sequence, and the facial similarity features are determined based on the similarity values.

[0055] In an exemplary embodiment, after obtaining a video frame sequence, a histogram comparison method can be directly used to first obtain the histogram data of an image area corresponding to a certain face in a video frame and an image area corresponding to the face in a previous video frame, and then compare the histogram data to obtain a similarity value corresponding to the face. In addition, before comparing the histogram data, the video frame and the previous video frame can be normalized for ease of comparison. It should be noted that when calculating the similarity value, other image similarity calculation methods can also be used for calculation.

[0056] In an exemplary embodiment, when determining the facial similarity feature based on the similarity value, for each face, an average value may be calculated for all similarity values ​​calculated based on the target video frame and the reference video frame, and the calculated average value may be determined as the facial similarity feature corresponding to the face. It should be noted that, when determining the facial similarity feature, in addition to taking the average value, other methods may also be used to determine the facial similarity feature.

[0057] In an exemplary embodiment, after the target video frame and the reference video frame are determined, the face gradient features corresponding to each face can be calculated based on the target video frame and the reference video frame. Specifically, for each face, the target video frame and the reference video frame can be sorted according to their order in the video data to obtain a video frame sequence, and then the gradient difference between the image area corresponding to a face in each video frame and the image area corresponding to the face in the previous video frame is calculated according to the order in the video frame sequence, and the face similarity feature is determined based on the gradient difference.

[0058] Among them, when calculating the gradient, you can choose Tenengrad gradient function, Laplacian gradient function, SMD (grayscale variance) function, SMD2 (grayscale variance product) function, Brenner gradient function and other gradient functions, and the present disclosure does not make any special restrictions on this.

[0059] In an exemplary embodiment, when determining the face gradient feature based on the gradient difference, for each face, an average value may be calculated for all the gradient differences calculated based on the target video frame and the reference video frame, and the calculated average value may be determined as the face gradient feature corresponding to the face. It should be noted that, when determining the face gradient feature, in addition to taking the average value, other methods may also be used to determine the face gradient feature.

[0060] In step S530, the facial posture features, facial similarity features and facial gradient features corresponding to each face are fused to obtain fused features, and the facial blurriness corresponding to each face is determined based on the fused features corresponding to each face.

[0061] In an exemplary embodiment, after obtaining the face posture feature, the face similarity feature and the face gradient feature, the above three features can be fused and spliced ​​together to obtain a fused feature. It should be noted that when splicing, the three features can be spliced ​​in any order, and the present disclosure does not make any special limitation on this.

[0062] In an exemplary embodiment, when determining the facial blurriness corresponding to each face based on the fusion features corresponding to the face, the fusion features corresponding to each face can be regressed and predicted according to a preset regression algorithm to obtain the facial blurriness corresponding to each face. The preset regression algorithm may include various types of regression algorithms, such as support vector machine regression, linear regression, logistic regression, etc., and the present disclosure does not limit the features thereof.

[0063] Wherein, the preset regression algorithm may include a preset regression model. Correspondingly, before the fusion feature is regressed and predicted by the preset regression algorithm, the sample video frame and the reference video frame corresponding to the sample video frame may be obtained first, and the face blurriness corresponding to the sample video frame may be marked, and then the regression model may be trained based on the sample video frame, the reference video frame corresponding to the sample video frame and the face blurriness corresponding to the sample video frame. Wherein, the regression model may include regression models such as support vector machine and fully connected neural network, and the present disclosure does not make any special limitation on this. Wherein, the sample video frame and the reference video frame corresponding to the sample video frame are pre-collected training samples for model training, and the face blurriness corresponding to the sample video frame is the pre-marking of the face blurriness of the sample video frame. The supervised learning of the regression model can be achieved through the training samples and the marking.

[0064] Specifically, refer to Figure 7 As shown, step S710, based on the sample video frame and the reference video frame corresponding to the sample video frame, the above-mentioned facial posture features, facial similarity features and facial gradient features are calculated to obtain the facial posture features, facial similarity features and facial gradient features; step S720, the obtained facial posture features, facial similarity features and facial gradient features are fused to obtain fused features; step S730, the obtained fused features are input into the regression model, and the weight parameters in the regression model are updated through back propagation through the face blurriness and loss function corresponding to the marked sample video frame, so as to obtain the trained regression model, that is, the preset regression model.

[0065] The following example takes the target video frame as containing only one face and takes the previous 4 frames as reference video frames. Figure 8 , the technical solution of the embodiment of the present disclosure is elaborated in detail:

[0066] In step S801, after determining that the sixth frame in the video data is a video frame to be detected, face detection is performed on the sixth frame.

[0067] In step S803, it is determined whether there is a face in the sixth frame.

[0068] In step S805, when the sixth frame includes at least one human face, the sixth frame may be determined as the target video frame.

[0069] In step S807, after the target video frame is determined, 4 frames forward may be taken as reference video frames according to a preset time condition, that is, frames 2 to 5 are taken as reference video frames.

[0070] It is assumed that the user's head posture includes pitch, yaw, roll, and the two-dimensional coordinates of facial key points in the target video frame and the reference video frame.

[0071] In step S809, the target video frame and the reference video frame are sorted to obtain a video frame sequence.

[0072] Specifically, assuming that the target video frame is the 6th frame in the video data, and the reference video frame is obtained by taking 4 frames forward in the video data, that is, the 2nd to 5th frames in the video data, then the obtained video frame sequence is: the 2nd frame, the 3rd frame, the 4th frame, the 5th frame, and the 6th frame.

[0073] In step S811, the head posture is calculated.

[0074] For the target video frame or the reference video frame, first extract the 68 two-dimensional facial key points included in the video frame, calculate the rotation vector based on the two-dimensional facial key points, and calculate the rotation matrix corresponding to the rotation vector, and then obtain the Euler angles such as pitch angle, yaw angle and roll angle. Based on the above process, the head posture of each face can be obtained as shown in Formula 1:

[0075] Face={Pitch,Yaw,Roll,Keyp} Formula 1

[0076] Among them, Pitch, Yaw, and Roll represent the pitch angle, yaw angle, and roll angle of the head corresponding to the face, respectively, and Keyp represents the two-dimensional coordinates of 68 facial key points.

[0077] After obtaining the head postures corresponding to all target video frames and reference video frames, the head postures of all video frames can be expressed as the following formula 2 according to the order of the above video frame sequence:

[0078] Face = {Face 2 ,Face 3 ,Face 4 ,Face 5 ,Face 6} Formula 2

[0079] Among them, Face 2 ,Face 3 ,Face 4 ,Face 5 ,Face 6 Represent the head postures in the 2nd to 6th video frames respectively.

[0080] In step S813, the posture change between every two frames is calculated according to the order of the video frame sequence.

[0081] Specifically, the Face set in Formula 2 can be calculated based on the following Formula 3:

[0082]

[0083] Among them, F-motion i It represents the change in the head posture of the face in the ith video frame in the video frame sequence relative to the head posture in the previous video frame; t represents time; mean() represents the calculation of the average value.

[0084] In step S815, PCA dimensionality reduction is performed on the posture change amount.

[0085] It should be noted that since there are multiple facial key point coordinates, the average value of the facial key point coordinates can be calculated to simplify the calculation process. When there are 5 video frames involved in the calculation, the above formula can obtain 4 groups of one-dimensional vectors, and a corresponding 4x4 matrix vector F_motion can be obtained. In order to simplify the calculation process, the 4x4 matrix vector can be reduced in dimension based on PCA dimensionality reduction to obtain a one-dimensional vector, which can be expressed by the following formula 4:

[0086] Fm_simi=PCA(F_motion) Formula 4

[0087] Among them, Fm_simi represents the facial posture features after dimensionality reduction, F_motion represents the facial posture features before dimensionality reduction, and PCA represents the PCA dimensionality reduction algorithm.

[0088] In step S817, similarity calculation is performed based on the video frame sequence determined in step S809, and facial similarity features are determined.

[0089] Specifically, it can be calculated by the following formula 5 and formula 6:

[0090] hist_simi i =Histogram(face i ,face i-1 ) Formula 5

[0091] H_simi=Mean(hist_simi) Formula 6

[0092] Among them, hist_simi i Indicates the similarity value of the image area corresponding to the face in the i-th video frame in the video frame sequence relative to the image area corresponding to the face in the previous video frame; Histogram (face i ,face i-1 ) indicates a histogram comparison between the image area corresponding to the face in the i-th video frame and the image area corresponding to the face in the previous video frame (i-1); Mean() indicates the calculation of the average value; H_simi indicates the face similarity feature.

[0093] In step S819, based on the video frame sequence determined in step S809, gradient calculation is performed to determine the face gradient features.

[0094] Specifically, it can be calculated by the following formula 7 and formula 8:

[0095] grad_simi i =Grad(face i ,face i-1 ) Formula 7

[0096] G_simi=Mean(grad_simi) Formula 8

[0097] Among them, grad_simi i It represents the gradient difference between the image area corresponding to the face in the i-th video frame and the image area corresponding to the face in the previous video frame in the video frame sequence; Grad(face i ,face i-1 ) means calculating the difference between the gradient of the image area corresponding to the face in the i-th video frame and the gradient of the image area corresponding to the face in the previous video frame (i-1); Mean() means calculating the average value; G_simi represents the face gradient feature.

[0098] In step S821, feature fusion is performed on the face posture features, face similarity features and face gradient features.

[0099] After obtaining the above-mentioned face posture features, face similarity features and face gradient features, feature splicing can be performed based on the following formula 9 to obtain the fusion features corresponding to the face:

[0100] Feature={Fm_simi,H_simi,G_simi} Formula 9

[0101] Among them, Fm_simi represents the face posture feature after dimensionality reduction, H_simi represents the face similarity feature, and G_simi represents the face gradient feature.

[0102] In step S823, regression prediction is performed based on the fusion features of the face to obtain the face blurriness corresponding to the face.

[0103] After obtaining the above fusion features, the fusion features are input into a preset support vector machine model trained in advance for regression prediction to obtain a regression prediction value, namely, the blurriness of the face.

[0104] It should be noted that the preset support vector machine can be trained by a support vector machine through sample video frames, reference video frames corresponding to the sample video frames, and the face blurriness corresponding to the sample video frames to obtain the above-mentioned preset vector machine model. Wherein, during the training, the face posture features, face similarity features and face gradient features can be calculated based on the sample video frames and the reference video frames, and then the three features are spliced, and then the parameters in the support vector machine are updated according to the fusion features and the face blurriness corresponding to the sample video frames to obtain the preset support vector machine.

[0105] In summary, in this exemplary embodiment, the blur calculation of motion and image quality of the face is performed through multiple frames of images and semantic information of multiple frames of images, thereby improving the accuracy of the face blur evaluation result. At the same time, the traditional solution can only distinguish motion blur or blur caused by poor image quality, but the face has both blur caused by rigid motion and blur related to image quality. Therefore, it is proposed to evaluate the blur of the face by fusing the three features.

[0106] In addition, it is difficult to achieve sufficient accuracy for traditional features, single deep learning, and solutions that combine traditional features and deep learning. They can only identify clear and severely blurred images. Moreover, deep learning solutions are difficult to deploy on mobile terminals. Regression prediction of features based on machine learning not only improves the accuracy of face blur evaluation, but is also easy to deploy in mobile terminals, providing more accurate blur semantic information for portrait photography scenes, and assisting subsequent image quality algorithms to achieve optimal results.

[0107] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0108] For further reference, Fig. 9 As shown, in the embodiment of this example, a face blur evaluation device 900 is also provided, including a video frame acquisition module 910, a feature calculation module 920 and a blur evaluation module 930. Among them:

[0109] The video frame acquisition module 910 can be used to extract a target video frame containing a face from the video data, and acquire a reference video frame corresponding to the target video frame from the video data.

[0110] The feature calculation module 920 may be configured to calculate facial posture features, facial similarity features, and facial gradient features corresponding to each face included in the target video frame based on the target video frame and the reference video frame.

[0111] The blur evaluation module 930 can be used to fuse the facial posture features, facial similarity features and facial gradient features corresponding to each face to obtain a fused feature, and determine the facial blur corresponding to each face based on the fused feature corresponding to each face.

[0112] In an exemplary embodiment, the feature calculation module 920 can be used to calculate the head posture in the target video frame and the reference video frame for each face; sort the target video frame and the reference video frame based on their order in the video data to obtain a video frame sequence; in the video frame sequence, for each face, calculate the posture change of each video frame relative to the previous video frame based on the head posture corresponding to each video frame; and determine the facial posture feature corresponding to each face based on the posture change.

[0113] In an exemplary embodiment, the feature calculation module 920 may be used to perform dimensionality reduction processing on the posture variation to obtain facial posture features.

[0114] In an exemplary embodiment, the feature calculation module 920 can be used to sort the target video frames and the reference frames based on their order in the video data to obtain a video frame sequence; in the video frame sequence, for each face, calculate the similarity value of the image area corresponding to the face in each video frame relative to the image area corresponding to the face in the previous video frame; and determine the facial similarity features corresponding to each face based on the similarity values.

[0115] In an exemplary embodiment, the feature calculation module 920 can be used to sort the target video frames and the reference frames based on their order in the video data to obtain a video frame sequence; in the video frame sequence, for each face, calculate the gradient difference between the image area corresponding to the face in each video frame and the image area corresponding to the face in the previous video frame; and determine the face gradient feature based on the gradient difference.

[0116] In an exemplary embodiment, the blur evaluation module 930 may be used to perform regression prediction on the fusion features corresponding to each face using a preset regression algorithm to obtain the facial blur corresponding to each face.

[0117] In an exemplary embodiment, the blur evaluation module 930 may be used to train a regression model based on a sample video frame, a reference video frame corresponding to the sample video frame, and a face blur corresponding to the sample frame to obtain a preset regression model.

[0118] In an exemplary embodiment, the video frame acquisition module 910 may be used to extract a reference video frame corresponding to the target video frame in the video data based on a preset time condition and a time point of the target video frame in the video data.

[0119] In an exemplary embodiment, the video frame acquisition module 910 can be used to extract a video frame to be tested from the video data and perform face detection on the video frame to be tested; when the video frame to be tested includes at least one face, the video frame to be tested is determined as a target video frame.

[0120] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0121] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0122] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figure 5 , Figure 7 and Figure 8 Any one or more steps in .

[0123] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0124] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0125] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0126] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0127] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for evaluating the blurriness of a face, characterized in that: include: Extracting a target video frame containing a human face from the video data, and obtaining a reference video frame corresponding to the target video frame from the video data, wherein the number of the reference video frames is greater than 1; Calculate the facial posture features, facial similarity features and facial gradient features corresponding to each of the faces contained in the target video frame based on the target video frame and the reference video frame; The facial posture features, facial similarity features and facial gradient features corresponding to each of the faces are respectively fused to obtain fused features, and the facial blurriness corresponding to each of the faces is determined based on the fused features corresponding to each of the faces; The obtaining a reference video frame corresponding to the target video frame in the video data includes: Based on a preset time condition and a time point of the target video frame in the video data, the reference video frame is retrieved forward, backward, or both forward and backward in the video data with the target video frame as a reference.

2. The method according to claim 1, characterized in that Calculating the facial posture feature corresponding to the face based on the target video frame and the reference video frame includes: Sort the target video frame and the reference video frame based on the order of the target video frame and the reference video frame in the video data to obtain a video frame sequence; For each of the human faces, respectively calculating the head posture in the target video frame and the reference video frame; In the video frame sequence, for each face, based on the head posture corresponding to each video frame, respectively calculate the posture change amount of each video frame relative to the previous video frame; The facial posture features corresponding to each of the faces are determined based on the posture change amount.

3. The method according to claim 2, characterized in that The determining of the facial posture features corresponding to each of the faces based on the posture change amount includes: The posture variation is subjected to dimensionality reduction processing to obtain facial posture features.

4. The method according to claim 1, characterized in that Calculating the face similarity feature based on the target video frame and the reference video frame includes: Sort the target video frame and the reference video frame based on the order of the target video frame and the reference video frame in the video data to obtain a video frame sequence; In the video frame sequence, for each of the human faces, respectively calculating a similarity value between an image region corresponding to the human face in each video frame and an image region corresponding to the human face in a previous video frame; The facial similarity features corresponding to the faces are determined based on the similarity values.

5. The method according to claim 1, characterized in that Calculating the face gradient feature based on the target video frame and the reference video frame includes: Sort the target video frame and the reference video frame based on the order of the target video frame and the reference video frame in the video data to obtain a video frame sequence; In the video frame sequence, for each of the human faces, respectively calculating a gradient difference between an image region corresponding to the human face in each video frame and an image region corresponding to the human face in a previous video frame; The face gradient feature is determined based on the gradient difference.

6. The method according to claim 1, characterized in that The determining the facial blurriness corresponding to each of the faces based on the fusion features corresponding to each of the faces comprises: The fusion features corresponding to each of the faces are regressed and predicted respectively through a preset regression algorithm to obtain the facial blurriness corresponding to each of the faces.

7. The method according to claim 6, characterized in that The preset regression algorithm includes a preset regression model. Before performing regression prediction on the fusion feature by using the preset regression algorithm, the method further includes: The regression model is trained based on the sample video frame, the reference video frame corresponding to the sample video frame and the face blurriness corresponding to the sample video frame to obtain a preset regression model.

8. The method according to claim 1, characterized in that The step of extracting a target video frame containing a face from the video data comprises: Extracting a video frame to be tested from the video data, and performing face detection on the video frame to be tested; When the video frame to be tested includes at least one human face, the video frame to be tested is determined as a target video frame.

9. A device for evaluating the blurriness of a face, characterized in that: include: A video frame acquisition module, used to extract a target video frame containing a face from the video data, and obtain a reference video frame corresponding to the target video frame from the video data, wherein the number of the reference video frames is greater than 1; A feature calculation module, used for calculating the face posture feature, face similarity feature and face gradient feature corresponding to each face contained in the target video frame based on the target video frame and the reference video frame; A fuzziness evaluation module is used to fuse the facial posture features, facial similarity features and facial gradient features corresponding to each of the faces to obtain fused features, and determine the facial fuzziness corresponding to each of the faces based on the fused features corresponding to each of the faces; The video frame acquisition module is specifically used to obtain the reference video frame forward, backward, or both forward and backward in the video data based on preset time conditions and the time point of the target video frame in the video data, with the target video frame as a reference.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

Images