Video quality auditing method and device for multi-stage face image integrity detection

Abstract

The invention provides a video quality auditing method and device for multi-stage face image integrity detection. The method comprises the following steps: constructing a data set; training the initial TSCCN through a model training sample obtained based on the data set to obtain a target TSCCN, and constructing an auditing model based on the target TSCCN; inputting the key frame image sequence to a target TSCCN through an auditing model; based on any key frame image of the key frame image sequence, obtaining a first face region image based on the key frame image in a first stage, obtaining a first key point region image based on the first face region image in a second stage, and determining a state classification result and an integrity score based on the first key point region image in a third stage; and outputting a face quality auditing report based on the state classification result and the complete integrity score. According to the technical scheme of the embodiment of the invention, progressive detection is realized through the target TSCCN, the integrity detection precision is improved, and the face image availability in a complex working environment is effectively evaluated through the face quality auditing report.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video quality detection, in particular to a multi-stage face image integrity detection video quality auditing method and device. BACKGROUND

[0002] In industrial production, construction operation and other field environments, the video monitoring system needs to record the operation process of the operating personnel in real time to ensure safe production and standard operation. However, due to the particularity of the operating site environment, such as equipment shielding, insufficient light or backlight, etc., the face image in the video obtained through the video monitoring system is often incomplete, blurred or the key features of the face image are missing, which leads to the inability to perform subsequent standard checking and safety behavior analysis.

[0003] The current traditional method uses a single threshold to judge the face image, which can only determine whether there is a face in the video and is prone to face misjudgment or face omission, and cannot effectively evaluate the usability of the face image in the video obtained through the video monitoring system in a complex operating environment. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a multi-stage face image integrity detection video quality auditing method and device, which outputs a face quality auditing report of the key frame image in the video to be audited in three stages through a target TSCCN, which can improve the integrity detection accuracy and effectively evaluate the usability of the face image in a complex operating environment.

[0005] In a first aspect, the embodiments of the present application provide a multi-stage face image integrity detection video quality auditing method, comprising: Obtaining a plurality of face image samples, and constructing a data set based on all the face image samples; Constructing an initial three-stage cascaded convolutional neural network TSCCN, obtaining a plurality of model training samples based on the data set, training the initial TSCCN through all the model training samples to obtain a target three-stage cascaded convolutional neural network TSCCN, and constructing an auditing model based on the target TSCCN, wherein the target TSCCN includes a first stage, a second stage and a third stage; Reading a video to be audited through the auditing model, extracting a key frame image sequence based on the video to be audited, and inputting the key frame image sequence into the target TSCCN; In the first stage, a first face region image is cropped from any of the key frame images based on the sequence of key frame images; in the second stage, a first key point region image is obtained based on the first face region image; and in the third stage, a state classification result and a completeness score are determined based on the first key point region image, wherein the state classification result is used to represent a degree of occlusion of the first face region image, and the completeness score is used to represent a completeness degree of the first face region image. A completeness score sequence is generated based on all the completeness scores, and a face quality review report is output based on the state classification result and the completeness score sequence.

[0006] According to some embodiments of the present application, in the second stage, a first key point region image is obtained based on the first face region image, comprising: In the second stage, the first face region image is obtained, key point prediction is performed on the first face region image, and a first key point prediction result is obtained. The first key point region image is cropped based on the first face region image and the first key point prediction result, wherein the first key point region image comprises all first key points, the number of the first key points is greater than or equal to zero, and the first key points are facial feature points of the first face region image.

[0007] According to some embodiments of the present application, in the third stage, a state classification result and a completeness score are determined based on the first key point region image, comprising: In the third stage, a first key point missing rate is determined based on all the first key points and a first formula, wherein the expression of the first formula is , is the first key point missing rate, N 1total is a preset number of first key points, N 1miss is the preset number of first key points minus the number of first key points in the first key point region image. The state classification result is determined based on the first key point missing rate and a first preset threshold, and the completeness score is determined based on the state classification result and a second formula, wherein the expression of the second formula is , is a completeness score sequence.

[0008] According to some embodiments of the present application, a plurality of face image samples are obtained, and a data set is constructed based on all the face image samples, comprising: The second key point annotation result is obtained by key point annotation on the face image sample, the second key point missing rate is determined based on the second key point annotation result and a third formula, the second key point annotation result comprises a second key point whose number is greater than or equal to zero, and the third formula is expressed as , is the second key point missing rate, N 2total is a preset number of the second key points, N 2miss is the preset number of the second key points minus the number of the second key points in the face image sample; The face image samples are classified based on the second key point missing rate and a first preset threshold, and a data set is generated based on the classified face image samples.

[0009] According to some embodiments of the present application, a plurality of model training samples are obtained based on the data set, and a target TSCCN is obtained by training an initial TSCCN based on all the model training samples, comprising: The model training samples are obtained by constructing an orthogonal test scheme based on all the face image samples in the data set based on key factors, wherein the model training samples comprise at least one face image sample, and the key factors comprise an occlusion ratio factor, an illumination condition factor and a facial expression factor; Based on any face image sample in any model training sample, the initial TSCCN crops the face image sample to obtain a second face region image, performs key point prediction on the second face region image to obtain a third key point prediction result, and crops a second key point region image based on the second face region image according to the third key point prediction result, wherein the second key point region image comprises all third key points, and the number of the third key points is greater than or equal to zero; A final loss function is determined based on all the second key points and the second key point region image, and the target TSCCN is determined based on the initial TSCCN and the final loss function.

[0010] According to some embodiments of the present application, the final loss function is determined based on all the second key points and the second key point region image, comprising: The prediction state classification and the prediction completeness are determined based on all the third key points in the second key point region image, the actual state classification and the actual completeness are determined based on all the second key points; The classification loss function is determined based on the prediction state classification and the actual state classification, the regression loss function is determined based on the prediction completeness and the actual completeness, and the final loss function is determined based on the classification loss function and the regression loss function; wherein an expression of the classification loss function is an expression of the regression loss function is an expression of the final loss function is , is a weighting factor of the classification loss function, is a weighting factor of the regression loss function, is an actual state classification, is a predicted state classification, S is an actual completeness, is a predicted completeness.

[0011] According to some embodiments of the present application, the review model reads a video to be reviewed, extracts a key frame image sequence based on the video to be reviewed, comprising: The review model reads a video to be reviewed, obtains the total frame number and frame rate of the video to be reviewed; Based on the frame rate, interval sampling is performed based on the video to be reviewed to obtain a plurality of key frame images; All the key frame images are sorted in time sequence, and the key frame image sequence is generated based on the sorted key frame images.

[0012] In a second aspect, the embodiments of the present application provide a video quality review device for multi-stage face image completeness detection, comprising at least one control processor and a memory in communication connection with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the multi-stage face image completeness detection video quality review method as described in the first aspect above.

[0013] In a third aspect, the embodiments of the present application provide an electronic device comprising the multi-stage face image completeness detection video quality review device as described in the second aspect above.

[0014] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium storing computer executable instructions for executing the multi-stage face image completeness detection video quality review method as described in the first aspect above.

[0015] The multi-stage face image integrity detection video quality auditing method according to the embodiment of the present application has at least the following beneficial effects: a plurality of face image samples are obtained, a data set is constructed based on all the face image samples; an initial three-stage cascade convolutional neural network TSCCN is constructed, a plurality of model training samples are obtained based on the data set, the initial TSCCN is trained through all the model training samples to obtain a target three-stage cascade convolutional neural network TSCCN, and an auditing model is constructed based on the target TSCCN, wherein the target TSCCN includes a first stage, a second stage and a third stage; the auditing model is used to read a video to be audited, a key frame image sequence is extracted based on the video to be audited, and the key frame image sequence is input into the target TSCCN; based on any key frame image of the key frame image sequence, a first face region image is obtained by cropping the key frame image in the first stage, a first key point region image is obtained based on the first face region image in the second stage, and a state classification result and an integrity score are determined based on the first key point region image in the third stage, wherein the state classification result is used to represent the occlusion degree of the first face region image, and the integrity score is used to represent the integrity degree of the first face region image; an integrity score sequence is generated based on all the integrity scores, and a face quality auditing report is output based on the state classification result and the integrity score sequence. According to the technical scheme of the embodiment of the present application, the first face region image is obtained based on the key frame image in the first stage of the target TSCCN, the first key point region image is obtained based on the first face region image in the second stage of the target TSCCN, and the state classification result and the integrity score are determined based on the first key point region image in the third stage of the target TSCCN, so that progressive detection of the face region in the key frame is realized, thereby improving the face integrity detection precision in the video to be audited; and the face quality auditing report obtained based on the integrity score and the state classification result can determine the working performance of the target TSCCN in different working environments, thereby effectively evaluating the face image availability in a complex working environment. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of a multi-stage face image integrity detection video quality auditing method provided by an embodiment of the present application; Figure 2 is a structure diagram of a target three-stage cascade convolutional neural network TSCCN provided by another embodiment of the present application; Figure 3 is a structure diagram of a multi-stage face image integrity detection video quality auditing device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0017] Embodiments of the present application are described below in detail with reference to examples illustrated in the accompanying drawings, in which like or similar elements or components are designated with the same or similar reference numerals throughout the drawings. The embodiments described below through reference to the drawings are illustrative, and are merely intended to explain the present application, and are not to be understood as limiting the present application.

[0018] In the description of the present application, it is to be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the purpose of facilitating the description of the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0019] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is two or more. Greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0020] The multi-stage face image integrity detection video quality auditing method according to the embodiment of the present application has at least the following beneficial effects: a plurality of face image samples are obtained, a data set is constructed based on all the face image samples, an initial three-stage cascade convolutional neural network TSCCN is constructed, a plurality of model training samples are obtained based on the data set, the initial TSCCN is trained through all the model training samples to obtain a target three-stage cascade convolutional neural network TSCCN, and an auditing model is constructed based on the target TSCCN, wherein the target TSCCN includes a first stage, a second stage and a third stage; the auditing model is used to read a video to be audited, a key frame image sequence is extracted based on the video to be audited, and the key frame image sequence is input into the target TSCCN; based on any key frame image of the key frame image sequence, a first face region image is obtained by cropping the key frame image in the first stage, a first key point region image is obtained based on the first face region image in the second stage, and a state classification result and an integrity score are determined based on the first key point region image in the third stage, wherein the state classification result is used to represent the occlusion degree of the first face region image, and the integrity score is used to represent the integrity degree of the first face region image; an integrity score sequence is generated based on all the integrity scores, and a face quality auditing report is output based on the state classification result and the integrity score sequence. According to the technical scheme of the embodiment of the present application, the first face region image is obtained based on the key frame image in the first stage of the target TSCCN, the first key point region image is obtained based on the first face region image in the second stage of the target TSCCN, and the state classification result and the integrity score are determined based on the first key point region image in the third stage of the target TSCCN, so that progressive detection of the face region in the key frame is realized, thereby improving the face integrity detection precision in the video to be audited; and the face quality auditing report obtained based on the integrity score and the state classification result can determine the working performance of the target TSCCN in different working environments, thereby effectively evaluating the face image availability in a complex working environment.

[0021] The following drawings further illustrate the technical scheme of the embodiment of the present application.

[0022] Reference Figure 1 and Figure 2 , Figure 1 is a flowchart of a multi-stage face image integrity detection video quality auditing method provided by the embodiment of the present application, Figure 2 is a structure diagram of a target three-stage cascade convolutional neural network TSCCN provided by another embodiment of the present application, and the multi-stage face image integrity detection video quality auditing method includes but is not limited to the following steps: S10, a plurality of face image samples are obtained, and a data set is constructed based on all the face image samples; S20, an initial three-stage cascaded convolutional neural network TSCCN is constructed, a plurality of groups of model training samples are obtained based on a data set, the initial TSCCN is trained through all the model training samples to obtain a target three-stage cascaded convolutional neural network TSCCN, and an audit model is constructed based on the target TSCCN, wherein the TSCCN includes a first stage, a second stage and a third stage; S30, a video to be audited is read through the audit model, a key frame image sequence is extracted based on the video to be audited, and the key frame image sequence is input into the target TSCCN; S40, based on any key frame image of the key frame image sequence, a first face region image is obtained by cropping the key frame image in the first stage, a first key point region image is obtained based on the first face region image in the second stage, and a state classification result and a completeness score are determined based on the first key point region image in the third stage, wherein the state classification result is used to represent the occlusion degree of the first face region image, and the completeness score is used to represent the completeness degree of the first face region image; S50, a completeness score sequence is generated based on all the completeness scores, and a face quality audit report is output based on the state classification result and the completeness score sequence.

[0023] It should be noted that the face image samples are collected from different sources, including public face data, actual video monitoring segments, mobile terminal captured images, etc. The face image samples are from different sources to ensure the diversity of the face image samples, covering face image samples of different angles, different illuminations and different occlusion conditions.

[0024] It should be noted that the data set is constructed based on the face image samples, and is used to train the initial three-stage cascaded convolutional neural network (Three-Stage Cascaded Convolutional Networks, abbreviated as TSCCN) and evaluate the target three-stage cascaded convolutional neural network TSCCN obtained by training the initial TSCCN.

[0025] It should be noted that the initial TSCCN is an untrained three-stage cascaded convolutional neural network, the target TSCCN is obtained by training the initial TSCCN based on the initial TSCCN and all the model training samples, the initial TSCCN is trained and evaluated through the data set, so as to determine the final loss function, and the target TSCCN is determined based on the initial TSCCN and the final loss function.

[0026] It should be noted that a plurality of groups of model training samples are designed based on the data set, the model training samples provide a high-quality data basis for neural network training and classification, and enhance the model's adaptability to face states in complex scenes.

[0027] It should be noted that the target TSCCN can determine the face image positioning, perform the five feature points prediction and the fine-grained contour feature extraction from coarse to fine, so as to realize the face quality detection with layer-by-layer refinement, and improve the face quality detection precision and the video auditing precision. The target TSCCN is connected between the stages of networks in a feature concatenation manner, that is: wherein, is the feature map extracted by the i-th stage of the target TSCCN, is the convolution network operation of the i+1-th stage of the target TSCCN. That is, the first stage and the second stage are connected in a feature concatenation manner, the second stage is refined on the basis of the output of the first stage, and the second stage performs the convolution network operation based on the first face region image to obtain the first key point region image; the second stage and the third stage are connected in a feature concatenation manner, the third stage is refined on the basis of the output of the second stage, and the third stage obtains the state classification result and the completeness score based on the first key point region image.

[0028] It should be noted that, referring to Figure 2 , the first stage of the target TSCCN is composed of convolution layers (Convolutional layer) and max-pooling layers (MaxPool) in series, and the ReLU activation function is used after each convolution layer. After passing through all the convolution layers and all the max-pooling layers, the face region coordinates are output through the fully connected layer (Fully Connected Layer), and finally the key frame image is cropped according to the face region coordinates through the crop layer (Crop Layer). The key frame images in the key frame image sequence are sequentially input to the first stage, and the key frame images sequentially pass through the convolution layer 1_1, the max-pooling layer, the convolution layer 1_2, the max-pooling layer, the convolution layer 1_3 and the crop layer. After some key frame images pass through the convolution layer 1_3, they pass through the fully connected layer and then enter the crop layer, and the first face region image is output, wherein the convolution layer 1_1 is the first convolution layer of the first stage of the target TSCCN, the convolution layer 1_2 is the second convolution layer of the first stage of the target TSCCN, and the convolution layer 1_3 is the third convolution layer of the first stage of the target TSCCN.

[0029] It should be noted that, referring to Figure 2The second stage of the target TSCCN includes a medium complexity CNN, which is a High-Resolution Network (HRNet), and the second stage is implemented based on a light-weight variant of the HRNet. The HRNet is used to perform key point prediction on the first face region image, so as to determine whether the face in the first face region is occluded based on the result of the key point prediction. The first stage outputs the first face region image to the second stage, and the second stage obtains the first face region image. The first face region image passes through the HRNet layer, so as to complete the key point prediction. The first key point region image is obtained by cropping the first face region image based on the result of the key point prediction, and the first key point region image is a high-resolution image. The second stage outputs the key points, the key point coordinates and the confidence through the HRNet layer, and focuses on enhancing the robustness of the mouth, eyes and other easily-occluded regions of the face region.

[0030] It should be noted that, with reference to Figure 2 The third stage of the target TSCCN is composed of two convolutional blocks and fully connected layers. One convolutional block is composed of two convolutional layers and a max-pooling layer, and the two convolutional blocks are connected in series to extract features. A ReLU activation function is used after each convolutional layer. The last stage of the third stage uses two FCs connected in series to output a state classification result and a completeness score sequence. The third stage includes a deep CNN, which is implemented based on a MODNet framework. The first key point region image is output from the second stage to the third stage. The first key point region image passes through the convolutional layer 3_1, the convolutional layer 3_2 and the max-pooling layer of the first convolutional block, and the convolutional layer 3_1, the convolutional layer 3_2 and the max-pooling layer of the second convolutional block, and the fully connected layer 2 and the fully connected layer 3 in sequence, to output the state classification result and the completeness score sequence.

[0031] It should be noted that the audit model is an integrated face completeness intelligent audit system for a video audit scene, which is implemented based on Python+tkinter, and integrates video reading, key frame image extraction, calling of the target TSCCN for detection and structured output of a face key point detection report, etc. After the video frame image is input to the audit model, a face recognition state, a completeness score sequence and a face quality audit report are output, so as to realize automatic evaluation of the face quality of the key frame image of the video, and improve the efficiency and accuracy of intelligent video auditing in a complex working environment.

[0032] It should be noted that the video to be audited is a video stream obtained by a video acquisition device arranged in a complex working environment, and the video to be audited is a video with an unconfirmed face quality. The video to be audited includes a plurality of video frame images, a plurality of key frame images are extracted from all the video frame images, and a key frame image sequence is obtained based on the plurality of key frame images, wherein the key frame is a key image frame of a shot in the video, and the key frame can reflect the main content of a shot.

[0033] It should be noted that the key frame image includes a face region and an environment background of a complex working environment, and the first face region image is obtained by cropping based on the key frame image in the first stage, thereby reducing the size of the image required for subsequent processing, and the influence of irrelevant environment background can be reduced. If the key frame image includes multiple faces, multiple face region images are output, and the number of face region images is the same as the number of faces in the key frame image.

[0034] It should be noted that the first key point is obtained by performing key point prediction on the face region image, and the key point is a facial feature point such as an eye corner, a nose tip, and a mouth corner. In the second stage, the first key point region is obtained by cropping based on the first face region image, and in this step, only the first key point is retained, and the remaining face region image irrelevant to the state classification result and the completeness score is cropped.

[0035] It should be noted that the state classification result and the completeness score are determined based on the first key point region image in the third stage, the state classification result ∈ {0, 1, 2}, 0 represents no face, 1 represents a blocked face, and 2 represents a complete face; the completeness score ∈ [0, 1], the closer the completeness score is to 1, the more the number of first key points, that is, the lower the blocking degree of the face located in the key frame image, and the higher the completeness. The face quality audit report is generated based on the state classification result and the completeness score sequence, and is used to audit the face quality in the video, thereby auditing the face availability of the video.

[0036] It should be noted that the present application is aimed at the special needs of quality evaluation of face images in work site recorded videos, adopts a three-level key point missing rate grading mechanism, quantitatively evaluates the completeness of the facial features of the key frame image, avoids the problem of high false rejection rate caused by the traditional single threshold method, and effectively solves the evaluation problems caused by equipment shielding, light changes and other factors in the industrial environment. The three-level cascaded convolutional neural network realizes progressive detection from face region positioning to facial feature point prediction to fine-grained contour feature, and improves the detection accuracy compared with the traditional method by extracting features from the cropped image. The specially introduced face completeness scoring mechanism can automatically output quality scores, greatly improving the model generalization and ensuring its working performance in different working environments. The present scheme has significant technical advantages and practical value. Compared with the prior art, it solves the problems of poor adaptability and high false rejection rate of traditional methods, can be seamlessly integrated into existing detection systems, provides reliable technical support for safety production management, operation specification verification and other applications, and fills the technical gap in this field.

[0037] In addition, in an embodiment, in Figure 1 In S40 shown, the first key point region image is obtained based on the first face region image in the second stage, including but not limited to the following steps: S41, the first face region image is obtained in the second stage, the first face region image is subjected to key point prediction to obtain a first key point prediction result; S42, the first key point region image is obtained by cropping based on the first face region image according to the first key point prediction result, wherein the first key point region image includes all the first key points, the number of the first key points is greater than or equal to zero, and the first key points are facial feature points of the first face region image.

[0038] It should be noted that each stage of the target TSCCN respectively completes a face feature extraction task at different levels, the first stage of the target TSCCN includes a lightweight convolutional neural network (CNN), the lightweight CNN can detect the approximate position of the face, quickly locate the face region in the key frame image, and output the face region coordinates , so as to output the first face region image based on the key frame image and the face region coordinates , wherein is the coordinate of one vertex of the rectangular frame of the face region, is the coordinate of another vertex of the rectangular frame of the face region, and is opposite to .

[0039] It should be noted that the number of first key points obtained by performing key point prediction on the first face region image is greater than or equal to zero. When the number of first key points is equal to zero, there is no face in the key frame image. When the number of first key points is greater than zero, the first key point prediction result includes at least one first key point and a corresponding first key point coordinate. The first key point region image is obtained by cropping the first face region image based on all first key point coordinates.

[0040] It should be noted that in the second stage of the target TSCCN, the first face region image is predicted by a medium complexity HRNet to output first key points and first key point coordinates. The purpose of predicting the first face region image is to detect whether the face part is blocked. The number of first key points obtained represents the degree of blocking of the face part.

[0041] It should be noted that after the high-resolution network, the first key point region image obtained in the second stage is a high-resolution image, which is beneficial to more accurately locate the facial feature points and further improve the face detection accuracy and video review accuracy. In the second stage, the HRNet layer outputs the first key point, the first key point coordinate and the confidence, which can enhance the robustness of the mouth, eyes and other easily blocked areas of the face region.

[0042] In addition, in an embodiment, in S40 shown in the figure, in the third stage, the state classification result and the completeness score are determined based on the first key point region image, including but not limited to the following steps: Figure 1 S43, in the third stage, the first key point missing rate is determined based on all first key points and a first formula, wherein the expression of the first formula is , N 1total is the first key point missing rate, N 1miss is the preset number of first key points, N 1miss is the number of first key points in the first key point region image; S44, based on the first key point missing rate and the first preset threshold, the state classification result is determined, and based on the state classification result and a second formula, the completeness score is determined, wherein the expression of the second formula is , is the completeness score.

[0043] It should be noted that for a complete and unblocked face, the number of first key points is a preset number. If the face region has blocking or missing, the number of first key points is less than the preset number.

[0044] It should be noted that, in the third stage, the result of key point prediction based on the first face region image is obtained to obtain the first key point region image; the number of first key points in the first key point region image is calculated; for faces in different occlusion conditions, the number of first key points is different, and for faces without occlusion, the number of first key points is the preset number of first key points; the number of first key points in the first key point region image and the preset number of first key points are calculated to obtain N 1miss , and then calculate the first key point missing rate using the first formula.

[0045] It should be noted that the state classification result and completeness score are obtained based on the first key point missing rate, and the third stage outputs the state classification result and completeness score based on the key frame image; the audit model obtains the state classification image and completeness score sequence based on the key frame image sequence. The completeness score sequence includes the completeness scores of all key frame images in the key frame image sequence. The expression of the completeness score sequence is , The completeness score of the key frame image of the t-th frame of the video to be reviewed has T frames, T is the total number of video frames of the video to be reviewed, and t is the t-th frame of the video to be reviewed.

[0046] In addition, in one embodiment, Figure 1 The S10 shown includes but is not limited to the following steps: S11, based on any face image sample, perform key point annotation on the face image sample to obtain a second key point annotation result, and determine the second key point missing rate based on the second key point annotation result and a third formula, wherein the second key point annotation result includes a number of second key points greater than or equal to 0, and the expression of the third formula is , is the missing rate of the second key point, N 2total is the number of preset second key points, N 2miss The number of second key points in the face image sample is subtracted from the number of preset second key points; S12: Classify all facial image samples based on the second key point missing rate and the first preset threshold, and generate a data set based on the classified facial image samples.

[0047] It should be noted that, for the face image sample marked as a complete face, key points are marked to obtain a preset number of second key points and corresponding second key point coordinates. The plurality of second key point coordinates are ,in, is the horizontal coordinate of the kth second key point coordinate, is the ordinate of the kth second key point coordinate, is the kth second key point coordinate, and K is a preset number. The number of second key points of a face image sample labeled as an occluded face is less than the preset number and greater than zero, and the number of second key points of a face image sample labeled as no face is equal to zero.

[0048] It should be noted that for any face image sample subjected to key point prediction, the second key point missing rate of the face image sample is calculated by the third formula, the actual state classification of the face image sample is determined based on the preset threshold and the second key point missing rate, when the second key point missing rate is less than the preset threshold, the actual state classification of the face image sample is "complete face", or when the second key point missing rate is greater than the preset threshold and less than 100%, the actual state classification of the face image sample is "occluded face", or when the second key point missing rate is equal to 100%, i.e. no second key point is detected, the actual state classification of the face image sample is "no face", and the face images in the data set are divided into three categories: complete face, occluded face and no face.

[0049] It should be noted that by performing state three classification on the face image sample, a high-quality face image data set for training and evaluation is constructed, and a complete degree grading discrimination system based on facial key points is realized; through this standardized index system, each face image sample is given a clear label, providing a high-quality data basis for neural network training and classification, and enhancing the model's ability to adapt to complex scenes.

[0050] In addition, in an embodiment, in Figure 1 In S20 shown, a plurality of model training samples are obtained based on the data set, and the initial TSCCN is trained by all model training samples to obtain the target three-level cascade convolutional neural network TSCCN, including but not limited to the following steps: S21, based on the key factors, the orthogonal test scheme of all face image samples in the data set is constructed to obtain a plurality of model training samples, wherein the model training sample includes at least one face image sample, and the key factors include the occlusion ratio, the lighting condition and the facial expression; S22, based on any face image sample in any model training sample, the initial TSCCN crops the face image sample to obtain a second face region image, and performs key point prediction on the second face region image to obtain a third key point prediction result, and crops the second face region image based on the third key point prediction result to obtain a second key point region image, wherein the second key point region image includes all third key points, and the number of third key points is greater than or equal to zero; S23, based on all second key points and the second key point region image, a final loss function is determined, and based on the initial TSCCN and the final loss function, the target TSCCN is determined.

[0051] It should be noted that the key factors include the occlusion ratio, the lighting condition and the facial expression, all of which are factors affecting the accuracy of face recognition.

[0052] It should be noted that, in order to improve the speed and efficiency of constructing the orthogonal test scheme, the face image samples are labeled based on the key factors when labeling the key points and three-classifying the face image samples. The orthogonal test scheme is constructed, and the occlusion ratio, the lighting condition and the facial expression in the data set are directly selected as the orthogonal test factors, and different levels of each factor are set to construct a 3-factor 3-level orthogonal test scheme, which includes 9 groups of model training samples.

[0053] It should be noted that how to construct the orthogonal test scheme is prior art, and the present application does not improve it in any form, and will not be described here.

[0054] It should be noted that the model training samples are constructed to repeatedly train the initial TSCCN and improve the generalization ability and robustness of the target TSCCN in different working environments. The orthogonal test design method is introduced to systematically evaluate the performance of the target TSCCN under different external disturbance conditions.

[0055] It should be noted that the second key point is a key point obtained by labeling the face image samples based on the data set, and the third key point is a key point obtained by predicting the key points based on the face image samples based on the data set In addition, in an embodiment, in S23, the final loss function is determined based on all the second key points and the second key point region image, including but not limited to the following steps: S231, determining the predicted state classification and the predicted completeness based on all the third key points in the second key point region image, and determining the actual state classification and the actual completeness based on all the second key points; S232, determining the classification loss function based on the predicted state classification and the actual state classification, determining the regression loss function based on the predicted completeness and the actual completeness, and determining the final loss function based on the classification loss function and the regression loss function; The expression of the classification loss function is The expression of the regression loss function is The expression of the final loss function is , is a weighting factor of the classification loss function, is a weighting factor of the regression loss function, is the actual state classification, is the predicted state classification, S is the actual completeness, is the predicted completeness.

[0056] It should be noted that the second key point missing rate and the third key point missing rate are obtained based on all the second key points and all the third key points respectively, the actual state classification and the actual integrity are obtained based on the second key point missing rate, and the predicted state classification and the predicted integrity are obtained based on the third key point missing rate.

[0057] It should be noted that the actual integrity and the predicted integrity participate in the training of the target TSCCN as one of the objective functions, in order to improve the model's ability to identify different states, a multi-task learning strategy is used for joint training, a classification loss function is determined based on the actual state classification and the predicted state classification, a regression loss function is determined based on the actual integrity and the predicted integrity, and a final loss is determined based on the classification loss function and the regression loss function.

[0058] It should be noted that how to determine the final loss function based on the classification loss function and the regression loss function is prior art, and the present application does not make any form of improvement thereon, and will not be described here.

[0059] It should be noted that after the initial TSCCN inputs the image including the second key point and the second key point image to the deep CCN, the fine-grained features such as the facial contour, the skin boundary, and the hair line of the image are extracted, and the fine-grained features are used to improve the integrity scoring mechanism.

[0060] In addition, in an embodiment, in S30 shown in the figure, the to-be-audited video is read by the audit model, and a key frame image sequence is extracted based on the to-be-audited video, including but not limited to the following steps: Figure 1 S31, the audit model reads the to-be-audited video, and obtains the total frame number and the frame rate of the to-be-audited video; S32, based on the to-be-audited video, interval sampling is performed according to the frame rate, and a plurality of key frame images are obtained; S33, sort all the key frame images in time sequence, and generate a key frame image sequence based on the sorted key frame images.

[0061] It should be noted that a plurality of key frame images are extracted from the to-be-audited video, and the expression of the key frame image is T is the total frame number of the to-be-audited video, is the t-th frame of the to-be-audited video; interval sampling is performed based on the frame rate of the to-be-audited video, for example, 1 frame per second is extracted, and then the key frame images are sequentially input into the target TSCCN.

[0062] In order to better understand the technical scheme of the present application, the following specific embodiments are provided: S601, obtaining a plurality of face image samples from a plurality of sources; ​S602, based on any one face image sample, key point annotation is performed to obtain a second key point annotation result, and the face image sample is classified based on the number of second key points in the second key point annotation result and a third formula; S603, all face image samples are divided into three categories of complete face, occluded face and no face, a data set is constructed, the face image samples are labeled based on key factors, an orthogonal test scheme is constructed based on the data set and the key factors, 9 groups of model training samples are obtained, an initial TSCCN is constructed, and the initial TSCCN is trained through the 9 groups of model training samples. S604, based on any one face image sample of any one group of model training samples, a face region is located on the initial TSCCN in the first stage to output second face region coordinates, a second face region image is obtained by cropping the face image sample based on the face region coordinates, third key point prediction results are obtained by predicting key points on the initial TSCCN in the second stage, a second key point region image is obtained by cropping the second face region image based on the third key point prediction results, and a predicted state classification and a predicted completeness are obtained based on the second key point region image in the third stage of the initial TSCCN. S605, based on the second key point, an actual state classification and an actual completeness are determined, a classification loss function is determined based on the actual state classification and the predicted state classification, a regression loss function is determined based on the actual completeness and the predicted completeness, a final loss function is determined based on the classification loss function and the regression loss function, a target TSCCN is obtained based on the final loss function and the initial TSCCN, and an audit model is constructed based on the target TSCCN. S606, the audit model reads a to-be-audited video, a plurality of key frame images are obtained based on the to-be-audited video, a key frame image sequence is generated, the target TSCCN is called through the audit model, and the key frame images are input into the target TSCCN. S607, based on any one key frame image in the key frame image sequence, a face region is located on the target TSCCN in the first stage to output first face region coordinates, a first face region image is obtained by cropping the key frame image based on the first face region coordinates in the second stage of the target TSCCN, first key point prediction results are obtained by predicting key points on the first face region image, a first key point image is obtained by cropping the first face region image based on the first key point prediction results, and a state classification result and a completeness score are obtained based on the number of first key points of the first key point image, a first formula and a second formula in the third stage of the target TSCCN. S608, obtaining integrity scores of all key frame images in the key frame image sequence, generating an integrity score sequence based on all integrity scores, and determining a face quality review report based on the state classification result and the integrity score sequence.

[0063] According to the technical solution of the embodiment of the present invention, the data set is aimed at the quality control of complex work site videos, and a three-category classification system of complete faces, occluded faces and no faces is constructed to achieve accurate quantitative evaluation of the quality of face records; the audit model focuses on the quality audit of complex work site videos, and realizes the face availability evaluation in complex environments through a multi-stage target CNN cascade architecture. In the first stage of the target TSCCN, the first face area image is obtained based on the key frame image, in the second stage of the target TSCCN, the first key point area image is obtained based on the first face area image, and in the third stage of the target TSCCN, the state classification is determined based on the first key point area image. The results and completeness score are used to realize progressive detection of face areas in key frames, thereby improving the accuracy of completeness detection, and specifically solving the problem of face quality assessment caused by occlusion and light in industrial environments; the face quality audit report obtained based on the completeness score and state classification results can determine the working performance of the target TSCCN in different working environments, thereby effectively evaluating the availability of face images in complex working environments, which is more in line with the actual needs of complex working sites, focusing on the video quality review of complex working sites, and providing professional face availability assessment solutions for complex working sites by building a diverse data set and completeness scoring mechanism, filling the technical gap in this field.

[0064] like Figure 3 As shown, Figure 3 This is a structural diagram of a video quality review device for multi-stage facial image integrity detection provided by an embodiment of the present invention. The present invention also provides a video quality review device for multi-stage facial image integrity detection, comprising: The processor 701 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application. The memory 702 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 702 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the multi-stage face image integrity detection video quality auditing method of the embodiments of the present application; The input / output interface 703 is configured to realize information input and output. The communication interface 704 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 705 is configured to transmit information between various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device. The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other through the bus 705 to realize the communication connection between the devices.

[0065] The embodiments of the present application also provide an electronic device comprising the multi-stage face image integrity detection video quality auditing apparatus as described above.

[0066] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the multi-stage face image integrity detection video quality auditing method described above is realized.

[0067] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiment described above is only schematic, and the units described as separate components can be or can not be physically separated, and can be located in one place or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0068] Those skilled in the art will appreciate that all or certain steps of the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is common in the art for a person skilled in the art to appreciate that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0069] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are included in the scope defined by the claims of the present application.

Claims

1. A video quality audit method for multi-stage facial image integrity detection, characterized in that: include: Acquire multiple facial image samples, and construct a data set based on all of the facial image samples; Constructing an initial three-level cascade convolutional neural network TSCCN, obtaining multiple groups of model training samples based on the data set, training the initial TSCCN using all of the model training samples to obtain a target three-level cascade convolutional neural network TSCCN, and constructing an audit model based on the target TSCCN, wherein the target TSCCN includes a first stage, a second stage, and a third stage; Reading the video to be reviewed through the review model, extracting a key frame image sequence based on the video to be reviewed, and inputting the key frame image sequence into the target TSCCN; Based on any key frame image in the key frame image sequence, cropping the key frame image in the first stage to obtain a first face region image, obtaining a first key point region image based on the first face region image in the second stage, and determining a state classification result and a completeness score based on the first key point region image in the third stage, wherein the state classification result is used to represent the degree of occlusion of the first face region image, and the completeness score is used to represent the degree of completeness of the first face region image; A completeness score sequence is generated based on all the completeness scores, and a face quality review report is output based on the state classification result and the completeness score sequence.

2. The video quality audit method for multi-stage facial image integrity detection according to claim 1 is characterized in that: Obtaining a first key point region image based on the first face region image in the second stage includes: The second stage acquires the first face region image, performs key point prediction on the first face region image to obtain a first key point prediction result; The first key point area image is obtained by cropping based on the first face area image according to the first key point prediction result, wherein the first key point area image includes all first key points, the number of the first key points is greater than or equal to zero, and the first key points are facial feature points of the first face area image.

3. The video quality audit method for multi-stage facial image integrity detection according to claim 2 is characterized in that: In the third stage, determining a state classification result and a completeness score based on the first key point area image includes: The third stage determines the first key point missing rate based on all the first key points and the first formula, wherein the expression of the first formula is: , is the first key point missing rate, N 1total is the number of the first key points preset, N 1miss The number of the first key points is calculated by subtracting the number of the first key points located in the first key point area image from the preset number of the first key points; The state classification result is determined based on the first key point missing rate and a first preset threshold, and the completeness score is determined based on the state classification result and a second formula, wherein the expression of the second formula is: , A sequence is scored for the completeness.

4. The video quality audit method for multi-stage facial image integrity detection according to claim 3 is characterized in that: Acquiring multiple facial image samples and constructing a data set based on all of the facial image samples includes: Based on any of the facial image samples, key point annotation is performed on the facial image sample to obtain a second key point annotation result, and a second key point missing rate is determined based on the second key point annotation result and a third formula, wherein the second key point annotation result includes a number of second key points greater than or equal to zero, and the expression of the third formula is: , is the second key point missing rate, N 2total is the number of the second key points preset, N 2miss The number of the second key points in the facial image sample is subtracted from the number of the preset second key points; All the facial image samples are classified based on the second key point missing rate and the first preset threshold, and the data set is generated based on the classified facial image samples.

5. The video quality audit method for multi-stage facial image integrity detection according to claim 4 is characterized in that: Based on the data set, multiple groups of model training samples are obtained, and the initial TSCCN is trained using all of the model training samples to obtain a target three-level cascade convolutional neural network TSCCN, including: Based on key factors, an orthogonal test scheme is constructed for all the facial image samples in the data set to obtain multiple groups of model training samples, wherein the model training samples include at least one facial image sample, wherein the key factors include an occlusion ratio factor, a lighting condition factor, and a facial expression factor; Based on any of the face image samples in any of the model training samples, the initial TSCCN crops the face image sample to obtain a second face region image, performs key point prediction on the second face region image to obtain a third key point prediction result, and crops the second face region image according to the third key point prediction result to obtain a second key point region image, wherein the second key point region image includes all third key points, and the number of the third key points is greater than or equal to zero; A final loss function is determined based on all of the second key points and the second key point region image, and the target TSCCN is determined based on the initial TSCCN and the final loss function.

6. The video quality audit method for multi-stage facial image integrity detection according to claim 5 is characterized in that: Determining a final loss function based on all of the second key points and the second key point region images includes: Determining a predicted state classification and a predicted completeness based on all of the third key points in the second key point region image, and determining an actual state classification and an actual completeness based on all of the second key points; Determine a classification loss function based on the predicted state classification and the actual state classification, determine a regression loss function based on the predicted completeness and the actual completeness, and determine a final loss function based on the classification loss function and the regression loss function; Among them, the expression of the classification loss function is , the expression of the regression loss function is , the expression of the final loss function is , is the weighting factor of the classification loss function, is the weighting factor of the regression loss function, Classify the actual state, is the predicted state classification, S is the actual completeness, is the prediction completeness.

7. The video quality audit method for multi-stage facial image integrity detection according to claim 1 is characterized in that: Reading the video to be reviewed through the review model, and extracting a key frame image sequence based on the video to be reviewed, including: The review model reads the video to be reviewed and obtains the total number of frames and frame rate of the video to be reviewed; Performing interval sampling based on the video to be reviewed according to the frame rate to obtain a plurality of key frame images; All the key frame images are sorted in chronological order, and the key frame image sequence is generated based on the sorted key frame images.

8. A video quality review device for multi-stage facial image integrity detection, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the video quality review method for multi-stage facial image integrity detection as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: A video quality audit device comprising the multi-stage facial image integrity detection as described in claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video quality review method for multi-stage facial image integrity detection as described in any one of claims 1 to 7.

Images