A face recognition monitoring system based on paddlehub

Abstract

The present application relates to the technical field of face recognition, in particular to a face recognition monitoring system based on PaddleHub, which comprises an image reflection tracking module, a trusted area screening module, a dynamic score generation module, a covering state judgment module and a heat map auxiliary verification module. Through the introduction of the brightness difference value sequence of multiple facial key points and the combination of the standard deviation calculation method in the time window, the dynamic trend of facial reflection change can be obtained, the stability index of the facial area in the image can be provided, the trusted area can be obtained through index screening, the key point density distribution and the area ratio can be fused, the visual score selection of the image frame can be realized, the face recognition and the occlusion classification can be performed through the qualified image frame, multiple boundary features of the boundary image block can be extracted and fused with the mask recognition result, the low-quality image interference can be effectively avoided, the thermal occlusion feature can be judged through the comparison of the mean value of the multi-region infrared heat intensity and the temperature difference, and the accurate determination of the occlusion behavior and the thermal response presentation can be realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of face recognition, and in particular to a face recognition monitoring system based on PaddleHub. BACKGROUND

[0002] The technical field of face recognition mainly studies how to simulate the ability of a computer to recognize and distinguish human faces. The field covers core links such as image acquisition and preprocessing, face detection, face feature point positioning, feature extraction, face comparison and identity recognition. In recent years, with the evolution of deep learning technology, especially the application of algorithms such as convolutional neural networks, attention mechanisms and residual networks, the robustness of face recognition systems under complex background, light changes, expression occlusion, multi-pose and other interference conditions has been significantly improved. Face recognition is widely used in security monitoring, identity verification, financial risk control, access management, smart cities and other scenarios, and is one of the important basic support technologies for the informatization and intelligentization of modern society.

[0003] Among them, the face recognition monitoring system aims to build a monitoring system based on face recognition, which is used for automatic recognition and analysis of personnel state in public places. The system collects video data in real time through a camera device, and uses a face detection and recognition model to analyze the characters in the picture to determine whether they meet certain monitoring conditions. For example, whether to wear a mask. The system can automatically identify target behaviors, and is extended to measure temperature, track trajectories, and issue abnormality warnings, and is widely used in areas with high human traffic such as airports, stations, hospitals and campuses, and has important application value in improving public health and safety and the efficiency of epidemic prevention and control.

[0004] The traditional monitoring system only collects images in real time through a camera device and relies on a face detection model to determine the wearing state, ignoring the recognition interference caused by changes in light, motion blur or face occlusion in the image, which can easily lead to recognition errors in low-light or partially occluded situations. The system does not filter and evaluate the reliable areas in the image, resulting in low-quality images being directly involved in the judgment process. For example, when the mask occludes the upper half of the face, the traditional system cannot accurately extract the nose and mouth area information, thereby affecting the determination of the wearing type, and does not introduce thermal imaging for analysis, lacking the ability to perceive the thermal response of the occluded area, making it difficult to effectively distinguish in cases where the difference between the occlusion and the skin temperature is small, reducing the recognition stability and scene adaptability in high-density environments. SUMMARY

[0005] The purpose of the present application is to solve the problems existing in the prior art, and to provide a face recognition monitoring system based on PaddleHub.

[0006] In order to achieve the above object, the present application adopts the following technical scheme: A face recognition monitoring system based on PaddleHub, the system comprises:

[0007] The image reflection tracking module obtains a sequence of continuous image frames in the dynamic monitoring picture, locates a face image region in each frame, calls average brightness values of corresponding pixel blocks of face key points in the current frame and the previous frame, identifies a dynamic trend of reflectivity of the face in the current frame sequence with time, and obtains a sequence of brightness change stability indexes;

[0008] The trusted region screening module calls a key point set in an eye and a cheekbone region based on the sequence of brightness change stability indexes, groups the mapping points according to a Dlib standard face region division rule, and obtains a stable region annotation layer;

[0009] The dynamic score generation module obtains a horizontal and vertical distribution coordinate set in a Dlib face coordinate system based on the stable region annotation layer, calls a face region visual score threshold value for classification, selects image frames with a score result exceeding the threshold value, and obtains a set of qualified image frames with visual scores;

[0010] The covering state judgment module calls a mask recognition unit integrated by PaddleHub to identify a mask wearing type based on the set of qualified image frames with visual scores, intercepts a nose region and a mouth region boundary image block, and calculates three types of boundary feature data, i.e., an edge direction distribution, a pixel gradient density and a brightness section number, to obtain a set of shielding behavior labels.

[0011] The present application improves that the sequence of brightness change stability indexes comprises a time sequence brightness change value set, a weighted average difference value sequence, a cross-frame standard deviation average value and a region brightness fluctuation label, the stable region annotation layer comprises a face region grouping number picture, a trusted region position mask, a region pixel Boolean distribution table and a key point mapping picture, the set of qualified image frames with visual scores comprises a score judgment image index number, a face region distribution density value, a trusted region area proportion value and a set of score passing flag bits, and the set of shielding behavior labels comprises an image frame shielding category label, a mouth-nose region edge continuity level, a mask wearing state label and a shielding region classification index.

[0012] The present application improves that the image reflection tracking module comprises:

[0013] The image frame acquisition submodule obtains a sequence of continuous image frames in the dynamic monitoring picture, extracts a face region in each image frame, and respectively locates four groups of key points, i.e., a canthus, a nose bridge midpoint, a brow and a cheekbone edge, in each image frame, extracts average brightness values of pixel blocks corresponding to the key points, records the average brightness values as a cross-frame sequence, and generates a sequence of average brightness values of the key points;

[0014] The luminance difference value calculation sub-module calculates the difference value between the same key point luminance values in the current frame and the previous frame according to the key point average luminance value sequence, and constructs a difference value sequence set in chronological order, calls a set time window length parameter, performs weighted processing on the difference value sequence, calculates a weighted mean value sequence according to a weighting coefficient, and obtains a weighted luminance change sequence;

[0015] The time sequence fluctuation extraction sub-module respectively calculates the standard deviation of each group of key point change values in the sequence based on the weighted luminance change sequence, and calculates the average value of the standard deviations of all key points, which is set as the time sequence fluctuation of the whole point set, establishes an index set representing the consistency change trend of the reflectivity of the face on the time axis, and obtains a luminance change stability index sequence.

[0016] The weighting coefficient setting method in the weighted processing of the difference value sequence is that the luminance difference values at each time point in the difference value sequence are distributed in chronological order, and the difference values closer to the current frame time position are given higher weights, and the difference values farther away are given lower weights.

[0017] The weighting coefficient is calculated from the normalized ratio of the reverse index value of the time point relative to the end position of the frame sequence and the length of the difference value sequence, and the sum of the weighting coefficients is limited to a constant one after normalization.

[0018] The generation method of the weighted mean value sequence is to perform product operation on the weighting coefficient and the corresponding luminance difference value, and to perform weighted summation on all product values, and to take the weighted sum as the weighted luminance change value corresponding to the current frame position.

[0019] The application improves that the trusted area filtering module comprises:

[0020] The feature index comparison sub-module compares the index value corresponding to each key point with a set reflectivity stability standard deviation threshold based on the luminance change stability index sequence, marks the key points with index values lower than the reflectivity stability standard deviation threshold as trusted points, and obtains a region trusted key point set.

[0021] The key point screening and mapping sub-module locates the spatial position coordinates of the trusted key points in the image frame according to the region trusted key point set, constructs a trusted area graphical mask in pixel units, sets a region pixel minimum coverage area benchmark, removes the scattered point regions that do not meet the region pixel minimum coverage area benchmark, and generates trusted area image mask data.

[0022] The regional structure grouping submodule calls the trusted regional image mask data, obtains a structure regional label based on a Dlib face structure division rule, performs consistency detection on pixel labeling in a label corresponding region, performs structure classification on a labeled connected region, establishes structure regional grouping according to a structure position number, and obtains a stable regional labeling layer.

[0023] The dynamic score generation module comprises:

[0024] The regional proportion calculation submodule extracts the number of pixels in the labeled region and the total number of effective pixels in the whole face image based on the stable regional labeling layer, calculates the area ratio between the two and records it as the pixel regional proportion, and obtains the face trusted area area proportion value;

[0025] The density parameter extraction submodule calls the face trusted area area proportion value, obtains a coordinate set of the labeled region key points in the horizontal and vertical directions in the Dlib standard coordinate system, respectively counts the number of key points in each direction, and calculates the direction dispersion based on the coordinate mean value and the deviation value of each point to obtain the face region visual score value;

[0026] The threshold screening and classification submodule calls the face region visual score threshold according to the face region visual score value, compares the face region visual score value of the image frame by frame, screens the image frame number of the face region visual score value greater than the face region visual score threshold, and obtains a set of qualified image frames of the visual score.

[0027] The covering state determination module comprises:

[0028] The face structure extraction submodule calls the PaddleHub face detection unit to extract the face frame position and five feature key point coordinates in each image based on the set of qualified image frames of the visual score, positions the nose area and mouth area boundary image block, and simultaneously calls the PaddleHub mask recognition unit to obtain the mask wearing type classification label of the corresponding face in the current image, the type classification standard is no wearing, partial covering and complete covering, establishes a mapping data set of the corresponding frame number and the three types of wearing labels, and obtains a wearing structure recognition result set;

[0029] The boundary feature calculation submodule extracts the boundary image block and the surrounding image block corresponding to the nose area and the mouth area in each image frame according to the wearing structure recognition result set, respectively performs edge direction gradient statistics, pixel gray gradient histogram density extraction and continuous brightness section number counting in the image block region, performs numerical difference and ratio conversion according to the feature indexes of the center region and the surrounding region, obtains the boundary definition deviation value, and obtains a nose and mouth region boundary feature parameter group;

[0030] The occlusion label generation submodule adds three types of boundary feature indicators after weight normalization based on the nose and mouth area boundary feature parameter set, the set of wearing structure recognition results in the corresponding image frame, calculates a fusion occlusion score value, and performs classification according to a set boundary occlusion judgment reference value, delimits the occlusion behavior level of the corresponding image frame, establishes the corresponding relationship between the frame number and the occlusion label, and obtains an occlusion behavior label set.

[0031] The system further comprises:

[0032] The heat map assisted verification module calls the pixel of the upper nose region, the middle nose to the philtrum region and the philtrum to the lower jaw region based on the occlusion behavior label set, calculates the infrared heat intensity mean value of the region, and calculates the temperature difference value between adjacent regions, calls the heat occlusion identification temperature difference threshold value for comparison to determine whether the heat coverage feature is met, and obtains an occlusion image heat response layer.

[0033] The occlusion image heat response layer comprises a heat occlusion region temperature difference map, an infrared signal partition response table, a heat feature anomaly marking map and a region heat coverage state map.

[0034] The heat map assisted verification module comprises:

[0035] The region heat value extraction submodule extracts the frame number of the image frame labeled as partial covering and complete covering based on the occlusion behavior label set, obtains the corresponding frame heat imaging image, and intercepts the pixel data of the upper nose region, the middle nose to the philtrum region and the philtrum to the lower jaw region, calculates the average value of the infrared signal value of each region pixel point, and obtains a three-region heat intensity mean value group.

[0036] The temperature difference feature calculation submodule calls the three-region heat intensity mean value group, calculates the relative change range of the heat value of the philtrum region and the upper nose region and the lower jaw region, extracts the upper nose region heat intensity mean value, the lower jaw region heat intensity mean value and the middle nose to the philtrum region heat intensity mean value, and uses the formula:

[0037] ;

[0038] The operation obtains a heat change fusion coefficient;

[0039] wherein, the heat change fusion coefficient, the upper nose region infrared heat intensity mean value, the lower jaw region infrared heat intensity mean value, the middle nose to the philtrum region infrared heat intensity mean value, the addition denominator stability factor;

[0040] The thermal response layer generation submodule calls a thermal occlusion identification temperature difference threshold according to the thermal change fusion coefficient, compares the thermal change fusion coefficient of each frame of image with the thermal occlusion identification temperature difference threshold one by one, screens the image frame number with the thermal change fusion coefficient greater than the thermal occlusion identification temperature difference threshold, and performs occlusion region labeling to obtain an occlusion image thermal response layer.

[0041] Compared with the prior art, the application has the advantages and positive effects that:

[0042] In the application, by introducing the brightness difference value sequence of multiple sets of face key points and combining the standard deviation calculation mode in the time window, the dynamic trend of face reflection change can be obtained, the stability index of the face region in the image can be provided, the reliable region can be obtained through index screening, the key point density distribution and area proportion are fused, the visual score selection of the image frame is realized, the face recognition and occlusion classification are performed through the qualified image frame, multiple boundary features of the boundary image block are extracted and the mask recognition result is fused, the low-quality image interference is effectively avoided, the thermal occlusion feature is judged through the comparison of the mean value of the infrared thermal intensity and the temperature difference, the accurate determination of the occlusion behavior and the thermal response presentation are realized, and the recognition accuracy and analysis stability of the wearing state under the condition of complex occlusion and low contrast are improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The system flowchart of the application is shown in the figure;

[0044] Figure 2 The flowchart of the image reflection tracking module of the application is shown in the figure;

[0045] Figure 3 The flowchart of the reliable region screening module of the application is shown in the figure;

[0046] Figure 4 The flowchart of the dynamic score generation module of the application is shown in the figure;

[0047] Figure 5 The flowchart of the covering state determination module of the application is shown in the figure;

[0048] Figure 6 The flowchart of the thermal map auxiliary verification module of the application is shown in the figure. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the application more clear, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the application.

[0050] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0051] Referring to Figure 1 The present application provides a technical solution: a face recognition monitoring system based on PaddleHub, the system comprising an image reflection tracking module, a trusted area screening module, a dynamic score generation module, a covering state determination module and a heat map auxiliary verification module;

[0052] The image reflection tracking module acquires a sequence of continuous image frames in a dynamic monitoring picture, locates a face image area in each frame, calls the average brightness values of the corresponding pixel blocks of the four groups of key points of the eye corner, the nose bridge midpoint, the brow and the cheekbone edge in the current frame and the previous frame, calculates the difference sequence between the average brightness values, and performs a weighted average on the differences based on the current time window, collects the standard deviation of the average result, and calculates the average standard deviation value of the full point set, acquires the dynamic trend of the reflectivity of the face in the current frame sequence with time, and obtains a sequence of brightness change stability indicators;

[0053] The average brightness value is an indicator commonly used in image processing to describe the brightness intensity of an image area; the weighted average is commonly used in time series data analysis, and the average is obtained by assigning different weights to the current frame and the historical frame to enhance the continuity analysis; the sequence of brightness change stability indicators indicates the time sequence stability degree of the brightness difference values of the multiple key points in the continuous frames, and reflects the consistency of the reflection characteristics in the dynamic scene;

[0054] The trusted area screening module calls the key point set in the eye and cheekbone areas based on the sequence of brightness change stability indicators, compares the point indicator values with the standard deviation threshold value item by item, screens the key points whose indicators are lower than the threshold value and maps them to the corresponding areas in the image, groups the mapped points according to the Dlib standard face area division rule, and obtains a stable area annotation layer;

[0055] The dynamic score generation module calculates the area ratio value of the stable area annotation layer in the face image based on the stable area annotation layer, acquires the horizontal and vertical distribution coordinate set in the Dlib face coordinate system, calculates the key point density in the corresponding direction, combines the density sum and the area ratio value, calls the face area visual score threshold value for classification screening, selects the image frames whose score results exceed the threshold value, and obtains a set of qualified image frames with visual scores.

[0056] The face region visual score threshold is a standard value for distinguishing whether the face region in the image frame has valid recognition conditions, and is usually set according to the area proportion of the face region of the successfully recognized image in the training sample and the uniformity of the region distribution;

[0057] The covering state judgment module calls the face detection unit integrated by PaddleHub to extract the face frame position and the five feature points based on the visual score qualified image frame set, calls the mask recognition unit integrated by PaddleHub to identify the mask wearing type of the corresponding face in each image frame, the type classification standard is no wearing, partial covering and complete covering, positions the nose area and the mouth area in the reliable region, intercepts the nose area and the mouth area boundary image block and calculates three types of boundary feature data including the edge direction distribution, the pixel gradient density and the brightness section number, compares the boundary change ratio between the nose and mouth area boundary image block and the surrounding area image block, and fuses the boundary feature and the PaddleHub recognition result to obtain a set of shielding behavior labels;

[0058] The heat map assisted verification module extracts the corresponding heat image region in the partial covering and complete covering image frame based on the shielding behavior label set, calls the pixels of the three regions of the upper nose area, the middle nose to the philtrum area and the philtrum to the lower jaw area, calculates the infrared thermal intensity mean value of the region, and calculates the temperature difference value between adjacent regions, compares and judges whether the heat covering feature is met by calling the heat shielding recognition temperature difference threshold to obtain a shielding image heat response layer;

[0059] The infrared thermal intensity mean value represents the average value of the infrared signal intensity of a region in the heat image, which is used to identify the thermal response difference caused by shielding in different regions; the heat shielding recognition temperature difference threshold is usually obtained based on the actual thermal imaging equipment and the face temperature difference sample data, which can be obtained by experience parameter adjustment or heat image data analysis;

[0060] The brightness change stability index sequence includes a set of time sequence brightness change values, a weighted average difference value sequence, a cross-frame standard deviation average value and a region brightness fluctuation label, the stable region label layer includes a face region grouping number map, a reliable region position mask, a region pixel Boolean distribution table and a key point mapping map, the visual score qualified image frame set includes a score judgment image index number, a face region distribution density value, a reliable region area proportion value and a score passing flag bit set, the shielding behavior label set includes an image frame shielding category label, a mouth and nose region edge continuity level, a mask wearing state label and a shielding region classification index, and the shielding image heat response layer includes a heat shielding region temperature difference map, an infrared signal partition response table, a heat feature anomaly label map and a region heat covering state map.

[0061] Please refer to Figure 2 , the image reflection tracking module includes:

[0062] The image frame acquisition submodule acquires a sequence of continuous image frames in the dynamic monitoring picture, extracts a face region in each image frame, and respectively locates four groups of key points, i.e., an outer corner of an eye, a midpoint of a nose bridge, a brow, and a cheekbone edge in each image frame, extracts average brightness values of pixel blocks corresponding to the key points, and records the average brightness values as a sequence of key point average brightness values;

[0063] According to a sequence of 10 continuous image frames, for example, image frames F1 to F10, in the dynamic monitoring picture, first, the first image frame F1 is processed, a face detection algorithm is called to locate and extract a face region image with a size of 480x640 pixels, then, in the face region, four groups of key points, i.e., an outer corner of a left eye, an outer corner of a right eye, a midpoint of a nose bridge, a left brow, a right brow, a left cheekbone edge point, and a right cheekbone edge point, are located by using an active shape model, a pixel block in a 5x5 pixel region around each key point is extracted, an arithmetic mean value of brightness values of all pixels in the pixel block is calculated, for example, the average brightness value of the outer corner of the left eye in F1 is 158, the average brightness value of the midpoint of the nose bridge is 182, the average brightness value of the left brow is 145, and the average brightness value of the left cheekbone edge is 165, the operation is repeated for each image frame of F1 to F10, and average brightness values of all key points in each frame are recorded, thereby forming seven independent sequences of cross-frames, for example, the sequence of average brightness values of the outer corner of the left eye is [158, 159, 157, 160, 161, 158, 156, 159, 160, 158], and finally, a sequence set of average brightness values of all four groups of key points (a total of seven specific points) is generated.

[0064] The brightness difference calculation submodule calculates a difference value between the same key point brightness values in the current frame and the previous frame according to the sequence of key point average brightness values, constructs a sequence set of difference values in chronological order, calls a set time window length parameter, performs weighted processing on the sequence of difference values, calculates a weighted mean value sequence according to the weighted coefficients, and obtains a weighted brightness change sequence;

[0065] The weighted coefficient setting mode in the weighted processing of the sequence of difference values is that the brightness difference values at each time point in the sequence of difference values are distributed in chronological order, the difference values closer to the current frame time position are given higher weights, and the difference values farther away are given lower weights;

[0066] The weighted coefficient is calculated from a reverse index value of a time point relative to a position at the end of the frame sequence and a normalized ratio of the length of the sequence of difference values, and after normalization, the sum of the weighted coefficients is limited to a constant one;

[0067] The generation mode of the weighted mean value sequence is that the weighted coefficients and the corresponding brightness difference values are multiplied, and all the products are weighted and summed, and the weighted sum is taken as a weighted brightness change value corresponding to the current frame position;

[0068] The brightness value sequence [158, 159, 157, 160, 161, 158, 156, 159, 160, 158] of the left outer eye corner key point is processed, and the brightness difference value between the current frame and the previous frame is calculated. From the second frame, the difference between F2 and F1 is 159-158=1, the difference between F3 and F2 is 157-159=-2, and so on. The difference sequence [1, -2, 3, 1, -3, -2, 3, 1, -2] is obtained. The time window length parameter is set to 5, that is, only the difference values of the last 5 time points are considered. The difference sequence [-3, -2, 3, 1, -2] at the position of the current frame F10 is weighted. The setting method of the weighting coefficient is as follows: first, get the length of the difference sequence, which is 5 here, then calculate the reverse index value of each time point relative to the end position of the sequence. The difference -3 of F6 corresponds to the reverse index 5, the difference -2 of F7 corresponds to the reverse index 4, the difference 3 of F8 corresponds to the reverse index 3, the difference 1 of F9 corresponds to the reverse index 2, and the difference -2 of F10 corresponds to the reverse index 1. Normalize these reverse index values, calculate the sum of the reverse index values as 5+4+3+2+1=15, and the initial weights of each position are 5 / 15, 4 / 15, 3 / 15, 2 / 15, and 1 / 15. To limit the sum of the weighting coefficients to a constant 1, directly use the normalized ratio as the final weighting coefficient, that is, the weight sequence is [0.333, 0.267, 0.200, 0.133, 0.067]. Then generate the weighted mean value sequence. Multiply the weighting coefficient and the corresponding brightness difference value. Specifically, -3*0.333+(-2)*0.267+3*0.200+1*0.133+(-2)*0.067, that is, -0.999-0.534+0.600+0.133-0.134. Sum all the products to get the weighted brightness change value corresponding to the position of the current frame F10, which is -0.934. Repeat this process for all frames and all key point sequences to obtain the complete weighted brightness change sequence.

[0069] The time sequence fluctuation extraction submodule respectively calculates the standard deviation of each group of key point change values in the sequence based on the weighted brightness change sequence, and calculates the average of the standard deviations of all key points. Set as the time sequence fluctuation of the whole point set. Establish an index set representing the reflectivity consistency change trend of the face on the time axis to obtain the brightness change stability index sequence.

[0070] Based on the weighted brightness change sequence of the left eye outer corner key point within 10 frames, for example [-0.5, 0.2, -0.8, 1.1, 0.4, -0.9, 0.6, 1.2, -0.3, -0.934], the standard deviation of the sequence is calculated, first calculate the average value of the sequence as (-0.5+0.2-0.8+1.1+0.4-0.9+0.6+1.2-0.3-0.934) / 10=0.0066, then calculate the square of the difference between each value and the average value, divide the sum by the sequence length 10, and take the square root to get the standard deviation about 0.843, the weighted brightness change sequence of the four groups of eye corner, nose bridge midpoint, brow and cheekbone edge key points is calculated in the same way, and 7 standard deviation values are obtained, for example [0.843, 0.912, 0.795, 0.881, 0.865, 0.901, 0.857], calculate the arithmetic mean of the 7 standard deviations, (0.843+0.912+0.795+0.881+0.865+0.901+0.857) / 7=0.865, set the average value as the full point set time sequence fluctuation of the current image frame sequence, through the sliding window calculation of the continuous frame sequence of the whole video stream, an index set representing the reflectivity consistency change trend of the face on the time axis is established, and a brightness change stability index sequence is obtained.

[0071] Please refer to Figure 3 The trusted area screening module includes:

[0072] The feature index comparison submodule calls the preset eye and cheekbone area key point set in the face image based on the brightness change stability index sequence, compares the index value corresponding to each key point with the set reflection stability standard deviation threshold, marks the key points with index values lower than the reflection stability standard deviation threshold as trusted points, and obtains a region trusted key point set;

[0073] Based on the luminance change stability index sequence [0.865, 0.871, 0.855, …], a preset eye and cheek region key point set in the face image is called, which includes the left outer corner of the eye, the right outer corner of the eye, the left cheek edge point and the right cheek edge point. The index value corresponding to each key point, that is, its own weighted luminance change sequence standard deviation, is compared with the set reflection stability standard deviation threshold value. The threshold value is set with reference to a benchmark database containing 1000 samples. The images in the database are all unobstructed faces shot under stable light source (illuminance 500 lux, color temperature 4000K). The 30th percentile of the standard deviation of the key point luminance change sequence corresponding to all samples in the database is calculated. For example, for the left outer corner of the eye key point, the 30th percentile value of the standard deviation in the database is 0.850. Therefore, 0.850 is set as the reflection stability standard deviation threshold value of the key point. The existing individual left outer corner of the eye key point index value is 0.843, which is lower than 0.850. Therefore, the key point is marked as a reliable point. If the right outer corner of the eye index value is 0.912, which is higher than the threshold value 0.855, it is marked as an unreliable point. The comparison is performed on all key points in the set to obtain a region reliable key point set.

[0074] The key point screening and mapping sub-module locates the spatial position coordinates of the reliable key points in the image frame according to the region reliable key point set, and constructs a reliable region image mask in pixel area units. The region pixel minimum coverage area benchmark is set. The scattered point regions that do not meet the region pixel minimum coverage area benchmark are removed to generate reliable region image mask data.

[0075] For example, the set contains two reliable point positions of the outer corner of the left eye and the edge point of the left cheekbone. First, the spatial position coordinates of the two reliable key points in the image frame are located. The coordinates of the outer corner of the left eye are (125, 230), and the coordinates of the edge point of the left cheekbone are (140, 310). A circular pixel area with a radius of 10 pixels is constructed with these coordinates as the center. All pixels in the circular area are marked as 1, and pixels outside the area are marked as 0. In this way, a reliable region mask is constructed. Then, a minimum coverage area criterion for region pixels is set. The criterion value is set to the area of a circular region with a diameter of 15 pixels, that is, π*(15 / 2)^2≈177 pixels. Each connected region in the mask image is detected, and the total number of pixels in each connected region is calculated. It is assumed that the two circular regions generated from the outer corner of the left eye and the edge point of the left cheekbone are not connected in space, and the area of each region is π*10^2≈314 pixels. Since 314 is greater than 177, both regions meet the minimum coverage area criterion. If there is a connected region generated by a single noise point, the area of the region is only 50 pixels, which is less than 177. Therefore, the scatter region that does not meet the minimum coverage area criterion of the region pixels is removed, and the pixel value is set to 0 from 1. Finally, the reliable region mask data is generated.

[0076] The region structure grouping submodule calls the reliable region mask data, obtains the structure region label based on the Dlib face region structure division rule, and performs consistency detection on the pixel label in the region corresponding to the label. The structure classification is performed on the labeled connected region, and the structure region grouping is established according to the structure position number to obtain a stable region label layer.

[0077] The generated reliable region mask data is called. The data is a binary matrix as large as the face image, wherein the pixel value of the reliable region is 1, and the rest is 0. Based on the region structure division rule of the 68 key points of the Dlib face, the face is divided into structure regions such as left eye, right eye, nose, mouth, left eyebrow, right eyebrow, and chin. The label of each pre-defined structure region is obtained, for example, the pixel range corresponding to the “left eye region” label. Then, consistency detection is performed on the pixel label in the region corresponding to the label, that is, the proportion of the number of pixels with a mask data of 1 in the “left eye region” to the total number of pixels in the region is calculated. If the proportion exceeds 75%, it is considered that the label of the structure region is consistent. The structure classification is performed on all connected regions with a mask data of 1. The connected regions that pass the consistency detection are numbered according to the structure position to which they belong, for example, the label layer numbered 1 belongs to the left eye region, and the label layer numbered 2 belongs to the left cheekbone region. The structure region grouping is established to obtain a stable region label layer.

[0078] Please refer to Figure 4 , the dynamic score generation module comprises:

[0079] The area proportion calculation submodule extracts the number of pixels in the marked area and the total number of effective pixels in the whole face image based on the stable area marking layer, calculates the area ratio between the two and records it as the pixel area proportion, and obtains the face reliable area area proportion value;

[0080] The layer marks the area in the face image that is determined to be stable and reliable, extracts the number of pixels marked as stable area in the marking layer, for example, the total number of stable area pixels is 85400 pixels by pixel counting, at the same time, the total number of effective pixels in the whole face image is extracted, which is the number of all pixels in the face detection frame, for example, 480*640=307200 pixels, the area ratio between the two is calculated, that is, 85400 / 307200≈0.278, and the ratio is recorded as the pixel area proportion, and the face reliable area area proportion value is obtained .

[0081] The density parameter extraction submodule calls the face reliable area area proportion value, obtains the coordinate set of the key points in the marking area in the horizontal and vertical directions in the Dlib standard coordinate system, respectively counts the number of key points in each direction, and calculates the direction dispersion based on the coordinate mean value and the deviation value of each point, using the formula:

[0082] ;

[0083] The operation obtains the face region visual score value;

[0084] wherein, is the face region visual score value, represents the coordinate value of the i-th key point in the horizontal direction, which is obtained by the Dlib standard face key point detection model, represents the arithmetic mean value of the i-th key point in the horizontal direction, reflecting the mean value distribution of the key points in the horizontal direction, represents the normalized value of the horizontal length of the image, which is obtained by converting the ratio of the original width of the image to the width of the standard face region, represents the coordinate value of the i-th key point in the vertical direction, which is obtained by the Dlib standard face key point detection model, represents the arithmetic mean value of the i-th key point in the vertical direction, reflecting the mean value distribution of the key points in the vertical direction, represents the normalized value of the vertical length of the image, which is obtained by converting the ratio of the original height of the image to the height of the standard face region, represents the number of key points in the horizontal direction and the vertical direction, respectively, which is obtained by counting the distribution of the total number of key points in the two directions, represents the number of key points in the horizontal direction and the vertical direction, respectively, which is obtained by counting the distribution of the total number of key points in the two directions, represents the number of key points in the horizontal direction and the vertical direction, respectively, which is obtained by counting the distribution of the total number of key points in the two directions, represents the number of key points in the horizontal direction and the vertical direction, respectively, which is obtained by counting the distribution of the total number of key points in the two directions, represents the number of key points in the horizontal direction and the vertical direction, respectively, which is obtained by counting the distribution of the total number of key points in the two directions, represents the number of key points in the horizontal direction and the vertical direction, respectively, which is obtained by counting the distribution of the total number of key points in the two directions, The face reliable area area proportion value represents a face region visible score threshold setting mode, reference is extracted in the training set face structure clear, no shelter, light balanced image score parameter mean, according to the 70th percentile corresponding score value as a threshold reference, ensure that the selected image frame in the score parameter on the stable structure distribution and region coverage characteristics meet the identification conditions;

[0085] Call face reliable area area proportion value , and get the coordinate set of all key points in the horizontal and vertical directions in the stable region marked in the Dlib standard 68-point coordinate system, as shown in table 1, the number of key points in the horizontal direction (X coordinate) and the vertical direction (Y coordinate) is , the coordinate mean value and the deviation value of each point are calculated respectively, and the direction dispersion is calculated, the process is realized by formula:

[0086] ;

[0087] In the formula, is the face region visible score value, represents the coordinate value of the first trusted key point in the horizontal direction, the subscript represents the index of the key point, is the arithmetic mean of the horizontal coordinates of all trusted key points , represents the summation operation of the horizontal coordinates from the first key point to the key point, calculate the absolute deviation of each horizontal coordinate and its average value, the whole numerator part adds the total dispersion of all key points in the horizontal direction, and the denominator is the total number of key points in the horizontal direction, is the normalized value of the image horizontal direction length, and similarly, , , , , , respectively correspond to the parameters of the vertical direction, the two score items calculate the normalized average dispersion in the horizontal and vertical directions respectively, add the two dispersion degrees, and then multiply the face reliable area area proportion value , finally get the face region visible score value which considers the key point distribution dispersion and the reliable area coverage area , the benefit of the formula is that the normalized dispersion of key points in the horizontal and vertical directions and the area proportion of reliable area The multiplication and comprehensive evaluation of the geometric stability of the facial features and the effectiveness of the regional signal avoid misjudgment caused by good local lighting but overall region being blocked, or complete region but feature points being distributed abnormally due to blur.

[0088] Table 1Dlib trusted key point coordinate table

[0089] Key point number Horizontal coordinate (pixels) Vertical coordinate (pixels) 1 180 255 2 260 258 3 220 310 4 200 360 5 250 362

[0090] As shown in Table 1, the coordinates of 5 trusted key points are extracted, and first, the parameter values are calculated, the horizontal coordinates The set is [180, 260, 220, 200, 250], and the arithmetic mean value The original width of the image is 640 pixels, and the reference value of the standard facial region width is 300 pixels, so the normalized value The set of vertical coordinates is [255, 258, 310, 360, 362], and the arithmetic mean value The original height of the image is 480 pixels, and the reference value of the standard facial region height is 400 pixels, so the normalized value The number of horizontal key points The number of vertical key points The area ratio of the trusted facial region Substitute the parameters into the formula to calculate:

[0091] ;

[0092] The operation obtains a face region visual score value of 13.17, and the setting mode of the face region visual score threshold value is to collect a training set containing 5000 images, wherein the images have clear facial structure, no occlusion, and balanced lighting, and the visual score value of each image is calculated The 5000 score values are sorted from low to high, and the score value at the 70th percentile, i.e., the 3500th position after sorting, is selected, for example, the value is 11.50, which is taken as the threshold reference. The result shows that the visual score value 13.17 calculated in the current frame is higher than the threshold value 11.50, which means that the face visual quality of the image frame meets the requirements.

[0093] The threshold screening and classification submodule calls the face region visual score threshold value according to the face region visual score value, compares the face region visual score value of the image frame by frame, screens the image frame number with a face region visual score value greater than the face region visual score threshold value, and obtains a set of qualified visual score image frames;

[0094] According to the calculated face region visibility score value 13.17, the set face region visibility score threshold 11.50 is called for comparison with the current image frame, since 13.17 is greater than 11.50, it is determined that the visibility score of the current frame is qualified, and the number of the image frame, for example, “Frame_0010”, is recorded. The calculation and comparison process of the visibility score value is repeatedly performed for each image frame in the video sequence, and the image frame numbers that satisfy the condition of the face region visibility score value being greater than the face region visibility score threshold are screened out, for example, [Frame_0010, Frame_0011, Frame_0012, Frame_0015, …], to obtain a set of qualified visibility score image frames.

[0095] Please refer to Figure 5 , the covering state determination module comprises:

[0096] The face structure extraction submodule is based on the set of qualified visibility score image frames, calls the PaddleHub face detection unit to extract the face frame position and five feature point coordinates in each image, locates the nose area and mouth area boundary image blocks, and simultaneously calls the PaddleHub mask recognition unit to obtain the mask wearing type classification label of the corresponding face in the current image. The type classification standard is no wearing, partial covering, and complete covering. A mapping data set of the corresponding frame number and the three types of wearing labels is established to obtain a wearing structure recognition result set.

[0097] Based on the set of qualified visibility score image frames [Frame_0010, Frame_0011, …], the first frame Frame_0010 is processed, the PaddleHub face detection unit is called to process the frame image data, and the face frame position coordinates [x1, y1, x2, y2] are returned as [150, 200, 350, 450]. The five feature point coordinates are obtained, and then the nose area and mouth area boundaries are located according to these key point coordinates. The corresponding image blocks are intercepted, the PaddleHub mask recognition unit is called, the image content in the face frame [150, 200, 350, 450] is analyzed, and the mask wearing type classification label of the corresponding face in the current image is returned. According to the preset type classification standard, the classification label is determined as “partial covering”, a mapping relationship between the current frame number “Frame_0010” and the wearing label “partial covering” is established, and the mapping relationship is stored in the data set. The same operation is performed on all frames in the qualified image frame set to obtain a wearing structure recognition result set.

[0098] The boundary feature calculation sub-module extracts the boundary image block and the peripheral image block corresponding to the nose area and the mouth area in each image frame according to the wearing structure recognition result set, respectively performs edge direction gradient statistics, pixel gray gradient histogram density extraction and continuous brightness section number counting on the image block area, performs numerical difference and ratio conversion according to the feature indexes of the central area and the peripheral area, obtains the boundary definition deviation value, and obtains the nose and mouth area boundary feature parameter group;

[0099] The boundary image block corresponding to the nose area and the mouth area in the image frame Frame_0010 is extracted, the size of the image block is 100x50 pixels, and the peripheral image block is expanded by 10 pixels. In the boundary image block area, first, the edge direction gradient is calculated, the number of main direction gradients is 1280, second, the pixel gray gradient histogram is extracted, the information entropy density is 4.75, and then the image block is binarized, and the number of continuous brightness sections is 35. The same operation is performed on the peripheral image block, and the number of main direction gradients is 850, the information entropy density is 6.20, and the number of continuous brightness sections is 60. According to the feature indexes of the central area and the peripheral area, the numerical difference and the ratio conversion are performed, the definition deviation value is (1280-850) / 850+(6.20-4.75) / 6.20+(60-35) / 60≈0.506+0.234+0.417=1.157, the boundary definition deviation value is obtained, and the nose and mouth area boundary feature parameter group is obtained.

[0100] The occlusion label generation sub-module calculates the fusion occlusion score value by adding the three types of boundary feature indexes after weight normalization based on the nose and mouth area boundary feature parameter group and the wearing structure recognition result set in the corresponding image frame, performs classification according to the set boundary occlusion judgment reference value, divides the occlusion behavior level of the corresponding image frame, establishes the corresponding relationship between the frame number and the occlusion label, and obtains the occlusion behavior label set.

[0101] Based on the nose and mouth region boundary feature parameter set, the boundary sharpness deviation value 1.157 is combined with two other feature indicators, such as texture complexity difference 0.85 and color consistency distance 0.62, to obtain the corresponding wearing structure recognition result "partial covering" of Frame_0010. After weight normalization and addition of the three boundary feature indicators, the weights are adjusted according to the occlusion type. For "partial covering", the weight distribution is 0.5 for sharpness, 0.3 for texture, and 0.2 for color. The calculated fusion occlusion score value = 1.157*0.5 + 0.85*0.3 + 0.62*0.2 = 0.5785 + 0.255 + 0.124 = 0.9575. The boundary occlusion judgment reference value is set based on the fusion occlusion score value distribution calculated from 500 known occlusion samples and 500 non-occlusion samples. The middle value 0.70 in the overlapping region of the two sample distributions is taken as the classification reference. The score value 0.9575 is compared with the reference value 0.70. Since 0.9575 is greater than 0.70, the occlusion behavior level of the corresponding image frame Frame_0010 is determined as "confirmed occlusion". The corresponding relationship between the frame number "Frame_0010" and the occlusion label "confirmed occlusion" is established to obtain the occlusion behavior label set.

[0102] Please refer to Figure 6 , the heat map assisted verification module includes:

[0103] The regional heat value extraction submodule extracts the image frame numbers labeled as partial covering and complete covering based on the occlusion behavior label set, obtains the corresponding thermal imaging image, and extracts the pixel data of the upper nose region, the middle nose to the philtrum region, and the philtrum to the lower jaw region. The average value of the infrared signal value of each region is calculated to obtain the three regional heat intensity mean value group.

[0104] Extract all image frame numbers labeled as "partial covering" and "complete covering", such as Frame_0010. Obtain the corresponding thermal imaging image of the frame. On the thermal imaging image, the pixel data of the three rectangular regions of the upper nose region (brow center to nose bridge), middle nose to philtrum region, and philtrum to lower jaw region is extracted according to the position of the face key points. For example, the upper nose region is extracted as a 20x30 pixel region. The infrared signal values of the 600 pixel points in the region are counted, with a signal value range of 0-255. The arithmetic mean value is 198.5. Similarly, the middle nose to philtrum region heat intensity mean value is calculated as 150.2, and the philtrum to lower jaw region heat intensity mean value is calculated as 195.8. The three regional heat intensity mean value group [198.5, 150.2, 195.8] is obtained.

[0105] The temperature difference feature calculation submodule calls the three-section region thermal intensity mean group, calculates the relative change range of the thermal values of the philtrum region and the upper nose region and the mandible region, extracts the upper nose region thermal intensity mean, the mandible region thermal intensity mean, and the middle nose to philtrum region thermal intensity mean, and uses the formula:

[0106] ;

[0107] The thermal change fusion coefficient is obtained by operation;

[0108] wherein, is the thermal change fusion coefficient, represents the upper nose region infrared thermal intensity mean, which is obtained by extracting the pixels in the region above the nasal bridge in the thermal map and calculating the infrared gray average value thereof, represents the mandible region infrared thermal intensity mean, which is obtained by intercepting the pixels in the region below the philtrum to the mandible boundary and calculating the infrared gray average value thereof, represents the middle nose to philtrum region infrared thermal intensity mean, which is obtained by extracting the middle pixels in the three-section region and calculating the thermal value mean, represents an additive denominator stabilizing factor, which is a fixed value of 0.01, and the thermal occlusion recognition temperature difference threshold is set in the following manner: according to the abnormal value distribution range extracted from the thermal coverage region feature statistical results in multiple groups of mask wearing images, the thermal change fusion coefficient value corresponding to the 95% quantile point is used as the thermal occlusion threshold setting standard, so that the determination result has a response boundary facing the abnormal occlusion region;

[0109] The three-section region thermal intensity mean group [198.5, 150.2, 195.8] is called, the relative change range of the thermal values of the philtrum region and the upper nose region and the mandible region is calculated, the upper nose region thermal intensity mean , the mandible region thermal intensity mean , and the middle nose to philtrum region thermal intensity mean are extracted, and the thermal change fusion coefficient is obtained by operation using the following formula:

[0110] ;

[0111] In the formula, is the thermal change fusion coefficient, represents the upper nose region infrared thermal intensity mean, represents the mandible region infrared thermal intensity mean, represents the middle nose to philtrum region infrared thermal intensity mean, is a fixed additive denominator stabilizing factor, and its value is 0.01, and the first term of the formula calculates the thermal value of the middle nose region and the thermal value mean the absolute difference of the heat values of the two side regions, and the square root of the sum of the squares of the three regions is used for normalization, which measures the degree of temperature depression or protrusion of the middle region relative to the two sides, and the second term calculates the absolute difference of the heat values of the upper and lower regions, and normalizes it with the sum of the two, which measures the temperature difference of the upper and lower regions themselves, and the two are added together to obtain a fusion coefficient that can reflect the temperature difference between the mouth and nose regions and the temperature gradient of the face itself The advantage of the formula is that by combining the relative temperature difference of the central region and the absolute temperature difference of the two end regions, it can more robustly detect the atypical temperature distribution caused by the heat accumulation or dissipation of objects such as masks, the first term is sensitive to the obstruction of the mouth and nose region, and the second term provides a baseline for environmental or physiological temperature difference, and the combination of the two improves the recognition accuracy.

[0112] Substitute the obtained parameter values into the formula for calculation, where , , , :

[0113] ;

[0114] The operation obtains a thermal change fusion coefficient of 0.2635, and the setting method of the thermal obstruction recognition temperature difference threshold is as follows: 1000 groups of thermal imaging images of subjects wearing masks of different materials are collected, the thermal change fusion coefficient of each group of images is calculated, the 1000 coefficient values are sorted, and the 95th percentile value after sorting, i.e. the 950th value, is used as the thermal change fusion coefficient value 0.2250 as the thermal obstruction threshold setting standard. The result shows that the currently calculated thermal change fusion coefficient 0.2635 is greater than the threshold 0.2250, which means that there is a significant thermal obstruction feature in the image.

[0115] The thermal response layer generation submodule compares the thermal change fusion coefficient of each frame of image with the thermal obstruction recognition temperature difference threshold one by one according to the thermal change fusion coefficient, screens the image frame number whose thermal change fusion coefficient is greater than the thermal obstruction recognition temperature difference threshold, and performs obstruction region labeling to obtain the thermal response layer of the obstructed image;

[0116] According to the calculated thermal change fusion coefficient 0.2635, the thermal obstruction recognition temperature difference threshold 0.2250 is called, the thermal change fusion coefficient of the current frame is compared with the thermal obstruction recognition temperature difference threshold, since 0.2635 is greater than 0.2250, the image frame number Frame_0010 is screened out, and the obstruction region labeling is performed in the nose and mouth region of the frame image, for example, a translucent red layer covering the middle nose to the center of the face is generated, and the comparison and labeling process is repeated for all frames to obtain the thermal response layer of the obstructed image.

[0117] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in other forms. Any skilled person in the art can modify or change the disclosed technical content into equivalent embodiments with equivalent changes, and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solution content of the present application, still falls within the protection scope of the present application.

Claims

1.A face recognition monitoring system based on PaddleHub, characterized in that, The system comprises: The image reflection tracking module acquires a sequence of continuous image frames in a dynamic monitoring picture, locates a face image region in each frame, calls average brightness values of corresponding pixel blocks of face key points in the current frame and the previous frame, identifies a dynamic trend of reflectivity of the face in the current frame sequence over time, and obtains a sequence of brightness change stability indexes; The trusted region screening module calls a key point set in an eye and a cheekbone region based on the sequence of brightness change stability indexes, groups mapping points according to a Dlib standard face region division rule, and obtains a stable region annotation layer; The dynamic score generation module acquires a horizontal and vertical distribution coordinate set in a Dlib face coordinate system based on the stable region annotation layer, classifies the face region visual score threshold, selects image frames with a score result exceeding the threshold, and obtains a set of qualified image frames. The covering state judgment module calls a mask recognition unit integrated by PaddleHub to identify a mask wearing type based on the set of qualified image frames, intercepts a nose region and a mouth region boundary image block, and calculates three types of boundary feature data, including an edge direction distribution, a pixel gradient density, and a brightness section number, to obtain a set of shielding behavior labels. 2.The PaddleHub-based face recognition monitoring system of claim 1, wherein, The sequence of brightness change stability indexes comprises a time sequence brightness change value set, a weighted average difference value sequence, a cross-frame standard deviation average value, and a region brightness fluctuation label, the stable region annotation layer comprises a face region grouping number picture, a trusted region position mask, a region pixel Boolean distribution table, and a key point mapping picture, the set of qualified image frames comprises a score judgment image index number, a face region distribution density value, a trusted region area proportion value, and a set of score passing flag bits, and the set of shielding behavior labels comprises an image frame shielding category label, a mouth and nose region edge continuity level, a mask wearing state label, and a shielding region classification index. 3.The face recognition monitoring system based on PaddleHub of claim 2, characterized in that, The image reflection tracking module comprises: An image frame acquisition submodule acquires a sequence of continuous image frames in a dynamic monitoring picture, extracts a face region in each image frame, and respectively locates four groups of key points, including an eye corner, a nose bridge midpoint, a brow, and a cheekbone edge, in each image frame, extracts average brightness values of pixel blocks corresponding to the key points, records the average brightness values as a cross-frame sequence, and generates a sequence of average brightness values of the key points; A brightness difference calculation submodule calculates a difference value between brightness values of the same key point in a current frame and a previous frame according to the sequence of average brightness values of the key points, constructs a difference value sequence set in a time sequence, calls a set time window length parameter, performs weighted processing on the difference value sequence, calculates a weighted mean value sequence according to a weighting coefficient, and acquires a weighted brightness change sequence; A time sequence fluctuation extraction submodule respectively calculates standard deviations of change values of each group of key points in the weighted brightness change sequence, calculates an average value of standard deviations of all key points, sets the average value as a time sequence fluctuation amount of a full point set, establishes an index set representing a consistency change trend of reflectivity of the face on a time axis, and obtains the sequence of brightness change stability indexes. 4.The face recognition monitoring system based on PaddleHub of claim 3, characterized in that, The weighting coefficient setting mode in the weighting process of the difference value sequence is that the luminance difference value at each time point in the difference value sequence is distributed in time sequence, and the difference value closer to the current frame time position is given a higher weight, and the difference value farther away is given a lower weight; The weighting coefficient is calculated by a normalized ratio of the reverse index value of the time point relative to the end position of the frame sequence and the length of the difference value sequence, and the sum of the weighting coefficients is limited to a constant one after normalization; The generation mode of the weighted mean value sequence is that the weighting coefficient and the corresponding luminance difference value are multiplied, and all the product values are weighted and summed, and the weighted sum is taken as the weighted luminance change value corresponding to the current frame position. 5.The face recognition monitoring system based on PaddleHub according to claim 4, wherein, The trusted area screening module comprises: The feature index comparison submodule compares the index value corresponding to each key point with the set reflection stability standard deviation threshold based on the luminance change stability index sequence, marks the key points with index values lower than the reflection stability standard deviation threshold as trusted points, and obtains a set of regional trusted key points; The key point screening and mapping submodule locates the spatial position coordinates of the trusted key points in the image frame based on the set of regional trusted key points, constructs a trusted area graphical mask in units of pixel areas, sets a regional pixel minimum coverage area benchmark, removes the scattered point areas that do not meet the regional pixel minimum coverage area benchmark, and generates trusted area image mask data; The regional structure grouping submodule obtains a structure area label based on the Dlib face region structure division rule by calling the trusted area image mask data, detects the consistency of pixel labels in the label corresponding area, classifies the labeled connected regions by structure, establishes structure area grouping according to the structure position number, and obtains a stable area label layer. 6.The face recognition monitoring system based on PaddleHub of claim 5, characterized in that, The dynamic score generation module comprises: The area proportion calculation submodule extracts the number of pixels in the labeled area and the total amount of effective pixels in the entire face image based on the stable area label layer, calculates the area ratio between the two and records it as the pixel area proportion, and obtains the face trusted area area proportion value; The density parameter extraction submodule obtains the coordinate set of the labeled area key points in the horizontal and vertical directions in the Dlib standard coordinate system by calling the face trusted area area proportion value, respectively counts the number of key points in each direction, and calculates the direction dispersion based on the coordinate mean value and the deviation value of each point to obtain the face region visibility score value; The threshold screening and classification submodule compares the face region visibility score value of the image frame by frame according to the face region visibility score value, calls the face region visibility score threshold, screens the image frame number with a face region visibility score value greater than the face region visibility score threshold, and obtains a set of qualified image frames. 7.The PaddleHub-based face recognition monitoring system of claim 6, characterized in that, The covering state determination module comprises: The face structure extraction submodule calls a PaddleHub face detection unit to extract the face frame position and five facial feature point coordinates in each image based on the visual score qualified image frame set, locates the nose area and mouth area boundary image block, and simultaneously calls a PaddleHub mask recognition unit to obtain the mask wearing type classification label of the corresponding face in the current image, the type classification standard being no wearing, partial covering and complete covering, establishes a mapping data set of the corresponding frame number and the three types of wearing labels, and obtains a wearing structure recognition result set; The boundary feature calculation submodule extracts the boundary image block and the surrounding image block corresponding to the nose area and the mouth area in each image frame according to the wearing structure recognition result set, respectively performs edge direction gradient statistics, pixel gray gradient histogram density extraction and continuous brightness section number counting on the image block area, performs numerical difference and ratio conversion according to the feature indexes of the central area and the surrounding area, obtains a boundary definition deviation value, and obtains a nose and mouth area boundary feature parameter group; The occlusion label generation submodule jointly processes the wearing structure recognition result set in the corresponding image frame based on the nose and mouth area boundary feature parameter group, adds the three types of boundary feature indexes after weight normalization, calculates a fusion occlusion score value, and performs classification according to a set boundary occlusion judgment reference value, delimits the occlusion behavior level of the corresponding image frame, establishes a corresponding relationship between the frame number and the occlusion label, and obtains an occlusion behavior label set. 8.The face recognition monitoring system based on PaddleHub of claim 7, characterized in that, The system further comprises: The heat map auxiliary verification module calls the upper nose area, the nose to the philtrum area and the philtrum to the lower jaw area pixel data, calculates the average value of the infrared signal value of each area pixel, and obtains a three-region heat intensity mean value group based on the occlusion behavior label set. The occlusion image heat response layer comprises a heat occlusion area temperature difference map, an infrared signal partition response table, a heat feature abnormality marking map and a regional heat coverage state map. 9.The PaddleHub-based face recognition monitoring system of claim 8, characterized in that, The heat map auxiliary verification module comprises: The regional heat value extraction submodule extracts the image frame number labeled as partial covering and complete covering based on the occlusion behavior label set, obtains the corresponding frame heat imaging image, and intercepts the pixel data of the upper nose area, the nose to the philtrum area and the philtrum to the lower jaw area, calculates the average value of the infrared signal value of each area pixel, and obtains a three-region heat intensity mean value group. The temperature difference feature calculation submodule calculates the relative change range of the heat value of the philtrum area and the upper nose area and the lower jaw area by calling the three-region heat intensity mean value group, extracts the upper nose area heat intensity mean value, the lower jaw area heat intensity mean value and the nose to the philtrum area heat intensity mean value, and obtains a heat change fusion coefficient by using the formula: ； The heat response layer generation submodule compares the heat change fusion coefficient of each image frame with the heat occlusion recognition temperature difference threshold value one by one by calling the heat occlusion recognition temperature difference threshold value according to the heat change fusion coefficient, screens the image frame number whose heat change fusion coefficient is greater than the heat occlusion recognition temperature difference threshold value, and labels the occlusion area, to obtain the occlusion image heat response layer. wherein, is the thermal change fusion coefficient, represents the mean infrared thermal intensity of the upper nasal region, represents the mean infrared thermal intensity of the mandibular region, represents the mean infrared thermal intensity of the mid-nasal to pronasale region, represents the additive denominator stabilizing factor; ​

Images