Low-quality video stream-oriented real-time tracking face recognition method and low-quality video stream-oriented real-time tracking face recognition system

Abstract

The invention discloses a low-quality video stream-oriented real-time face tracking recognition method and system. The method comprises the following steps of: obtaining a preliminary detection event and a detection frame; the detection frame is used for positioning face information, and the preliminary detection event represents that a face is recognized; calculating the overlapping degree of the appearance similarity and the intersection-to-union ratio; wherein the appearance similarity is calculated based on the distance between the feature vectors of the face information; the intersection-union ratio is used for evaluating the overlapping degree of the two detection frames; generating a similarity matrix based on the overlapping degree of the appearance similarity and the intersection-to-union ratio; and determining a tracking target matched with the detection frame based on the similarity matrix. By enhancing the quality of the input image, enhancing a tracking mechanism and a tracking process self-correction mechanism, the reliability problem of the method under the condition is improved, high-robustness detection is realized, and the accuracy rate and recall rate of events are improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a real-time tracking face recognition method and system for low-quality video streams. Background Art

[0002] With the development of deep learning technology, especially the popularization of convolutional neural networks, the accuracy of face recognition has been greatly improved. However, due to equipment limitations, there are still many low-quality videos in practice. Low-quality video refers to image data with low resolution, frequent brightness changes, and a lot of background interference. This type of video is common in surveillance videos shot by low-end camera equipment, and has problems such as unclear facial features, blurred motion, and insufficient light. Definition of low-quality video:

[0003] Let V be a video sequence, containing K frames, V t is the tth frame. The following quality indicators are defined:

[0004] Resolution R: R = W × H, where W and H are the width and height of the frame respectively;

[0005] Brightness change ΔL: ΔL t =|L(V t )-L(V t+1 )|, average brightness change:

[0006] Noise level N: Where P i is the pixel in the video frame, is the average pixel value of the whole frame, and M is the total number of pixels;

[0007] Motion blur B: Blur(V t ) is a function that measures the blurriness of the tth frame, such as the result of Laplace transform.

[0008] Let Q be the video quality index, which is defined as follows:

[0009] Q=α·R+β·Δ - L+γ·N+δ·B

[0010] Among them, α, β, γ, δ are weight coefficients used to adjust the contribution of each indicator to the overall quality. The conditions under which a video can be considered low quality can be expressed as:

[0011] Q>Q th

[0012] Where Q th is a preset threshold used to distinguish low-quality and high-quality videos.

[0013] In existing monitoring solutions, face detection for real-time video streams is often based on single-frame images, with little consideration of the continuity and correlation between video frames. The face detection frame is obtained through deep learning model prediction, and then the face in the detection frame is matched with the face in the database to determine whether the target person appears in the scene. This approach has some problems when applied to low-quality video streams captured by cameras.

[0014] First, existing deep learning models still have difficulty in handling face recognition or face matching tasks on images with obvious distortion. Especially when the distance from the camera is far, the features of the face will be blurred, which seriously affects the accuracy, precision and recall rate of model recognition.

[0015] Secondly, the recognition and matching based on a single frame image does not fully explore and utilize the correlation between the previous and subsequent frames in the video stream. Processing each frame image in isolation will lead to the previous detection results of the model not being fully utilized. It is even easy to cause the recognition event of the same person to be mistaken for the appearance of multiple different people.

[0016] Finally, most solutions that rely on deep learning will inevitably suffer from insufficient generalization when applied to actual special scenarios such as construction sites and warehouses. A mechanism is needed to adaptively adjust the algorithm detection results to improve the robustness of the method.

[0017] In the case of low video quality, when using existing face recognition models for migration, a large number of misidentification and missed recognition problems are often inevitable, affecting user experience. The effect of face recognition and tracking using existing models alone still needs to be further improved. Therefore, a technology that can reduce the false positive and missed negative rates of the method is needed to improve the accuracy of real-time tracking face recognition in low-quality video streams. Summary of the invention

[0018] The present invention provides a real-time tracking face recognition method and system for low-quality video streams, which are used to solve the problem of missed reports and high false alarm rates in the prior art when facing low-quality videos, thereby affecting user experience.

[0019] A real-time tracking face recognition method for low-quality video streams, the method comprising:

[0020] Obtaining a preliminary detection event and a detection frame; the detection frame is used to locate face information, and the preliminary detection event indicates that a face has been recognized;

[0021] Calculating the degree of overlap between the appearance similarity and the intersection-over-union ratio; wherein the appearance similarity is calculated based on the distance between the feature vectors of the face information; and the intersection-over-union ratio is used to evaluate the degree of overlap between the two detection frames;

[0022] Generate a similarity matrix based on the overlap between the appearance similarity and the intersection-over-union ratio;

[0023] A tracking target that matches the detection frame is determined based on the similarity matrix.

[0024] Optionally, before obtaining the preliminary detection event and the detection frame, the method includes:

[0025] Initialize a face detection algorithm model and a recorder; the recorder is used to record and maintain face information appearing in the video image; the face detection algorithm model includes: a face detection model and a face matching model;

[0026] Loading the model parameters of the face detection model and the face feature database; the face feature database records known face information;

[0027] Acquire the video image and locate the face information and key points of the face in the video image through the face detection model; wherein locating the face information in the video image comprises: inputting the video image and preprocessing the video image;

[0028] Extracting a feature map from the preprocessed video image, and generating and fusing a first multi-scale feature map through a feature pyramid network;

[0029] Generate a classification score and a bounding box regression offset for each anchor point by convolution on the first multi-scale feature map, and generate a candidate bounding box using the anchor box and the bounding box regression offset;

[0030] The detection frame is selected from the candidate frames by non-maximum suppression, and face information is located in the detection frame to generate a preliminary detection event.

[0031] Optionally, after determining the tracking target matching the detection frame based on the similarity matrix, the method further includes:

[0032] Update the matching target to the tracking target that matches the detection frame;

[0033] Update target status;

[0034] The tracking target that does not match the detection box is treated as a new target and added to the tracking list.

[0035] Optionally, after updating the matching target to a tracking target that matches the detection frame, the method further includes:

[0036] resetting a counter of the recorder;

[0037] When the tracking is finished, the counter of the recorder is incremented by one; if the counter exceeds the counting threshold, the tracking record of the face information is deleted.

[0038] Optionally, the method further includes:

[0039] Based on BC=ω c ·C+ω P ·P+ω KP KP calculates the trust index BC;

[0040] Where C is the confidence of the face matching model output, P is the quality index of the detection frame, ω c ,ω p ,ω KP are three weight coefficients, which are used to indicate the contribution of the three factors to the trust index BC, and KP is the standardized value of the number of key points;

[0041] In the process of detecting facial features in each frame, the confidence index BC of each detection result is calculated, and the detection result with the maximum confidence index BC is selected as the identity ID of the tracking target in the whole process. track :

[0042] ID track = argmax ID (BC ID ).

[0043] Optionally, the method further includes:

[0044] Outputting the offset of the key point position of the face based on the convolution layer parameters of the key point detection and the first multi-scale feature map to obtain the key points and key point positions of the face;

[0045] A standardized value of the number of key points is determined based on the key points and key point positions of the face.

[0046] Optionally, the method further includes:

[0047] Determining the number of facial information located in the detection frame;

[0048] If the number of faces is 0, continue to recognize the next frame of the video image.

[0049] Optionally, the method further includes:

[0050] Extracting a second multi-scale feature map of the video image using a multi-scale feature extraction module;

[0051] Performing feature fusion and enhancement on the second multi-scale feature map to obtain enhanced features;

[0052] The enhanced features are passed through an upsampling and reconstruction module to generate an enhanced output image to perform image enhancement on the extracted facial information.

[0053] The present invention also provides a real-time tracking face recognition system for low-quality video streams, the system comprising:

[0054] An acquisition unit, used to acquire a preliminary detection event and a detection frame; the detection frame is used to locate face information, and the preliminary detection event indicates that a face has been recognized;

[0055] A calculation unit, used to calculate the degree of overlap between the appearance similarity and the intersection-over-union ratio; wherein the appearance similarity is calculated based on the distance between the feature vectors of the face information; and the intersection-over-union ratio is used to evaluate the degree of overlap between the two detection frames;

[0056] A matrix generating unit, used for generating a similarity matrix based on the overlap degree of the appearance similarity and the intersection-over-union ratio;

[0057] A matching unit is used to determine a tracking target that matches the detection frame based on the similarity matrix.

[0058] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any of the methods described above.

[0059] The present invention improves the reliability problem of the method in this case by enhancing the quality of input images, strengthening the tracking mechanism and the self-correction mechanism of the tracking process, achieving high-robustness detection, and improving the precision and recall rate of events. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of a real-time tracking face recognition method for low-quality video streams in an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of a facial feature database in an embodiment of the present invention;

[0062] Figure 3 This is a flow chart of a real-time tracking face recognition method for low-quality video streams in an embodiment of the present invention;

[0063] Figure 4 This is a structural diagram of a real-time tracking face recognition system for low-quality video streams in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0065] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0066] An embodiment of the present invention provides a real-time tracking face recognition method for low-quality video streams, the method comprising:

[0067] Step 100, obtaining a preliminary detection event and a detection frame; the detection frame is used to locate face information, and the preliminary detection event indicates that a face has been recognized;

[0068] Step 200, calculating the degree of overlap between the appearance similarity and the intersection-over-union ratio; wherein the appearance similarity is calculated based on the distance between the feature vectors of the face information; and the intersection-over-union ratio is used to evaluate the degree of overlap between the two detection frames;

[0069] Step 300, generating a similarity matrix based on the overlap degree of the appearance similarity and the intersection-over-union ratio;

[0070] Step 400: Determine a tracking target that matches the detection frame based on the similarity matrix.

[0071] In a preferred embodiment, before obtaining the preliminary detection event and the detection frame, the method further includes:

[0072] Initialize the face detection algorithm model and the recorder; the recorder is used to record and maintain the face information appearing in the video image; the face detection algorithm model includes: a face detection model and a face matching model. Specifically, the face detection model includes a Retinaface model. The embodiment of the present invention combines the Retinaface and Arcface models for face positioning and matching, and introduces the MIRNetv2 image enhancement model to improve image quality. By using this image enhancement model, the contrast, brightness and other characteristics of low-quality video images are significantly improved, thereby reducing the overall difficulty in face recognition tasks and solving the impact of poor video quality on recognition effects.

[0073] The model parameters of the face detection algorithm model and the face feature database are loaded; the face feature database records known face information. Specifically, the model parameters θ of the face detection algorithm model are loaded Retina ,θArc ,θ MIR :

[0074] θ Retina ←Load(RetinaFace)

[0075] θ Arc ←Load(ArcFace)

[0076] θ MIR ←Load(MIRv2)

[0077] The facial feature database can refer to Figure 2 As shown, the face feature database records known face information in advance, different face information corresponds to different ID numbers, and the face detection model is used to pre-identify to obtain the corresponding 128-dimensional face features, which are then saved in the face feature database.

[0078] The video image is acquired and the face information in the video image is located by using the face detection model. Specifically, a video stream I(t) of the video image is acquired from a camera in real time in a multi-threaded manner through an OpenCV video interface, and the Retinaface face detection model is used to detect and locate the face information in the video stream.

[0079] Based on the face feature database, face detection is performed on the face information and the recorder is updated. Specifically, the recorder is an important component in the embodiment of the present invention, which uses the single-frame recognition result of the face detection model to maintain and generate records of all face information that has existed in the scene. The recorder includes: a dictionary D that saves the record R, and maintains a counter C for each record R in the dictionary D, and uses a tracking algorithm Tracker to track the face information. The format of each record R is:

[0080] R={ID,C,S}

[0081] Among them, ID is the target that the face information currently matches in the face feature database, or other targets UNKNOWN that do not exist in the face feature database, and S is the face matching model ArcFace, which is the matching similarity output during the face detection process. By introducing a recorder to maintain all face information that appears in the video image, and by comparing the long-term information in the face feature database, the problem of false detection and missed detection is automatically corrected. This improvement effectively addresses the problem of face matching in multi-target scenarios and improves the accuracy of the system.

[0082] The method described in the embodiment of the present invention improves the effectiveness and robustness of algorithm recording and enhances user experience by combining multiple deep learning models and utilizing mature tracking algorithms.

[0083] The real-time tracking face recognition method for low-quality video streams described in the specific embodiment of the present invention preferably further comprises: after acquiring the video image:

[0084] The frame image of the video image is constructed according to the algorithm's region of interest for subsequent editing and processing. Specifically, the frame image I of the video image I(t) is constructed according to the algorithm's region of interest (ROI). ROI (t):

[0085] I ROI (t) = ROI(I(t))

[0086] In the real-time tracking face recognition method for low-quality video streams described in a specific embodiment of the present invention, preferably, locating the face information in the video image includes:

[0087] Input the video image and pre-process the video image. Specifically, input the frame image I of the video image ROI And adjust it to a specified size, for example: H×W, and perform normalization processing to meet the input requirements of the model. Let the preprocessed video image be I norm :

[0088] I norm =Normalize(Resize(I ROI ,H,W))

[0089] Wherein, H and W are the height and width of the frame image respectively.

[0090] A feature map is extracted from the preprocessed video image, and a first multi-scale feature map is generated and fused through a feature pyramid network (FPN). Specifically, the RetinaFace model uses a backbone network based on ResNet to process the processed video image I norm Perform feature extraction to obtain the extracted feature map F Ret :

[0091] F Ret =Backbone(I norm θ backbone )

[0092] where θ backbone are the parameters of the backbone network.

[0093] Use the feature pyramid network to generate the first multi-scale feature map {F s} to adapt to face detection of different sizes:

[0094] {F S =FPN(F Ret θ FPN ),s∈{P3,P4,P5}

[0095] Among them, s represents the feature map of P3, P4, P5 scale, θ FPN is the parameter of FPN.

[0096] Anchor boxes are a set of predefined frames used in object detection tasks. Anchor boxes correspond to different feature scales of the input image and are used to predict the bounding box of the target. Anchor boxes allow the model to make predictions at different scales and aspect ratios. The input image is divided into a grid, and each cell of the grid corresponds to multiple anchor boxes. The classification score and bounding box regression offset of each anchor point are generated by convolution on the first multi-scale feature map, and the classification score and bounding box regression offset are used to generate candidate bounding boxes. Specifically, in the first multi-scale feature map {F s Each feature map F in s The RetinaFace model generates the classification score p for each anchor through the convolution layer (Conv) cls and the bounding box regression offset Δb reg :

[0097] p cls =Sigmoid(Conv cls (F s θ cls ))

[0098] Δb reg =Conv reg (F s θ reg )

[0099] where θ cls and θ reg They are respectively the classification convolution layer Conv cls And regression convolution layer Conv reg Parameters. Using anchor boxes Anchors and bounding box regression offset Δb reg , recalculate the position and width and height of the box to generate a candidate border B candidate :

[0100] B candidate =Anchor s +Δb reg

[0101] A detection frame is selected from the candidate frames by non-maximum suppression, and the face information is located in the detection frame. Specifically, in order to eliminate redundant detection frames, non-maximum suppression (NMS) is used to select the detection frame B:

[0102] B = NMS(B candidate ,p cls ,IoU threshold )

[0103] Where IoU threshold It is the intersection-over-union (IOU) threshold, usually 0.5.

[0104] Preferably, in addition to face detection, RetinaFace also outputs the locations of n key points of the face The offset Δl kp , to assist in face matching or other tasks such as expression analysis:

[0105] Δl kp =Conv kp (F s θ kp )

[0106] where θ kp It is the convolution layer Conv for key point detection kp Finally, RetinaFace outputs the detection box B after NMS and the corresponding key point positions.

[0107] The real-time tracking face recognition method for low-quality video streams described in the specific embodiment of the present invention preferably further comprises: after locating the face information in the detection frame:

[0108] Determine the number of facial information located in the detection frame;

[0109] If the number of faces is 0, continue to recognize the next frame of the video image. Specifically, when the number of faces is 0, directly update the recorder and continue to recognize the next frame of the video image. The specific recognition scheme is the same as the above specific embodiment.

[0110] The real-time tracking face recognition method for low-quality video streams described in the specific embodiment of the present invention preferably further includes:

[0111] The multi-scale feature extraction module is used to extract the second multi-scale feature map of the video image. The MIRNetv2 model is used to perform noise reduction, brightness enhancement, contrast enhancement and other operations on the obtained video image B(t) to improve the image quality. The main steps are as follows: the multi-scale feature extraction module MultiScaleFeaturtor is used to extract the second multi-scale feature map F of the input video image. MIR :

[0112] F MIR =MultiScaleFeatureExtractor(I norm θ MSFE )

[0113] where θ MSFE are the parameters of the multi-scale feature extraction module.

[0114] The second multi-scale feature map is subjected to feature fusion and enhancement to obtain enhanced features. MIR Perform feature fusion and enhancement to obtain enhanced feature F enhanced :

[0115] F enhanced =FeatureEnhance(F MIR θ FE )

[0116] where θ FE It is the parameter of the feature enhancement module FeatureEnhance.

[0117] The enhanced features are passed through the upsampling and reconstruction modules to generate an enhanced output image for the entire image area. enhanced Through upsampling and reconstruction module Reconstruction, the enhanced output image I is generated enhanced :

[0118] I enhanced =Reconstruction(F enhanced θ rec )

[0119] where θ rec is the parameter of the reconstruction module Reconstruction. Finally, the output enhanced output image I enhanced On this image, properties such as brightness ΔL, noise level N, and blur B are all improved in a targeted manner.

[0120] The real-time tracking face recognition method for low-quality video streams described in the specific embodiment of the present invention preferably performs face detection on the face information based on the face feature database, including:

[0121] Extracting facial features through the face matching model;

[0122] The facial features are matched with known facial information recorded in the facial feature database.

[0123] Specifically, the face matching model Arcface is used to extract facial features f i (t):

[0124] f i (t) = ArcFeatureExtractor(I enhanced θ AFE )

[0125] where θ AFE is the parameter of the feature extraction network ArcFeatureExtractor, f i (t) is a 128-dimensional feature vector. i (t) Match the facial features in the facial feature database. Matching similarity S i (t) is calculated as:

[0126]

[0127] Among them, ||f i (t)||2-norm of the pointing quantity.

[0128] Set a similarity threshold τ, if the similarity S i (t) If the similarity threshold is exceeded, the match is considered successful, and the matching result is the face target with the maximum similarity. Output the ID of all faces in the face feature database, the detection frame B of each face, and the maximum similarity S corresponding to each face in this frame of video image.

[0129] The real-time tracking face recognition method for low-quality video streams described in the specific embodiment of the present invention preferably further includes:

[0130] Tracking and updating facial information of a person to the recorder through the recorder;

[0131] Comparing the appearance similarity and intersection-over-union overlap between the face information located by the detection frame and the face information tracked in the record dictionary of the recorder to determine whether the face information belongs to a face recorded by the recorder;

[0132] When the face matching model is matched successfully, resetting the counter of the recorder;

[0133] When the tracking is finished, the counter of the recorder is incremented by one; if the counter exceeds the counting threshold, the tracking record of the face information is deleted.

[0134] It should be noted that even if the quality of the input image is improved as described above, the detection results of the model may not be completely reliable due to the generalization problem of the model itself. In the specific embodiment of the present invention, the Deepsort tracking algorithm is innovatively combined to utilize the results of face matching and improve the reliability of the tracking algorithm. Specifically, when each new detection event is generated, for all the incoming detection boxes B i (i∈{0,1…,N}, N is the total number of detection boxes), calculate the appearance similarity and the overlap of the intersection-over-union ratio (IOU). The appearance similarity is calculated based on the distance between feature vectors, and the cosine similarity is used to measure the similarity of appearance features:

[0135]

[0136] Among them, f i For the initial detection event, f i R Represents the appearance features of all records R in the dictionary D, with dimensions d, f i [k] and f i R [k] represents the kth component of the feature vector, where the output features of ArcFace can be used directly.

[0137] IOU is a common metric for evaluating the degree of overlap between two bounding boxes, and the formula is as follows:

[0138]

[0139] Among them, R i Area(R i ∩B i ) is the area of ​​the overlap of the two boxes, Area(R i ∪B i ) is the area of ​​the union of the two boxes.

[0140] Combining the appearance similarity and IOU overlap indicators, a similarity matrix SM is generated, where the element SM[i][j] represents the detection box B i and the current target R j The comprehensive similarity of:

[0141] SM[i][j]=α a ·Sim a +βIOU IOU(R j ,B i )

[0142] where α a , β IOU They represent the weights of appearance similarity and IOU overlap indicators in the score respectively.

[0143] Subsequently, the Hungarian algorithm is used to solve the problem of minimizing the cost matching. By maximizing the similarity matrix SM, a matching tracking target is found for each detection box, which can be used as the best matching tracking target, and can be used as the minimum cost match. Once the minimum cost match is found, the algorithm returns an index of the matching pair. Specifically, for each detection box B i , find the matching tracking target R j , and record this matching relationship.

[0144] Further processing is performed based on the matching result: first, the matching target is updated to the tracking target that matches the detection frame. Specifically, if the matching similarity is greater than the set threshold T, the detection frame B is updated. i and tracking target R j Association, update the state of the target (such as position and appearance features). Then deal with the unmatched situation. If a detection box does not find the corresponding tracking target, it is regarded as a new target and added to the tracking list.

[0145] For the successfully matched record R, the counter T is reset to 0. After the tracking process is completed, the counter T of all records R is increased by one. If the counter of a record exceeds a certain threshold, it is considered that the character has left the scene, and the record r is deleted from D, and a record is triggered for the entire event about the character.

[0146] In addition, in order to deal with the situation where distant targets in low-quality video frames are difficult to successfully match with clear facial features already in the database, thereby resulting in a large number of possible recognition failure records, the present invention continuously corrects the tracking identity of a single person by adopting the highly reliable results output by the face matching model.

[0147] Specifically, a trust index BC is introduced and expressed as:

[0148] BC=ω c ·C+ω P ·P+ω KP ·KP

[0149] Where: C is the confidence output by the face matching model, ranging from [0,1]. P is the quality index of the detection frame, which is calculated by the position and size of the detection frame. Let the detection frame be represented as (x, y, w, h), where (x, y) represents the position of the upper left corner of the frame, and w and h represent the width and height of the detection frame respectively. Calculate the minimum distance from the edge of the picture:

[0150] d edge =min(x,W-(x+w),y,H-(y+h))

[0151] Define the area of ​​the detection box as A = W × H, and use an exponential decay function to define P:

[0152]

[0153] Among them, α edge and β A is a parameter that controls the decay rate, which is used for the influence of distance and area respectively. P can reduce the confidence when it is close to the edge of the picture, while considering the influence of the size of the detection box.

[0154] KP is the standardized value of the number of key points. If the detection result has n key points, then:

[0155]

[0156] Where N max is the maximum number of key points of this category of faces.

[0157] ω c ,ω p ,ω KP is the weight coefficient, which indicates the contribution of the three factors to BC.

[0158] In each frame F t During the detection process, the BC of each detection result is calculated. Then, the detection result with the largest BC is selected as the identity ID of the tracking target in the whole process. track :

[0159] ID track = argmax ID (BC ID )

[0160] By introducing the trust index BC, we can comprehensively evaluate the credibility of the detection results and then select the most credible identity as the identity of the tracking target. This method can effectively improve the reliability of face recognition in complex scenarios.

[0161] For a record R that already exists in D, if S in a certain detection result is greater than the matching similarity in the record, then update S corresponding to the record and the identity ID of the tracking target track， That is, the corresponding face.

[0162] like Figure 3 As shown in the figure, in the process of processing multiple consecutive frames, the face moves from the far left side of the monitoring area to the near right side. Since the tracker tracks the person according to each recognition result and updates and maintains the record in the record dictionary D, the long-distance misdetection can be automatically corrected as the target is recognized with high confidence at a close distance, thereby avoiding false alarms in this case. When processing low-quality video streams captured by cameras in construction site monitoring, this method can well improve the recognition of complete events in poor lighting conditions and blurred faces.

[0163] The embodiment of the present invention also provides a real-time tracking face recognition system for low-quality video streams, such as Figure 4 As shown, the system comprises:

[0164] An acquisition unit 401 is used to acquire a preliminary detection event and a detection frame; the detection frame is used to locate face information, and the preliminary detection event indicates that a face has been recognized;

[0165] A calculation unit 402 is used to calculate the degree of overlap between the appearance similarity and the intersection-over-union ratio; wherein the appearance similarity is calculated based on the distance between the feature vectors of the face information; and the intersection-over-union ratio is used to evaluate the degree of overlap between the two detection frames;

[0166] A matrix generating unit 403, configured to generate a similarity matrix based on the overlap between the appearance similarity and the intersection-over-union ratio;

[0167] The matching unit 404 is configured to determine a tracking target that matches the detection frame based on the similarity matrix.

[0168] A specific embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any of the methods described above.

[0169] The embodiments of the present invention have the following beneficial effects:

[0170] 1. Based on events where people appear in the entire scene rather than isolated single-frame images, the output of the deep learning model is fully utilized to improve the accuracy of recognition records and reduce redundant reporting.

[0171] 2. The introduction of the recorder makes full use of the relationship between adjacent frames, and the long-term complete tracking makes up for the defect of limited recognition accuracy of deep learning models at long distances. The tracking combined with facial appearance features ensures the accuracy of tracking.

[0172] 3. The introduced model for improving image quality further improves the recognition performance and robustness of this method. The MIRNetv2 model improves the brightness ΔL, noise level N, and blur B of the obtained image through a variety of enhancement methods, thereby specifically reducing the difficulty of face recognition and matching, further adapting to the input distribution of the face recognition and matching model, and improving the detection effect.

[0173] 4. The introduction of a self-correction mechanism avoids the situation where the deep learning model causes false positives due to defects in the model itself during the processing of actual low-quality videos. Combined with the trust index BC, it comprehensively considers multiple factors such as face appearance, face position, and output confidence, and realizes the dynamic adjustment of the tracking mechanism to the identity of the tracked target, further ensuring the accuracy of tracking.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A real-time tracking face recognition method for low-quality video streams, characterized in that: The method comprises: Obtaining a preliminary detection event and a detection frame; the detection frame is used to locate face information, and the preliminary detection event indicates that a face has been recognized; Calculating the degree of overlap between the appearance similarity and the intersection-over-union ratio; wherein the appearance similarity is calculated based on the distance between the feature vectors of the face information; and the intersection-over-union ratio is used to evaluate the degree of overlap between the two detection frames; Generate a similarity matrix based on the overlap between the appearance similarity and the intersection-over-union ratio; A tracking target that matches the detection frame is determined based on the similarity matrix.

2. The real-time tracking face recognition method for low-quality video streams according to claim 1 is characterized in that: Before obtaining the preliminary detection event and the detection frame, the method includes: Initialize a face detection algorithm model and a recorder; the recorder is used to record and maintain face information appearing in the video image; the face detection algorithm model includes: a face detection model and a face matching model; Loading the model parameters of the face detection model and the face feature database; the face feature database records known face information; Acquire the video image and locate the face information and key points of the face in the video image through the face detection model; wherein locating the face information in the video image comprises: inputting the video image and preprocessing the video image; Extracting a feature map from the preprocessed video image, and generating and fusing a first multi-scale feature map through a feature pyramid network; Generate a classification score and a bounding box regression offset for each anchor point by convolution on the first multi-scale feature map, and generate a candidate bounding box using the anchor box and the bounding box regression offset; The detection frame is selected from the candidate frames by non-maximum suppression, and face information is located in the detection frame to generate a preliminary detection event.

3. The real-time tracking face recognition method for low-quality video streams according to claim 2 is characterized in that: After determining the tracking target matching the detection frame based on the similarity matrix, the method further includes: Update the matching target to the tracking target that matches the detection frame; Update target status; The tracking target that does not match the detection box is treated as a new target and added to the tracking list.

4. The real-time tracking face recognition method for low-quality video streams according to claim 3 is characterized in that: After updating the matching target to a tracking target that matches the detection frame, the method further includes: resetting a counter of the recorder; When the tracking is finished, the counter of the recorder is incremented by one; if the counter exceeds the counting threshold, the tracking record of the face information is deleted.

5. The real-time tracking face recognition method for low-quality video streams according to claim 3 is characterized in that: The method further comprises: Based on BC=ω c ·C+ω P ·P+ω KP KP calculates the trust index BC; Where C is the confidence of the face matching model output, P is the quality index of the detection frame, ω c ,ω p ,ω KP are three weight coefficients, which are used to indicate the contribution of the three factors to the trust index BC, and KP is the standardized value of the number of key points; In the process of detecting facial features in each frame, the confidence index BC of each detection result is calculated, and the detection result with the maximum confidence index BC is selected as the identity ID of the tracking target in the whole process. track : ID track =argmax ID (BC ID )。 6. The real-time tracking face recognition method for low-quality video streams according to claim 5, characterized in that: The method further comprises: Outputting the offset of the key point position of the face based on the convolution layer parameters of the key point detection and the first multi-scale feature map to obtain the key points and key point positions of the face; A standardized value of the number of key points is determined based on the key points and key point positions of the face.

7. The real-time tracking face recognition method for low-quality video streams according to claim 3 is characterized in that: The method further comprises: Determining the number of facial information located in the detection frame; If the number of faces is 0, continue to recognize the next frame of the video image.

8. The real-time tracking face recognition method for low-quality video streams according to claim 2, characterized in that: The method further comprises: Extracting a second multi-scale feature map of the video image using a multi-scale feature extraction module; Performing feature fusion and enhancement on the second multi-scale feature map to obtain enhanced features; The enhanced features are passed through an upsampling and reconstruction module to generate an enhanced output image to perform image enhancement on the extracted facial information.

9. A real-time tracking face recognition system for low-quality video streams, characterized in that: The system comprises: An acquisition unit, used to acquire a preliminary detection event and a detection frame; the detection frame is used to locate face information, and the preliminary detection event indicates that a face has been recognized; A calculation unit, used to calculate the degree of overlap between the appearance similarity and the intersection-over-union ratio; wherein the appearance similarity is calculated based on the distance between the feature vectors of the face information; and the intersection-over-union ratio is used to evaluate the degree of overlap between the two detection frames; A matrix generating unit, used for generating a similarity matrix based on the overlap degree of the appearance similarity and the intersection-over-union ratio; A matching unit is used to determine a tracking target that matches the detection frame based on the similarity matrix.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 8.

Images