Method and system for personnel fine-grained tracking based on face recognition under surveillance video

Abstract

The personnel fine-grained tracking method and system based on face recognition in monitored video disclosed by the application determine the independent time periods of different personnel target frames in the monitored video, determine the type of the time period in which the video frame at the set time point is located, determine the identity recognition result of each independent time period under each personnel number based on the identity information of multiple set time point video frames, and solve the personnel identity error tracking problem caused by ID exchange of the target tracking algorithm in the fine-grained scene; in order to improve the personnel identity confirmation accuracy, the adaptive face recognition mechanism is proposed, whether the face recognition is suitable at this time is judged according to the standard, so as to solve the face mis-detection problem caused by the overlapping of personnel frames when the personnel are dense; meanwhile, based on the above improvement, the application can realize the forward and backward tracking of the identity, and when the face recognition operation in the early and late stages is difficult to implement, the personnel identity can also be obtained in the identity continuous tracking dictionary according to the time period.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a personnel fine-grained tracking method and system based on face recognition in a monitoring video. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] In recent years, with the vigorous development of computer vision technology and the in-depth study of machine learning, face recognition and target tracking, as two important branches, have also undergone major changes and innovations, and are widely used in various fields of life such as intelligent supervision, access control security, intelligent transportation, etc.

[0004] In the monitoring video, pedestrian re-identification or face recognition technology is often used to confirm the identity of personnel. Compared with outdoor environment, there are more opportunities for the monitoring camera to capture clear face pictures in indoor environment, so face recognition can be used as a technical support for identity recognition. The combination of face recognition and target tracking technology plays a crucial role in confirming the identity of personnel in indoor monitoring video and further understanding their movement trajectory. In the existing method, for each video frame, the personnel coordinate information and image are usually obtained by using a target detection algorithm, and then sent to a face recognition algorithm to calculate the similarity between the extracted face features and the features of the face library. If it exceeds a certain threshold, the identity is confirmed. When the personnel number generated in the target tracking is matched with the personnel identity, two methods are usually used. One is to store the results of face recognition for each number, and first determine whether the personnel identity has been identified when predicting the next frame. If it has not been identified, further face recognition is performed. This method can confirm the identity after one or more face recognitions. The second method takes into account that the result of one or a few face recognitions may be accidental and may result in recognition errors. Therefore, based on statistics, the face recognition algorithm is further improved. The target box picture with a face is sent to the face recognition algorithm, and the multiple face recognition results of each number are saved. After all video frames are processed, based on the above identity information, the number of times each name appears is counted, and the name with the most appearance times is selected as the identity identifier of the number.

[0005] However, due to the defects of the existing target tracking algorithm, for example, the ID exchange problem in the tracking process will interfere with the tracking of the personnel identity, and even cause the identity identification disorder. Based on the above method two, if the ID exchange problem occurs in the tracking process, there will be two or even more people matching under the same number, if the effective face identity information is saved before and after the exchange, then in the subsequent identity statistics stage, a small amount of real identity information in a certain time period will be ignored, so that the different personnel under this number will have the problem of identity identification error. Especially for the fine-grained crowd with the same clothes, similar body posture and poor distinguishability, it will inevitably have an adverse effect on the target tracking algorithm based on the similarity matching of the target frame according to the external characteristics, the ID exchange problem will be further intensified, and thus greater interference will be caused to the confirmation of the personnel identity.

[0006] In addition, in the target tracking process, the face recognition technology is used, the target frame is sent into the face recognition model, and the personnel identity of the current target frame can be obtained. However, when the personnel in the monitoring video is too dense, a large proportion of overlap will occur between the target frames output by the personnel recognition algorithm, and multiple faces may exist in a target frame, which will cause the face recognition algorithm to have a false detection, and the identity information irrelevant to the target frame will be assigned to the identity identification of the target frame. If no corresponding targeted processing is performed, and if the face of the personnel is detected only once or not detected in the tracking process, great interference will be caused to the subsequent identity statistics, and the problem of identity matching disorder will be caused. SUMMARY

[0007] In order to solve the above problems, the present application provides a fine-grained personnel tracking method based on face recognition under a monitoring video and a system, which realizes accurate personnel target tracking positioning and identity confirmation in the monitoring video.

[0008] To achieve the above object, the present application adopts the following technical scheme:

[0009] In the first aspect, a fine-grained personnel tracking method based on face recognition under a monitoring video is provided, comprising:

[0010] obtaining a monitoring video;

[0011] selecting a personnel target in the monitoring video through a target frame, and setting a separate personnel number for the target frame of each personnel target; tracking each target frame, and identifying the identity of the target frame of a set time point video frame in the tracking process to obtain the identity information of each target frame in the corresponding set time point video frame;

[0012] For each target frame, the intersection-over-union with the remaining target frames in each frame of the monitoring video is calculated, and the independent time period of the target frame is determined through the intersection-over-union;

[0013] judging a time period in which each set time point video frame is located;

[0014] For each personnel number, identity information of the set time point video frames located in the same independent time period is counted respectively, and the identity information with the most occurrences is selected as the identity recognition result of each personnel number in each independent time period.

[0015] The monitoring video is labeled by the identity recognition result to obtain a personnel tracking video.

[0016] In a second aspect, a personnel fine-grained tracking system based on face recognition under a monitoring video is provided, which comprises:

[0017] a monitoring video acquisition module, configured to acquire a monitoring video;

[0018] a target recognition and target tracking module, configured to frame a personnel target in the monitoring video by a target frame, and set a separate personnel number for the target frame of each personnel target; perform target tracking on each target frame, and perform identity recognition on the target frame of a set time point video frame in the tracking process to obtain identity information of each target frame in the corresponding set time point video frame.

[0019] a target frame independent time period identity recognition module, configured to calculate an intersection-over-union of each target frame with the remaining target frames in each frame of the monitoring video, determine an independent time period of the target frame by the intersection-over-union, judge a time period in which each set time point video frame is located, and count the identity information of the set time point video frames located in the same independent time period for each personnel number respectively, and select the identity information with the most occurrences as the identity recognition result of each personnel number in each independent time period.

[0020] a personnel tracking video acquisition module, configured to label the monitoring video by the identity recognition result to obtain a personnel tracking video.

[0021] In a third aspect, an electronic device is provided, which comprises a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the personnel fine-grained tracking method based on face recognition under a monitoring video are completed.

[0022] In a fourth aspect, a computer readable storage medium is provided, configured to store computer instructions, when the computer instructions are executed by a processor, the steps of the personnel fine-grained tracking method based on face recognition under a monitoring video are completed.

[0023] Compared with the prior art, the personnel fine-grained tracking method based on face recognition under a monitoring video has the following beneficial effects:

[0024] 1. The application determines the independent time period of the target frame according to the change of the intersection-over-union of the target frame in each frame when tracking the target of the personnel target, and selects the identity information with the most appearance times for each independent time period under each personnel number, so as to realize the identity correction before and after the ID exchange, ensure the accuracy of the target tracking, and solve the problem of personnel identity error tracking caused by the ID exchange of the target tracking algorithm in the fine-grained scene.

[0025] 2. When performing face recognition, the application uses the intersection-over-union between the target frames as an index for whether it is suitable for face recognition, and uses the matching degree of the face and the personnel in the target frame as an auxiliary to eliminate the pictures that interfere with the identity confirmation, so as to ensure the accuracy of the face recognition and solve the problem of face mis-detection caused by the overlapping of the target frames when the personnel are dense.

[0026] 3. Since the face difference between adjacent video frames is small, the application obtains the face image in the current video frame every set time to save the calculation cost, and only performs identity recognition on the target frame in the set time point video frame.

[0027] The advantages of the additional aspects of the application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings accompanying the specification of this application form a part of this application and serve to further understand the application, the illustrative embodiments of the application and their descriptions are used to explain the application, and do not constitute an improper limitation on the application.

[0029] Figure 1 The flowchart of the method disclosed in embodiment 1 is shown in the figure;

[0030] Figure 2 The independent time period partitioning diagram disclosed in embodiment 1 is shown in the figure;

[0031] Figure 3 The identity information of different personnel numbers classified and stored according to the time period disclosed in embodiment 1 is shown in the figure;

[0032] Figure 4 The identity continuous tracking correction method based on the time period disclosed in embodiment 1 is shown in the figure. DETAILED DESCRIPTION

[0033] The application will be further described below in combination with the drawings and embodiments.

[0034] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0035] First, the terms are explained:

[0036] 1. Angular Margin: It is a decision boundary commonly used in face recognition model loss. It normalizes the feature vector and weight, adds an angular interval m between them, and thus punishes the angle between the sample depth feature and its corresponding weight, further reducing the intra-class difference and increasing the inter-class difference.

[0037] 2. Hungarian Algorithm: It is an optimization algorithm in the field of mathematics that solves the optimal combination scheme in the allocation task, and can achieve the minimum weight optimal matching.

[0038] 3. IoU: The full name is Intersection over Union, mainly used in target detection, which calculates the intersection rate of "predicted box" and "real box", that is, the ratio of their intersection and union, which is a standard for measuring the accuracy of detecting corresponding objects in a specific data set.

[0039] 4. ID exchange: In an ideal case, a multi-target tracking algorithm assigns a unique personnel number to different targets when tracking them. However, in actual applications, due to factors such as target occlusion, blur, and density, tracking may not be continuous, and personnel numbers may change. There is a problem that the tracking algorithm will match the tracking trajectories, target boxes, and personnel numbers of two people incorrectly when they are in contact with each other, which is manifested as the exchange of personnel numbers before and after the contact.

[0040] Example 1

[0041] In this embodiment, a personnel fine-grained tracking method based on face recognition under surveillance video is disclosed, as shown in Figure 1 , which includes:

[0042] S1: Obtain a surveillance video.

[0043] Each surveillance video includes multiple video frames, and the earliest video frame in time is defined as the first frame.

[0044] This embodiment involves face recognition, which requires high-quality face images. It is recommended to use a camera with a resolution of 2K or higher to record video. Download the surveillance video in the required time period, and use the improved system to read the video frame by frame.

[0045] S2: frame the personnel target in the monitoring video by a target frame, and set a separate personnel number for the target frame of each personnel target; track the target for each target frame, and identify the target frame of the set time point in the tracking process to obtain the identity information of each target frame of the set time point.

[0046] Specifically, the personnel target in each frame is detected and identified by using the target detection algorithm to obtain the position coordinates of all personnel targets in each frame.

[0047] When the target frame of the personnel target in a frame cannot be matched with the personnel tracking trajectory before the frame, it is determined that the position of the personnel target in the frame is the initial position of the personnel target.

[0048] All personnel targets of the monitoring video include the personnel target appearing in the first frame and the new personnel target appearing in the subsequent frame, and the personnel target appears for the first time in a frame.

[0049] The position of the personnel target appearing for the first time in the monitoring video is the initial position of the personnel target.

[0050] The standard for judging whether a new personnel target appears in the subsequent frame is:

[0051] Match the target frame in a frame with the personnel tracking trajectory before the frame;

[0052] When no personnel tracking trajectory is matched with the target frame in the frame, the personnel target framed by the target frame is a new personnel target, and the framed position is the initial position of the new personnel target.

[0053] Set a separate personnel number for the target frame of each personnel target.

[0054] Use a target tracking algorithm to track the target for each target frame. Specifically, a multi-target tracking model for efficient data association in the target tracking algorithm is used to track the target for the target frame. For low-score frames, instead of being removed roughly, they are first temporarily saved, and the low-score frames are matched with the tracking trajectory by using the position coincidence degree of the target frame and the target trajectory, so that the objects that are blocked or blurred are excavated from the low-score frames, and the coherence of the tracking trajectory is ensured.

[0055] Before the multi-target tracking model tracks the target, the target detection algorithm is used to give the position of the target frame in the video frame and assign a score to the target frame. If the target feature is obvious, the outline is clear, and there is no occlusion, etc., the target detection algorithm will assign a high score greater than or equal to a first score to the target frame, which is a high-score frame. If the target is blocked or blurred, etc., it is assigned a low score less than the first score, which is a low-score frame, indicating that the detection of the target by the target detection algorithm is not very certain.

[0056] The high-scored frame and the low-scored frame obtained by the target detection algorithm are processed independently, and the low-scored result is not discarded easily.

[0057] When tracking the target, the high-scored frame is first matched with the previous tracking trajectory, and the tracking trajectory that does not match the high-scored frame is matched with the low-scored frame, so as to improve the continuity of the tracking trajectory of the low-scored frame.

[0058] For the high-scored frame that does not match the tracking trajectory, it is determined that the high-scored frame selects a new personnel target, a new tracking trajectory is created for the high-scored frame, and a new personnel number is assigned.

[0059] For the tracking trajectory that does not match the target frame, if there is no target frame matching the tracking trajectory after a set time, the tracking trajectory is discarded. For example, for the tracking trajectory that does not match the target frame, 30 frames are retained, and if there is no reoccurrence, the tracking trajectory information is removed.

[0060] In order to obtain an accurate tracking trajectory of the target frame, Kalman filtering is used to predict the target frame of the next frame, the position of the predicted frame is obtained, the intersection over union (IoU) value between the predicted frame and the high-scored frame is used as a similarity matching index, and when the intersection over union between the predicted frame and the high-scored frame is less than a set threshold, the tracking trajectory is matched with the target frame by using the Hungarian algorithm.

[0061] Kalman filtering is an algorithm for periodically optimally estimating the state of observation data using linear state equations, and the specific formula is as follows:

[0062]

[0063] wherein, represents the estimated value of the system state x k , z k represents the observation value of the periodic state, g k is the Kalman gain, between 0 and 1, updated once per period, and represents the degree of trust in the measurement value.

[0064] In the target tracking process, the target frame of the set time point video frame is identified, the identity information of the target frame of the set time point video frame is obtained, and the identity information is associated with the personnel number of the target frame.

[0065] In this embodiment, a face quality adaptive model is used to perform face recognition on the target frame of the set time point video frame, and the identity information of the target frame of the set time point video frame is obtained. The face quality adaptive model takes the personnel target selected by the target frame as input, takes the identity information of the target frame as output, and is obtained by constructing a face recognition algorithm.

[0066] The embodiment combines a face recognition algorithm with a target tracking algorithm, constructs a basic identity confirmation and tracking system, and creates a face database to be queried.

[0067] The face database to be queried has a mode of adding, modifying, and deleting face pictures at any time, and according to a monitoring scene, a person to be identified and tracked is selected, and multiple face pictures of the person in multiple angles such as a front view, a micro side view, and a micro low head view are shot, and the face pictures are stored in the face database to be queried in a naming format of “name+serial number”.

[0068] By writing codes, functions of reading the face database, querying target personnel pictures on demand, and one-key deleting or renaming are realized.

[0069] The face recognition algorithm selects a face quality adaptive model with a large performance advantage and obvious recognition accuracy rate improvement for low-quality face pictures. In the training process, a loss function based on image quality adaptation is used to assign different weights to training samples of different quality and difficulty. Based on the angle margin scaling the gradient according to the difficulty of the training sample, an adaptive margin function mechanism is proposed to emphasize difficult samples when the image quality is high and ignore difficult samples when the image quality is low, that is, samples that cannot be recognized, to ensure the normal progress of model training and the quality of model training. At the same time, the model does not need to use an additional module to evaluate the image quality, and the feature norm is introduced to realize efficient and fast image quality representation.

[0070] The face quality adaptive model disclosed in the embodiment normalizes the feature norm of each sample to obtain a normalized feature norm, adjusts the margin function in the loss function through the normalized feature norm, adjusts the margin function based on the feature norm, and uses the principle that the margin function can change the position of the decision boundary to realize different training difficulties for different samples.

[0071] The feature norm of each sample is normalized to obtain a normalized feature norm Specifically,

[0072]

[0073] wherein, ‖z i ‖ represents the feature norm of the i-th sample, μ z and σ z respectively represent the mean and standard deviation of the feature norm of all samples in a batch size, and through corresponding processing, the output range of the above formula is limited to [-1, 1].

[0074] The margin function in the loss function is adjusted through the normalized feature norm to obtain a loss function, and the loss function formula is as follows:

[0075]

[0076] where m is a scale hyper-parameter, representing the margin; s is a hyper-parameter for scaling; θ j is the angle between the feature vector of the sample and the weight vector of the jth classifier; y i is the label index of the Ground Truth; g angle and g add are two adaptive terms, which are functions related to the image quality indicators, defined as follows:

[0077]

[0078] The code logic of the target tracking algorithm and the face recognition algorithm is analyzed, a virtual environment meeting the logical requirements is created, a pre-trained face recognition model is introduced into the target tracking algorithm, and after the target tracking algorithm reads the video frame and recognizes the personnel target frame, the intercepted picture is sent to the face recognition model for identity recognition of the current target frame personnel.

[0079] In the process of target tracking, face recognition is performed simultaneously. Since the pixel difference between adjacent frames is small, in order to save time cost and computing resources, the embodiment does not perform identity recognition on each frame of the monitoring video, but uses every 20 frames to obtain a set of time point video frames, and only the target frame of the set of time point video frames is subjected to identity recognition.

[0080] In order to eliminate pictures that interfere with identity confirmation and ensure the accuracy of face recognition, and solve the problem of face mis-detection caused by overlapping of target frames when personnel are dense, the embodiment further screens the target frame by the intersection over union of each target frame in each set of time point video frames and the rest of the target frames, and whether the face in the target frame matches the personnel in the target frame, and only the screened target frame is subjected to identity recognition, thereby ensuring the accuracy of face recognition of the target frame.

[0081] wherein in the same frame, the formula for calculating the intersection over union between target frames is:

[0082]

[0083] wherein A and B represent the position coordinates of different target frames, and the intersection and union between the position sets of the two target frames represent the degree of coincidence between the two target frames.

[0084] That is, when the target frame of the set time point is identified, the target frame with the IoU equal to zero and the target frame with the IoU greater than zero and less than the set threshold and the face in the target frame matching the person in the target frame are selected for face recognition to obtain the identity information of each target frame in the corresponding set time point video frame.

[0085] The threshold of the IoU is set to 0.5. When the IoU value between the current target frame and other target frames is greater than or equal to 0.5, it is indicated that there is a large overlap between the current target frame and a certain target frame, and the probability of the two target faces appearing in the overlap area at the same time is large, and it is easy to misjudge in the subsequent face recognition and assign wrong identities to different target frames. Therefore, when the IoU between two target frames is greater than 0.5, the face in the current target frame is ignored, and the face identity recognition of the person in the current target frame is not performed.

[0086] When the IoU value between the current target frame and other target frames is between 0 and 0.5, the overlap rate is low, and it can be further confirmed whether face recognition is suitable at this time. The current target frame is sent to the face detection module to determine whether there is a face in the picture. If there is, the face coordinate area is obtained. The position rationality of the area and the overall target frame is determined, and it is determined whether the face area is located in the upper quarter of the target frame area. If the face area is not located in the upper quarter of the target frame area, it is considered that the face does not belong to the person in the current target frame, and the face recognition is stopped. If the face area is located in the upper quarter of the target frame area, it is determined that the face in the target frame matches the person in the target frame, and the face area in the target frame is identified to obtain the identity information of the current target frame.

[0087] When the IoU value between the current target frame and all other target frames is 0, it is considered that there is no interference from other personnel frames, and the face detection and recognition operations can be normally performed.

[0088] S3: For each target frame, the IoU with the remaining target frames in each frame of the monitoring video is calculated, and the independent time period of the target frame is determined through the IoU. It is determined that each set time point video frame is located in a time period. For each person number, the identity information of the set time point video frame in the same independent time period is counted, and the identity information with the highest frequency is selected as the identity recognition result of each person number in each independent time period.

[0089] In order to determine the independent time period of the target frame, the IoU of each target frame of each frame of the monitoring video with the remaining target frames is calculated.

[0090] In the process of using target tracking algorithms to detect and track personnel, different personnel targets are assigned different personnel numbers. If a personnel target remains in the monitoring screen without external obstruction and without ID exchange during the tracking process, then the personnel number of that personnel target will continue to follow the target frame, thus linking personal information and serving as the basis for subsequent identity matching. However, if personnel number exchange occurs during the tracking process, corresponding processing is required to achieve continuous identity tracking and error correction. Therefore, this embodiment also implements continuous identity tracking and error correction based on the principle that there is no external information interference within an independent time period for a person.

[0091] During the tracking algorithm's operation, the personnel IDs of different targets in each frame are sequentially acquired, and the IoU (Intersection over Union) of their bounding box coordinates is calculated with the coordinates of other personnel at the same time point to determine if there is a possibility of ID swapping at the current time. Extensive experimental verification shows that this problem often occurs after occlusion or overlap between two or more people from a monitoring perspective. To achieve identity correction after ID swapping, instead of solely matching identities based on personnel IDs, the issue of time periods is also considered: the identity of this ID is different before and after the ID swap. All target box states are divided into two types: independent and overlapping. It is assumed that erroneous swapping may occur when other target boxes overlap with it. Therefore, identity statistics are focused on the independent state (excluding the overlapping state), and identity information is statistically analyzed for different independent time periods. Different identity information is assigned to different independent time periods for this personnel ID.

[0092] The process for determining the independent time period of the target box in this embodiment is as follows:

[0093] Select the maximum intersection-over-union ratio (IoU) of the target bounding box in each frame of the surveillance video;

[0094] The period from the first frame where the maximum intersection-union ratio (MIR) is greater than the first value to the last frame where the MIR is greater than or equal to the second value is defined as the non-independent time period of the target box.

[0095] After removing non-independent time periods from the surveillance video, the remaining time periods are determined as the independent time periods of the target frame.

[0096] The process of determining the independent time period of the target box is explained with the first value being 0.5 and the second value being 0.2.

[0097] The embodiment creates a list s for storing time nodes of different personnel numbers and different states, associates the index of s with the personnel number, and stores the time node information of different personnel numbers in the corresponding index position of s. For each personnel target frame, the maximum value of the IoU value with other target frames in the last frame is recorded, which is the maximum intersection ratio of the target frame. In order to prevent the fluctuation of the maximum IoU value of the continuous frame from interfering with the state judgment, the single threshold judgment standard is discarded, and two different values are used as the dividing points of different states. Assuming that the maximum IoU value of any target frame in the first frame is 0, if the maximum IoU in the current frame is greater than 0.5 and the last frame is less than or equal to 0.5, it is recorded as no longer independent, and the current frame number is recorded as the time node of the start of overlap. If the maximum IoU in the current frame is less than 0.2 and the last frame is greater than or equal to 0.2, the current frame number is recorded as the time node of the end of overlap and the start of independence. The last frame from the no longer independent frame to the end of overlap and the start of independent frame is the independent time period of the target frame. The above time nodes are stored in digital form in another list w, and the recording is completed after the target frame disappears. The list w is stored in the corresponding index of the list s which records all the state time nodes.

[0098] During the entire system operation, face recognition is continuously performed on the personnel in different video frames, and the recognition result is saved together with the current frame number, personnel number, target frame position coordinates and other information, facilitating subsequent identity statistics. The list r stores the personnel information.

[0099] The list s saved during the determination of the independent time period of the target frame is read, and all elements of the list s are time period divided. Because the first value in each element represents the time when the target frame of the corresponding personnel number starts to overlap, it is considered that the first independent time period of the target frame of the personnel is from the first frame to the first value. The second value stored in each element of the list represents the time point when the target frame starts to be independent, and it is considered that the target frame overlaps with other target frames between the first value and the second value, and the identity information in this stage is ignored. It is considered that the second independent time period of the target frame is between the second value and the third value, and so on. All independent time periods of all target frames are obtained. The independent time period partition diagram is shown in Figure 2 .

[0100] To store identity information for different personnel IDs within different independent time periods, a nested list approach is used. A general list `i` is created, and its index is associated with the personnel ID. The identity information for different personnel IDs is stored in the element corresponding to the index. Any element in `i` creates a list `t` of the appropriate length based on the number of independent time periods for that personnel ID. This list stores the identity information within different independent time periods. For example, if the target box for personnel ID 1 has three independent time periods, then the length of `t` for that personnel ID is 3, and each element of `t` is also created as a new list `t_n`. List `t` is then stored in the element at index 1 of the general list `i`. Elements in the list `r` storing personnel information are read sequentially. The personnel ID, identity information, and the current time frame number are obtained. Based on the independent time period information for that personnel ID, it is determined whether the time frame number falls within an independent time period and is in the nth independent time period. The existence of the identity information in the current information is checked. If it is stored as unknown, this information is ignored. If it is stored as another actual name, the result is stored in the nth element list `t_n` of the list `t` for that personnel ID. Repeat the above steps to group the facial recognition information of all personnel IDs based on independent time periods. A diagram illustrating the categorized storage of personnel ID information by time period is shown below. Figure 3 As shown.

[0101] This allows us to determine the identity information of each person ID within each time period. Since a single person ID may have multiple identities, following the creation principle of list i above, a nested list approach is used to save the identities of different person IDs within different independent time periods. This enables us to find all identities for a given person ID based on the list index.

[0102] A schematic diagram of a time-based continuous identity tracking and error correction method is shown below. Figure 4 As shown.

[0103] S4: Annotate the surveillance video using the identity recognition results to obtain personnel tracking video.

[0104] Based on the identity recognition results, the target bounding boxes and identity information appearing in each frame of the surveillance video are labeled to obtain personnel tracking video.

[0105] When annotating surveillance videos, the location coordinates and personnel number of the target frame are found based on the frame number of the video frame; the type of time period in which the target frame is located is determined based on the frame number and personnel number; the identification results of the independent time period in which the personnel number is located are obtained, and the identification results are annotated on the surveillance video based on the location coordinates and frame number of the target frame to obtain personnel tracking video.

[0106] Specifically, each frame of the surveillance video is acquired sequentially. Based on the current frame number, the coordinates of the target bounding box and the person's ID at that time point are located. Then, the time period for that person is determined based on the time point and the person's ID, thus obtaining the identification result within that independent time period. A bounding box is drawn at the corresponding location on the screen according to the target bounding box coordinates, and the identification result is marked above the bounding box. This same operation is performed on each frame of the video, ultimately outputting a continuous tracking video with confirmed person identification.

[0107] This embodiment discloses a method based on existing target tracking and face recognition technologies. It addresses some problems encountered when combining these two technologies to achieve fine-grained identity verification and personnel tracking, and proposes solutions. Based on the concept that there is no external interference within an independent time frame of the target bounding box, a time-based continuous identity tracking and error correction method is proposed. This improves the identity verification and tracking method based on personnel ID matching. It utilizes facial identity information between overlapping time points of the personnel target bounding boxes during tracking, sequentially calculating facial identity information within each independent time frame to complete identity error correction before and after ID exchange, thus solving the problem of incorrect personnel tracking caused by ID exchange in target tracking algorithms in fine-grained scenarios. To address the problem of false face detection caused by overlapping target bounding boxes in densely populated areas, an adaptive face recognition mechanism is proposed. This mechanism uses the IoU value between target bounding boxes as an indicator of whether face recognition is suitable at that moment, and uses the matching degree between face coordinates and the human bounding box as an auxiliary factor to eliminate images that interfere with identity verification.

[0108] The method disclosed in this embodiment can achieve forward and backward identity tracking. When it is difficult to obtain a clear face image, the current identity can be obtained based on the continuously tracked identity dictionary within the same independent time period, without having to perform face recognition at all times. This can achieve efficient, fast and accurate identity recognition and tracking.

[0109] Example 2

[0110] In this embodiment, a fine-grained personnel tracking system based on facial recognition under surveillance video is disclosed, including:

[0111] The surveillance video acquisition module is used to acquire surveillance videos;

[0112] The target recognition and target tracking module is used to select personnel targets in the surveillance video by using target boxes, and to set a unique personnel number for each personnel target's target box; it performs target tracking for each target box, and during the tracking process, it performs identity recognition on the target boxes of video frames at set time points to obtain the identity information of the corresponding video frames;

[0113] The target frame independent time period identity recognition module is configured to calculate, for each target frame, an intersection-over-union with the rest of the target frames in each frame of the monitoring video, determine the independent time period of the target frame through the intersection-over-union, and determine the time period in which each set time point video frame is located. For each personnel number, the identity information of the set time point video frames in the same independent time period is counted, and the identity information with the highest occurrence frequency is selected as the identity recognition result of each personnel number in each independent time period.

[0114] The personnel tracking video acquisition module is configured to label the monitoring video through the identity recognition result and obtain a personnel tracking video.

[0115] Embodiment 3

[0116] In this embodiment, an electronic device is disclosed, which comprises a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps of the personnel fine-grained tracking method based on face recognition in a monitoring video disclosed in Embodiment 1 are completed.

[0117] Embodiment 4

[0118] In this embodiment, a computer readable storage medium is disclosed, which is configured to store computer instructions. When the computer instructions are executed by a processor, the steps of the personnel fine-grained tracking method based on face recognition in a monitoring video disclosed in Embodiment 1 are completed.

[0119] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application rather than limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application. Any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A fine-grained method for personnel tracking based on facial recognition in surveillance video, characterized in that, include: Obtain surveillance video; The system selects personnel targets in the surveillance video by using target bounding boxes and assigns a unique personnel number to each target bounding box. It then tracks each target bounding box and identifies the target bounding boxes in the video frames at set time points during the tracking process, thereby obtaining the identity information of each target bounding box in the corresponding video frames at set time points. For each target bounding box, the intersection-over-union (IoU) ratio with other target bounding boxes in each frame of the surveillance video is calculated. The IoU ratio is then used to determine the independent time period of each target bounding box. The process is as follows: Select the maximum intersection-over-union ratio (IoU) of the target bounding box in each frame of the surveillance video; The period from the first frame where the maximum intersection-union ratio (MIR) is greater than the first value to the last frame where the MIR is greater than or equal to the second value is defined as the non-independent time period of the target box. After removing non-independent time periods from the surveillance video, the remaining time periods are determined as the independent time periods of the target frame. Determine the time period in which each video frame at a given time point belongs; For each person number, the identity information of the video frames at a set time point in the same independent time period is statistically analyzed, and the identity information that appears most frequently is selected as the identity recognition result for each person number in each independent time period. By labeling the surveillance video with the results of identity recognition, personnel tracking videos can be obtained.

2. The fine-grained personnel tracking method based on face recognition in surveillance video as described in claim 1, characterized in that, The target bounding boxes are classified into high-resolution bounding boxes and low-resolution bounding boxes; When performing target tracking, the high-resolution bounding box is first matched with the previous tracking trajectory, and the tracking trajectory that has not been matched with the high-resolution bounding box is matched with the low-resolution bounding box.

3. The fine-grained personnel tracking method based on face recognition in surveillance video as described in claim 2, characterized in that, For high-scoring bounding boxes that do not have a matching trajectory, the high-scoring bounding box is identified as a new personnel target, a new tracking trajectory is created for the high-scoring bounding box, and a new personnel number is assigned.

4. The fine-grained personnel tracking method based on face recognition in surveillance video as described in claim 1, characterized in that, When performing identity recognition on target boxes in video frames at a set time point, target boxes with an intersection-union ratio (IU) of zero with other target boxes, and target boxes with an IU of at least one greater than zero but less than a set threshold, and which satisfy the condition that the face in the target box matches the person in the target box, are selected for face recognition to obtain the identity information of each target box in the corresponding set time point video frame.

5. The fine-grained personnel tracking method based on face recognition in surveillance video as described in claim 1, characterized in that, A face quality adaptive model is used to detect faces in the bounding boxes and obtain the identity information of the bounding boxes. The face quality adaptive model takes the person target selected by the bounding box as input and the obtained identity information of the bounding box as output.

6. The fine-grained personnel tracking method based on face recognition in surveillance video as described in claim 1, characterized in that, When annotating surveillance videos, the location coordinates and personnel number of the target frame are found based on the frame number of each frame; the type of time period in which the target frame is located is determined based on the frame number and personnel number; the identification results of the independent time period in which the personnel number is located are obtained, and the identification results are annotated on the surveillance video based on the location coordinates and frame number of the target frame to obtain personnel tracking video.

7. A fine-grained personnel tracking system based on facial recognition under surveillance video, characterized in that, include: The surveillance video acquisition module is used to acquire surveillance videos; The target recognition and target tracking module is used to select personnel targets in the surveillance video by using target boxes, and to set a unique personnel number for each personnel target's target box; to track each target box, and during the tracking process, to identify the target boxes in the video frames at a set time point, and to obtain the identity information of each target box in the corresponding set time point video frame; The target bounding box independent time period identification module is used to calculate the intersection-over-union ratio (IoU) with other target bounding boxes in each frame of the surveillance video for each target bounding box. Based on the IoU, the independent time period of the target bounding box is determined. The process is as follows: Select the maximum intersection-over-union ratio (IoU) of the target bounding box in each frame of the surveillance video; The period from the first frame where the maximum intersection-union ratio (MIR) is greater than the first value to the last frame where the MIR is greater than or equal to the second value is defined as the non-independent time period of the target box. After removing non-independent time periods from the surveillance video, the remaining time periods are determined as the independent time periods of the target frame. Determine the time period of each video frame at a set time point; for each person number, statistically analyze the identity information of the video frames at the set time points in the same independent time period, select the identity information that appears most frequently, and use it as the identity recognition result for each person number in each independent time period; The personnel tracking video acquisition module is used to annotate surveillance videos based on identity recognition results to obtain personnel tracking videos.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the steps of the fine-grained personnel tracking method based on face recognition in surveillance video as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the fine-grained personnel tracking method based on face recognition in surveillance video as described in any one of claims 1-6.

Images