A method for identifying people in fixed places based on deep learning
By using deep learning in closed places combined with object detection and tracking algorithms, extracting and matching human body and clothing features, the problems of limited scope and high cost of personnel identification in the prior art are solved, and high accuracy and low cost of identity recognition are achieved.
Patent Information
- Application Number
- CN202411853605.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-04-08
AI Technical Summary
The existing closed places have limited scope of use or are expensive, and it is impossible to accurately identify personnel identities in multiple angles, multiple distances and occlusions.
A deep learning-based method is adopted, combined with object detection and target tracking algorithms, the human body and clothing are detected and tracked, and the characteristics of the human body and clothing are extracted, matching and identification is achieved to realize the identification of personnel's identity without obtaining the face images of the person.
It improves the accuracy and robustness of personnel identification, reduces the requirements for camera clarity and installation angle, does not require positioning devices, is lower in cost, has a wider range of applications, and can accurately identify personnel identity in the presence of occlusion.
Smart Images
Figure CN119785402A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a method for identifying the identity of people in a fixed place based on deep learning. Background Art
[0002] In specific closed places such as computer rooms, airports, and conference centers, identifying personnel identities is an important prerequisite for subsequent personnel management and service improvement. For example, in a computer room scenario, different operators have different permissions, corresponding to different operations that can be performed or areas that can be entered in the computer room. For the purpose of risk management, when a low-authority person enters a high-authority area or performs a high-authority operation, it is necessary to trigger an alarm or record the person's risky behavior in a timely manner. Therefore, identifying the identity of people in the computer room and matching the corresponding personnel permissions are necessary conditions for personnel risk behavior management.
[0003] There are two main methods for personnel identification. The most commonly used method for personnel identification is to identify people by collecting facial images. However, this method requires people to face the camera and there should be no obvious obstruction, which limits its use in most scenarios. In most closed places, since the camera installation position and angle are relatively fixed, but people can move arbitrarily, due to the distance, angle or possible obstruction of the face of the person, such as hats and masks, it is impossible to guarantee that a clear and complete facial image of the person can be obtained, and thus the identity of the person cannot be accurately identified; another method is to have the people in the venue wear positioning devices, such as positioning badges, etc. This method has high requirements for the positioning accuracy of the positioning device, and the method is more restricted in applicable scenarios. In places such as airports where the number of people and mobility are generally large, the number of equipment and management recovery requirements are very high, and the equipment cost is high, which does not have the value of wide application.
[0004] Chinese patent document CN 114495218 A discloses a method, device, computer equipment, storage medium and computer program product for identifying personnel in power places based on deep learning, the method comprising: obtaining a face image to be detected; the face image to be detected is a face image of a person who needs to enter the target power place; determining the face image features corresponding to the face image to be detected, and determining the target person corresponding to the face image to be detected according to the face image features; obtaining the personnel identity information of the target person, and the content of the power work ticket of the target person; determining the target qualification information required to perform the task to be performed in the content of the power work ticket, and if the qualification information in the personnel identity information matches the target qualification information, the target person is allowed to enter the target power place. This technical solution does not involve the comparison and matching of clothing.
[0005] Therefore, it is necessary to provide a method for identifying the identities of people in fixed places based on deep learning, which has higher accuracy and robustness, so as to solve the problems of limited scope of use or high cost of existing methods for identifying the identities of people in closed places. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a method for identifying the identity of people in fixed places based on deep learning, which has higher accuracy and robustness, and is used to solve the problem that the existing methods for identifying the identity of people in closed places are limited in scope of use or high in cost.
[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: the method for identifying the identity of a person in a fixed place based on deep learning specifically comprises the following steps:
[0008] S1 Personnel entry: Personnel enter by scanning their ID or face, obtain the identity and image of the person entering, extract the characteristics of the person entering, obtain the characteristics of the person entering, and establish the characteristic information table of the person entering;
[0009] S2 Personnel tracking: Track the movement trajectory of personnel in the venue, set a unique ID for each person in the venue, and extract the characteristics of the personnel in the venue to obtain the characteristics of the personnel in the venue;
[0010] S3 Personnel Matching: Match the characteristics of the personnel in the venue with the characteristic information table of the personnel entering the venue, and associate the identity of the personnel corresponding to the ID of the personnel in the venue;
[0011] S4 Personnel exit: Identify the exiting personnel and delete the exiting personnel information in the entry personnel information table.
[0012] By adopting the above technical solution, human bodies and clothing are detected based on target detection and target tracking algorithms, and human body movements are tracked. Face recognition is not required, and the identity of people in closed places can be identified without obtaining facial images of the people. This method has higher accuracy and robustness, and is used to solve the problem of limited scope of use or high cost of existing methods for identifying people in closed places.
[0013] Preferably, the specific steps of step S1 are:
[0014] S11: At the entrance of the venue, the personnel obtain their identity information by swiping a card or performing facial recognition. At the same time, multiple cameras at the entrance collect images of the personnel entering the venue from multiple angles.
[0015] S12: Use the preset human target detection algorithm to identify people and use the convolutional neural network to extract features to obtain the characteristics of the people entering the venue;
[0016] S13: using a preset clothing target detection algorithm to identify clothing features of the entrants, such as whether they are wearing glasses, whether they are carrying a backpack, clothing color, etc.;
[0017] S14: Establish an entry person feature information table, associate the identity information, the converted human feature vector, and the clothing feature of each entry person, and save them in the entry person feature information table.
[0018] Preferably, the human target detection algorithm used in step S12 is YOLO (You Only Look Once) or Faster R-CNN (Region-based Convolutional Neural Networks) or SSD (Single Shot MultiBox Detector). First, the human target in the personnel entry image is detected, and the human image is cropped through the identified bounding box; then a convolutional neural network (CNN) is used to extract features from the cropped human image, and the extracted human features are converted into human feature vectors.
[0019] Preferably, in step S2, for persons who have entered the venue, the DeepSORT multi-target tracking algorithm is used to continuously detect and track the persons in the video frames, and a unique ID is assigned to each detected person; the image of the detected person in the venue is cropped, and then the same method as in step S1 is used to extract human feature vectors and clothing features from the cropped image of the person.
[0020] Preferably, the specific steps of using the DeepSORT multi-target tracking algorithm to track people in step S2 are:
[0021] S21: First, use an object detection algorithm such as YOLO to detect people in the video frame and generate a bounding box and its confidence for each detected person;
[0022] S22: After detecting a person, a Kalman filter is used to predict the position of the person, and the detection result of the current frame is matched with the tracking result of the previous frame by a data association method;
[0023] S23: In each video frame, the DeepSORT multi-target tracking algorithm updates the status of the person, including location, speed and other information, based on the results of data association;
[0024] S24: For unmatched persons, update is performed based on the prediction of the Kalman filter.
[0025] Preferably, the step S22 of using the Kalman filter to predict the position of the person is divided into a prediction stage and an update stage; specifically, the following steps:
[0026] S221 prediction stage: In the prediction stage, for each known target, the state of the previous frame is used to predict the position of the target in the current frame, in which the state transition model and the covariance matrix of the process noise are used, specifically:
[0027] The formula for state prediction is:
[0028] X k|k-1 =F k X k-1|k-1 +B k u k ;
[0029] The formula for covariance prediction is:
[0030]
[0031] Among them, X k|k-1 is the state prediction at time k, F k is the state transfer matrix, B k is the control input matrix, u k is the control vector, P k|k-1 is the covariance matrix of the predicted state, Q k is the covariance matrix of the process noise;
[0032] S222 Update phase: In the update phase, after receiving a new measurement value, the filter uses the new measurement value to correct the previous prediction, thereby obtaining a more accurate position estimate; the update phase includes three sub-steps, namely: measurement prediction, Kalman gain calculation and state update;
[0033] Among them, the formula for measurement prediction is:
[0034] Z k =H k X k|k-1 ;
[0035] The formula for Kalman gain is:
[0036]
[0037] The formula for status update is:
[0038] X k|k =X k|k-1 +K k (z k -Z k );
[0039] The formula for covariance update is:
[0040] P k|k =(IK k Hk ) k|k-1 ;
[0041] Among them, Z k is the measurement prediction at time k, H k is the observation matrix, R k is the covariance matrix of the measurement noise, K k is the Kalman gain, X k|k is the state update at time k, P k|k is the updated state covariance matrix, z k is the actual measured value.
[0042] Preferably, the DeepSORT used in step S23 handles the occlusion and loss of the target by maintaining the life cycle and appearance features of a target; when the target is occluded or disappears temporarily, the DeepSORT multi-target tracking algorithm will re-identify the target based on historical information and appearance features, so that the target is identified as the same individual before and after being occluded or disappearing, and is assigned the same ID.
[0043] Preferably, the specific steps of step S3 are:
[0044] S31: firstly, matching of human feature vectors is performed, and each human feature vector extracted from the video frame is compared with the human feature vectors of all known identities when the person enters the venue, and the similarity between the two is measured using a distance metric, and a human similarity score is calculated;
[0045] S32: clothing feature matching is then performed, and the clothing feature of each person extracted from the video frame is matched with the clothing features of all persons with known identities when the person enters the venue; a weight is assigned to each clothing feature, and a clothing matching score is obtained by weighting according to the number of clothing features finally matched;
[0046] S33: Weights are given to the human body similarity score and the clothing matching score respectively, and the overall similarity score is weightedly calculated. The identities of the people in each video frame are matched according to the score sorting.
[0047] Preferably, the step S3 further includes S34: when a person is blocked or disappears briefly and then reappears in the video frame, the person ID of the previous frame is matched according to the person ID in the current frame, and the person identity in the previous frame is associated with the current frame; thus, when the human body image is blocked, the person identity can be accurately identified;
[0048] S35: If the person ID is consistent in the current frame and the previous frame of the video, but the person identity recognition results are inconsistent, the recognition result of the previous frame is retained as the final recognition result of the current person identity, and a counter is created to accumulate the number of frames in which the recognition result of the current frame appears continuously in subsequent video frames. When the value of the counter is greater than a specific value, the person identity recognition result is updated to the recognition result of the current frame, and the counter is cleared at the same time; otherwise, if the value of the counter is not greater than the specific value and the recognition results of subsequent video frames change, the value of the counter is set to 0, and the number of frames in which the recognition result of the current frame appears continuously in subsequent video frames is recounted to avoid the impact of misidentification of person identities in individual frames on the final recognition result.
[0049] The above technical solution is adopted, based on the similarity matching of the human body, a convolutional neural network (CNN) is used to extract features from the entire human body image, obtain a high-dimensional feature vector, compare the extracted feature vector with the feature vector of the known identity in the data table, and use distance metrics (such as Euclidean distance, cosine similarity) to determine the identity. Based on the similarity matching of the personnel clothing features, the clothing features of the personnel to be detected and the personnel with known identities in the data table are matched based on clothing check points (glasses, hats, clothing colors, etc.). Based on the target tracking algorithm, the ID of the moving personnel in the venue is marked, the movement trajectory of the personnel is tracked, and the identity of the personnel is identified based on the matching results of the personnel ID and the personnel identity. The identity recognition results of the same person ID in the current frame and the previous frame are compared to avoid the influence of the misidentification of the person identity in individual frames on the final recognition result; at the same time, the identity of the person can be effectively identified when the human body is partially blocked.
[0050] Preferably, the step S31 uses Euclidean distance or cosine similarity when using distance to measure the similarity between two pairs;
[0051] The Euclidean distance calculation formula is:
[0052]
[0053] Among them, x i and i are the coordinates of vector x and vector y in the i-th dimension respectively;
[0054] The formula for calculating cosine similarity is:
[0055]
[0056] Among them, ||x|| and ||y|| are the magnitudes of vector x and vector y, x i and i are the coordinates of vector x and vector y in the i-th dimension.
[0057] Preferably, the calculation formula for the clothing matching degree in step S32 is:
[0058]
[0059] Among them, f i is the matching result of the i-th feature, W i is the preset weight of the ith feature.
[0060] Compared with the prior art, the present invention has the following beneficial effects: it adopts target detection and target tracking algorithms to detect human bodies and clothing, etc., and track human body movements, without the need for face recognition. It can realize the identification of people in closed places without obtaining facial images of people. Compared with other existing methods, the present invention has no special requirements for the clarity and installation angle of the camera, and does not require a positioning device, so it has lower costs and a wider range of applications. The method uses a combination of human feature matching and clothing feature matching to avoid detection errors of a single identification method and effectively improve the identification accuracy. Based on the implementation of the target tracking algorithm, the detected human body is not directly identified, but the person ID identified by the associated target tracking algorithm is used to identify the person through the person's movement trajectory, thereby avoiding the situation of incorrect identification of people in individual frames, and realizing that the identity of people can be effectively identified even when there are obstructions in the place. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is a flow chart of the method for identifying a person in a closed place based on deep learning of the present invention;
[0062] Figure 2 It is a detailed structural diagram of a specific embodiment of the method for identifying a person in a closed place based on deep learning of the present invention;
[0063] Figure 3 This is an example of personnel detection results in a specific embodiment of the method for identifying personnel in a closed place based on deep learning of the present invention. DETAILED DESCRIPTION
[0064] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.
[0065] Example: Figure 1 As shown, the method for identifying the identity of a person in a fixed place based on deep learning specifically includes the following steps:
[0066] S1 Personnel entry: Personnel enter by scanning their ID or face, obtain the identity and image of the person entering, extract the characteristics of the person entering, obtain the characteristics of the person entering, and establish the characteristic information table of the person entering;
[0067] The specific steps of step S1 are:
[0068] S11: At the entrance of the venue, the personnel obtain their identity information by swiping a card or performing facial recognition. At the same time, multiple cameras at the entrance collect images of the personnel entering the venue from multiple angles.
[0069] S12: Use the preset human target detection algorithm to identify people and use the convolutional neural network to extract features to obtain the characteristics of the people entering the venue;
[0070] The human target detection algorithm used in step S12 is YOLO or Faster R-CNN or SSD. First, the human target in the personnel entry image is detected, and the human image is cropped by the identified bounding box; then a convolutional neural network (CNN) is used to extract features from the cropped human image, and the extracted human features are converted into human feature vectors;
[0071] S13: using a preset clothing target detection algorithm to identify clothing features of the entrants, such as whether they are wearing glasses, whether they are carrying a backpack, clothing color, etc.;
[0072] S14: Establishing a characteristic information table of the entrants, associating the identity information, the converted human feature vector, and the clothing features of each entrant, and saving them in the characteristic information table of the entrants;
[0073] S2 Personnel tracking: Track the movement trajectory of personnel in the venue, set a unique ID for each person in the venue, and extract the characteristics of the personnel in the venue to obtain the characteristics of the personnel in the venue;
[0074] In step S2, for the personnel who have entered the venue, the DeepSORT multi-target tracking algorithm is used to continuously detect and track the personnel in the video frame, and a unique ID is assigned to each detected person; the image of the detected person in the venue is cropped, and then the human body feature vector and clothing feature are extracted from the cropped person image using the same method as in step S1;
[0075] The specific steps of using the DeepSORT multi-target tracking algorithm to track people in step S2 are:
[0076] S21: First, use an object detection algorithm such as YOLO to detect people in the video frame and generate a bounding box and its confidence for each detected person;
[0077] S22: After detecting a person, a Kalman filter is used to predict the position of the person, and the detection result of the current frame is matched with the tracking result of the previous frame by a data association method;
[0078] The step S22 of using the Kalman filter to predict the position of the person is divided into a prediction stage and an update stage; specifically,
[0079] S221 prediction stage: In the prediction stage, for each known target, the state of the previous frame is used to predict the position of the target in the current frame, in which the state transition model and the covariance matrix of the process noise are used, specifically:
[0080] The formula for state prediction is:
[0081] X k|k-1 =F k X k-1|k-1 +B k u k ;
[0082] The formula for covariance prediction is:
[0083]
[0084] Among them, X k|k-1 is the state prediction at time k, F k is the state transfer matrix, B k is the control input matrix, u k is the control vector, P k|k-1 is the covariance matrix of the predicted state, Q k is the covariance matrix of the process noise;
[0085] S222 Update phase: In the update phase, after receiving a new measurement value, the filter uses the new measurement value to correct the previous prediction, thereby obtaining a more accurate position estimate; the update phase includes three sub-steps, namely: measurement prediction, Kalman gain calculation and state update;
[0086] Among them, the formula for measurement prediction is:
[0087] Z k =H k X k|k-1 ;
[0088] The formula for Kalman gain is:
[0089]
[0090] The formula for status update is:
[0091] X k|k =X k|k-1 +Kk (z k -Z k );
[0092] The formula for covariance update is:
[0093] P k|k =(IK k H k ) k|k-1 ;
[0094] Among them, Z k is the measurement prediction at time k, H k is the observation matrix, R k is the covariance matrix of the measurement noise, K k is the Kalman gain, X k|k is the state update at time k, P k|k is the updated state covariance matrix, z k is the actual measured value;
[0095] S23: In each video frame, the DeepSORT multi-target tracking algorithm updates the status of the person, including location, speed and other information, based on the results of data association;
[0096] The DeepSORT used in step S23 handles the occlusion and loss of a target by maintaining the life cycle and appearance features of a target; when a target is occluded or disappears temporarily, the DeepSORT multi-target tracking algorithm will re-identify the target based on historical information and appearance features, so that the target is identified as the same individual before and after being occluded or disappearing, and the same ID is assigned;
[0097] S24: For unmatched personnel, update according to the prediction of the Kalman filter;
[0098] S3 Personnel Matching: Match the characteristics of the personnel in the venue with the characteristic information table of the personnel entering the venue, and associate the identity of the personnel corresponding to the ID of the personnel in the venue;
[0099] The specific steps of step S3 are:
[0100] S31: firstly, matching of human feature vectors is performed, and each human feature vector extracted from the video frame is compared with the human feature vectors of all known identities when the person enters the venue, and the similarity between the two is measured using a distance metric, and a human similarity score is calculated;
[0101] When the step S31 uses the distance measure to measure the similarity between two pairs, the Euclidean distance or the cosine similarity is used;
[0102] The Euclidean distance calculation formula is:
[0103]
[0104] Among them, x i and i are the coordinates of vector x and vector y in the i-th dimension respectively;
[0105] The formula for calculating cosine similarity is:
[0106]
[0107] Among them, ||x|| and ||y|| are the magnitudes of vector x and vector y, x i and i are the coordinates of vector x and vector y in the i-th dimension;
[0108] S32: clothing feature matching is then performed, and the clothing feature of each person extracted from the video frame is matched with the clothing features of all persons with known identities when the person enters the venue; a weight is assigned to each clothing feature, and a clothing matching score is obtained by weighting according to the number of clothing features finally matched;
[0109] The calculation formula of the clothing matching degree in step S32 is:
[0110]
[0111] Among them, f i is the matching result of the i-th feature, W i is the preset weight of the i-th feature;
[0112] S33: giving weights to the human body similarity score and the clothing matching score respectively, and calculating the overall similarity score by weight, sorting by the scores, and matching the identity of the person in each video frame;
[0113] S34: When a person is blocked or disappears briefly and then reappears in a video frame, the person ID in the previous frame is matched according to the person ID in the current frame, and the identity of the person in the previous frame is associated with the current frame; thus, the identity of the person can be accurately identified even when the human body image is blocked;
[0114] S35: If the person ID is consistent in the current frame and the previous frame of the video, but the person identification results are inconsistent, the identification result of the previous frame is retained as the final identification result of the current person identity, and a counter is created to accumulate the number of frames in which the identification result of the current frame appears continuously in subsequent video frames. When the value of the counter is greater than a specific value, the person identity identification result is updated to the identification result of the current frame, and the counter is cleared at the same time; otherwise, if the value of the counter is not greater than the specific value, and the identification results of subsequent video frames change, the value of the counter is set to 0, and the number of frames in which the identification result of the current frame appears continuously in subsequent video frames is recounted to avoid the influence of the misidentification of the person identity in individual frames on the final identification result;
[0115] S4 Personnel exit: Identify the exiting personnel and delete the exiting personnel information in the entry personnel information table.
[0116] Specific embodiment: This embodiment is in the computer room scenario, such as Figure 1 and Figure 2 As shown, the method for identifying the identity of a person in a fixed place based on deep learning specifically includes the following steps:
[0117] S1 Personnel entry: Personnel enter by scanning their ID or face, obtain the identity and image of the person entering, confirm the identity of the person and ID through face recognition, extract the characteristics of the person, obtain the characteristics of the person entering, and establish the characteristic information table of the person entering;
[0118] The specific steps of step S1 are:
[0119] S11: A person enters the room by scanning his / her ID card at the entrance of the computer room. The ID card is used to obtain the person's identity information. At the same time, facial recognition is used to confirm whether the person's ID card matches the person's ID card. If the ID card matches the person's ID card, the person is allowed to enter the room. At the same time, the access control camera and multiple cameras at the entrance collect images of the person entering the room from multiple angles.
[0120] S12: Use the YOLO-based human detection algorithm to detect the multi-angle images of people entering the scene, identify the human bounding box, crop the human image, and use the convolutional neural network (CNN) to extract features from the cropped human image, and convert the extracted human features into human feature vectors;
[0121] S13: Use the YOLO-based clothing target detection algorithm to identify the clothing features of the entrants, including whether they are wearing glasses, backpacks, masks, and the color of their tops;
[0122] S14: Establishing a characteristic information table of the entrants, associating the identity information, the converted human feature vector, and the clothing features of each entrant, and saving them in the characteristic information table of the entrants;
[0123] The sample data of the characteristic information table of the entrance personnel is shown in Table 1:
[0124] Table 1 Characteristics of the personnel entering the venue
[0125]
[0126] S2 Personnel tracking: Track the movement trajectory of personnel in the venue, set a unique ID for each person in the venue, and extract the characteristics of the personnel in the venue to obtain the characteristics of the personnel in the venue;
[0127] In step S2, each frame of the image is obtained through the camera installed inside the computer room, and the YOLO+DeepSORT multi-target tracking algorithm is used to continuously detect and track the personnel in the video frame, and a unique ID is assigned to each detected person; the detected person image is cropped, and the human feature vector and clothing features are extracted using the same method as in S1;
[0128] The specific steps of using the DeepSORT multi-target tracking algorithm to track people in step S2 are:
[0129] S21: First, use an object detection algorithm such as YOLO to detect people in the video frame and generate a bounding box and its confidence for each detected person;
[0130] S22: After detecting a human body, a unique person ID is assigned to each human body, a Kalman filter is used to predict the position of the human body, and the detection result of the current frame is matched with the tracking result of the previous frame through a data association method;
[0131] The step S22 of using the Kalman filter to predict the position of the person is divided into a prediction stage and an update stage; specifically,
[0132] S221 prediction stage: In the prediction stage, for each known target, the state of the previous frame is used to predict the position of the target in the current frame, in which the state transition model and the covariance matrix of the process noise are used, specifically:
[0133] The formula for state prediction is:
[0134] X k|k-1 =F k X k-1|k-1 +B k u k ;
[0135] The formula for covariance prediction is:
[0136]
[0137] Among them, X k|k-1is the state prediction at time k, F k is the state transfer matrix, B k is the control input matrix, u k is the control vector, P k|k-1 is the covariance matrix of the predicted state, Q k is the covariance matrix of the process noise;
[0138] S222 Update phase: In the update phase, after receiving a new measurement value, the filter uses the new measurement value to correct the previous prediction, thereby obtaining a more accurate position estimate; the update phase includes three sub-steps, namely: measurement prediction, Kalman gain calculation and state update;
[0139] Among them, the formula for measurement prediction is:
[0140] Z k =H k X k|k-1 ;
[0141] The formula for Kalman gain is:
[0142]
[0143] The formula for status update is:
[0144] X k|k =X k|k-1 +K k (z k -Z k );
[0145] The formula for covariance update is:
[0146] P k|k =(IK k H k ) k|k-1 ;
[0147] Among them, Z k is the measurement prediction at time k, H k is the observation matrix, R k is the covariance matrix of the measurement noise, K k is the Kalman gain, X k|k is the state update at time k, P k|k is the updated state covariance matrix, z k is the actual measured value;
[0148] S23: In each video frame, the DeepSORT multi-target tracking algorithm updates the status of the person, including location, speed and other information, based on the results of data association. Therefore, for the same person in different video frames, the person ID remains consistent. For unmatched persons, the algorithm will update them based on the prediction of the Kalman filter and assign them a new person ID.
[0149] The DeepSORT used in step S23 handles the occlusion and loss of a target by maintaining the life cycle and appearance features of a target; when a target is occluded or disappears temporarily, the DeepSORT multi-target tracking algorithm will re-identify the target based on historical information and appearance features, so that the target is identified as the same individual before and after being occluded or disappearing, and the same ID is assigned;
[0150] The characteristic information of people in the computer room extracted from a certain frame is shown in Table 2 below:
[0151] Table 2 Characteristic information of personnel in the computer room
[0152] Personnel ID Human characteristics Backpack Glasses Face mask Top color ID_001 Eigenvector 001 0 0 0 black ID_002 Eigenvector 002 0 1 0 red
[0153] S24: For unmatched personnel, update according to the prediction of the Kalman filter;
[0154] S3 Personnel Matching: Match the characteristics of the personnel in the venue with the characteristic information table of the personnel entering the venue, and associate the identity of the personnel corresponding to the ID of the personnel in the venue;
[0155] The specific steps of step S3 are:
[0156] S31: First, human feature vectors are matched. Each human feature vector extracted from the video frame is compared with the human feature vectors of all known identities when the person enters the venue. The similarity between the two is measured using a distance metric to calculate a human similarity score. The maximum value of the similarity score between each person ID and each person's human feature vector at each angle is taken as the matching result between the person ID and the known person.
[0157] When the step S31 uses the distance measure to measure the similarity between two pairs, the Euclidean distance or the cosine similarity is used;
[0158] The Euclidean distance calculation formula is:
[0159]
[0160] Among them, x i and i are the coordinates of vector x and vector y in the i-th dimension respectively;
[0161] The formula for calculating cosine similarity is:
[0162]
[0163] Among them, ||x|| and ||y|| are the magnitudes of vector x and vector y, x i and i are the coordinates of vector x and vector y in the i-th dimension;
[0164] The matching results of the human feature vectors in Table 2 and the human feature vectors in Table 1 are shown in Table 3 below:
[0165] Table 3 Human feature vector matching results
[0166]
[0167]
[0168] S32: clothing feature matching is then performed, and the clothing feature of each person extracted from the video frame is matched with the clothing features of all persons with known identities when the person enters the venue; a weight is assigned to each clothing feature, and a clothing matching score is obtained by weighting according to the number of clothing features finally matched;
[0169] The calculation formula of the clothing matching degree in step S32 is:
[0170]
[0171] Among them, f i is the matching result of the i-th feature, W i is the preset weight of the i-th feature;
[0172] S33: giving weights to the human body similarity score and the clothing matching score respectively, and calculating the overall similarity score by weight, sorting by the scores, and matching the identity of the person in each video frame;
[0173] In this embodiment, the weights given to the backpack, glasses, mask, and top color are 1, 2, 3, and 4; then the clothing feature scores of ID_001 in Table 2 and Zhang San in Table 1 are as follows:
[0174]
[0175] The matching results of the clothing features of the personnel in Table 2 and the clothing features of the personnel in Table 1 are shown in Table 4 below;
[0176] Table 4 Results of matching personnel clothing features
[0177]
[0178] From the above table, we can see that the maximum similarity score in the current frame is 0.70668, which corresponds to identifying the person with ID_001 as Zhang San, and the corresponding person with the remaining ID_002 is identified as Li Si;
[0179] S34: When a person is blocked or disappears briefly and then reappears in a video frame, the person ID in the previous frame is matched according to the person ID in the current frame, and the identity of the person in the previous frame is associated with the current frame; thus, the identity of the person can be accurately identified even when the human body image is blocked;
[0180] S35: The person identification result of the current frame in the video is label1, and the identification result of the previous frame is label2. When the identification results of the current frame and the previous frame for the same person ID are inconsistent, that is, label1≠label2, the identification result label2 of the previous frame is retained as the final identification result label of the current person identity, and a counter count is generated at the same time to count the number of frames in which the person ID is continuously identified as label1 in subsequent frames. When the value of the counter count is greater than 48 (i.e., 2 seconds, 24 frames per second), the identification result of the person ID is updated to label1, and the counter is cleared at the same time; if the value of the counter count is not greater than 48 and the identification result of the current frame is changed from label1 to label3, the value of the counter count is set to 0, and the number of frames in which the person ID is continuously identified as label3 in subsequent frames is counted; the above steps are repeated continuously during the identification process, which can effectively avoid the impact of misidentification of the person identity in individual frames on the final identification result; examples of person detection results are as follows Figure 3 As shown;
[0181] S4 Personnel exit: A person swipes his card to exit the room at a restricted area inside the computer room, obtains his identity, and deletes the corresponding exiting personnel information from the existing entry personnel feature information table.
[0182] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present invention. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present invention, or the concepts and technical solutions of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.
Claims
1. A method for identifying the identity of people in a fixed place based on deep learning, characterized in that: The specific steps include: S1 Personnel entry: Personnel enter by scanning their ID or face, obtain the identity and image of the person entering, extract the characteristics of the person entering, obtain the characteristics of the person entering, and establish the characteristic information table of the person entering; S2 Personnel Tracking: Track the movement trajectory of personnel in the venue, set a unique ID for each person in the venue, and extract the characteristics of the personnel in the venue to obtain the characteristics of the personnel in the venue; S3 Personnel Matching: Match the characteristics of the personnel in the venue with the characteristic information table of the personnel entering the venue, and associate the personnel identity corresponding to the personnel ID in the venue; S4 Personnel exit: Identify the exiting personnel and delete the exiting personnel information in the entry personnel information table.
2. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 1, characterized in that: The specific steps of step S1 are: S11: At the entrance of the venue, the personnel obtain their identity information by swiping a card or performing facial recognition. At the same time, multiple cameras at the entrance collect images of the personnel entering the venue from multiple angles. S12: Use the preset human target detection algorithm to identify people and use the convolutional neural network to extract features to obtain the characteristics of the people entering the venue; S13: using a preset clothing target detection algorithm to identify clothing features of the person entering the venue; S14: Establish an entry person feature information table, associate the identity information, the converted human feature vector, and the clothing feature of each entry person, and save them in the entry person feature information table.
3. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 2, characterized in that: The human target detection algorithm used in step S12 is YOLO or Faster R-CNN or SSD. First, the human target in the personnel entry image is detected, and the human image is cropped through the identified bounding box; then a convolutional neural network is used to extract features from the cropped human image, and the extracted human features are converted into human feature vectors.
4. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 3 is characterized in that: In step S2, for persons who have entered the venue, the DeepSORT multi-target tracking algorithm is used to continuously detect and track persons in the video frames, and a unique ID is assigned to each detected person; the image of the detected person in the venue is cropped, and then the same method as in step S1 is used to extract human feature vectors and clothing features from the cropped image of the person.
5. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 4 is characterized in that: The specific steps of using the DeepSORT multi-target tracking algorithm to track people in step S2 are: S21: First, use the object detection algorithm to detect people in the video frame and generate a bounding box and its confidence for each detected person; S22: After detecting a person, a Kalman filter is used to predict the position of the person, and the detection result of the current frame is matched with the tracking result of the previous frame by a data association method; S23: In each video frame, the DeepSORT multi-target tracking algorithm updates the status of the person according to the result of data association; S24: For unmatched persons, update is performed based on the prediction of the Kalman filter.
6. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 5, characterized in that: The step S22 of using the Kalman filter to predict the position of the person is divided into a prediction stage and an update stage; specifically, S221 prediction stage: In the prediction stage, for each known target, the state of the previous frame is used to predict the position of the target in the current frame, in which the state transition model and the covariance matrix of the process noise are used, specifically: The formula for state prediction is: X k|k-1 =F k X k-1|k-1 +B k u k ; The formula for covariance prediction is: Among them, X k|k-1 is the state prediction at time k, F k is the state transfer matrix, B k is the control input matrix, u k is the control vector, P k|k-1 is the covariance matrix of the predicted state, Q k is the covariance matrix of the process noise; S222 Update phase: In the update phase, after receiving a new measurement value, the filter uses the new measurement value to correct the previous prediction, thereby obtaining a more accurate position estimate; the update phase includes three sub-steps, namely: measurement prediction, Kalman gain calculation and state update; Among them, the formula for measurement prediction is: Z k =H k X k|k-1 ; The formula for Kalman gain is: The formula for status update is: X k|k =X k|k-1 +K k (z k -Z k ); The formula for covariance update is: P k|k =(I-K k H k )P k|k-1 ; Among them, Z k is the measurement prediction at time k, H k is the observation matrix, R k is the covariance matrix of the measurement noise, K k is the Kalman gain, X k|k is the state update at time k, P k|k is the updated state covariance matrix, z k is the actual measured value.
7. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 5, characterized in that: In step S23, when the target is blocked or disappears temporarily, the DeepSORT multi-target tracking algorithm will re-identify the target based on historical information and appearance features, so that the target is identified as the same individual before and after being blocked or disappearing, and is assigned the same ID.
8. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 5, characterized in that: The specific steps of step S3 are: S31: firstly, matching of human feature vectors is performed, and each human feature vector extracted from the video frame is compared with the human feature vectors of all known identities when the person enters the venue, and the similarity between the two is measured using a distance metric, and a human similarity score is calculated; S32: clothing feature matching is then performed, and the clothing feature of each person extracted from the video frame is matched with the clothing features of all persons with known identities when the person enters the venue; a weight is assigned to each clothing feature, and a clothing matching score is obtained by weighting according to the number of clothing features finally matched; S33: Weights are given to the human body similarity score and the clothing matching score respectively, and the overall similarity score is weightedly calculated. The identities of the people in each video frame are matched according to the score sorting.
9. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 8, characterized in that: The step S3 also includes S34: when a person is blocked or disappears briefly and then reappears in the video frame, matching the person ID of the previous frame with the person ID of the current frame, and associating the person identity in the previous frame with the current frame; S35: If the person ID in the current frame and the previous frame of the video is consistent, but the person identification results are inconsistent, the identification result of the previous frame is retained as the final identification result of the current person identity, and a counter is created to accumulate the number of frames in which the identification result of the current frame appears continuously in subsequent video frames. When the value of the counter is greater than a specific value, the person identification result is updated to the identification result of the current frame, and the counter is cleared at the same time; Otherwise, if the counter value is not greater than a specific value and the recognition result of the subsequent video frame changes, the counter value is set to 0, and the number of consecutive frames in which the recognition result of the current frame appears in the subsequent video frames is recounted to avoid the impact of the misidentification of the identity of the person in individual frames on the final recognition result.
10. The method for identifying the identity of a person in a fixed place based on deep learning according to claim 7, characterized in that: When the step S31 uses the distance measure to measure the similarity between two pairs, the Euclidean distance or the cosine similarity is used; The Euclidean distance calculation formula is: Among them, x i and i are the coordinates of vector x and vector y in the i-th dimension respectively; The formula for calculating cosine similarity is: Among them, ||x|| and ||y|| are the magnitudes of vector x and vector y, x i and i are the coordinates of vector x and vector y in the i-th dimension; The calculation formula of the clothing matching degree in step S32 is: Among them, f i is the matching result of the i-th feature, W i is the preset weight of the ith feature.
Citation Information
Patent Citations
Electric power place personnel identity recognition method and device based on deep learning
CN114495218A