Identity recognition method fusing multimode biological characteristics

Abstract

The invention discloses an identity recognition method fusing multimode biological characteristics, which belongs to the field of image analysis and comprises the steps of video detection, gait characteristic extraction, facial characteristic extraction, characteristic fusion, establishment of a human body biological characteristic identity library, identification of identity information of a person to be detected and the like. The gait feature and the facial feature of the detected pedestrian are extracted and fused at the same time, the accuracy and the high applicability of identity recognition of the old people can be remarkably improved, and the method is suitable for the following monitoring situations: 1, the old people are close to a monitoring camera, only face information exists, and no gait information exists; 2, the old people are far away from the monitoring camera, face information does not exist, and only gait information exists; and 3, the distance between the old people and the monitoring camera is moderate, and face information and gait information exist, so that the system is suitable for many scenes.

Description

Technical Field

[0001] The present invention relates to the field of image analysis, and in particular to an identity recognition method integrating multi-modal biometric features. Background Art

[0002] The global population is rapidly aging, and it is estimated that by 2050, the total number of people aged 60 and above will reach 2 billion. my country has the largest elderly population in the world, and the aging level continues to deepen. Aging-related degenerative diseases, neurological diseases and cognitive dysfunction induce behavioral safety problems in the elderly. With the development of social economy and changes in family structure, the phenomenon of empty nest and low birth rate is obvious. More and more families find it difficult to take care of the elderly alone, and the demand for institutional care is increasing.

[0003] Thanks to the rapid development of computer vision and deep learning algorithms, intelligent motion behavior analysis based on consumer-grade standard videos has become a reality. In order to monitor the safety of the campus to the greatest extent and detect accidents involving the elderly in a timely manner, nursing homes often equip high-definition cameras on a large scale and at a high density, which provides a solid hardware foundation for the use of video behavior analysis technology to achieve behavioral safety detection for the elderly.

[0004] Surveillance cameras have effective monitoring angles and distances. Therefore, when monitoring the daily behavior of the elderly in the park, faces are often not captured or are blurred or obscured. Only facial recognition technology is used to identify the elderly, with poor accuracy and scenario universality. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, one of the purposes of the present invention is to provide a method for identifying the identity of the elderly by fusing facial features with gait features, which has high accuracy and is applicable to many scenarios.

[0006] One of the purposes of the present invention is achieved by the following technical solution:

[0007] An identity recognition method integrating multi-modal biometric features comprises the following steps:

[0008] Video detection: Obtain camera surveillance video, use the pedestrian detection model to infer each image frame of the surveillance video stream, and obtain the confidence score of the existence of pedestrians and the confidence score of the existence of faces in the image frame. When the confidence score of pedestrians and / or the confidence score of faces is greater than the threshold, it is determined that pedestrians and / or faces are detected;

[0009] Gait feature extraction: When a pedestrian is detected, the effective image area where the pedestrian exists is cropped, and a pedestrian image sequence frame group is constructed. The cropped pedestrian image area is aligned, and the pedestrian image sequence is judged using the behavior recognition model to obtain the action state of the pedestrian in each frame image. The pedestrian image sequence in the "walking" action state is selected, and the pedestrian gait features are obtained based on the pedestrian image sequence in the "walking" action state;

[0010] Facial feature extraction: When a face is detected, the effective image area where the face exists is cropped, the face image is enhanced, the face image is diffused in multiple dimensions, the data fusion of multi-angle face information is completed, and the facial features of the fused image are extracted;

[0011] Feature fusion: Calculate the gait feature significance and the facial feature significance, determine the feature fusion weighting method according to the numerical values ​​of the gait feature significance and the facial feature significance, and fuse the gait features and facial features according to the determined feature fusion weighting method to obtain a feature fusion vector;

[0012] Establishing a human biometric identity database: Collecting each person's identity information and biological information to establish a human biometric identity database, the biological information includes facial images and gait information;

[0013] Identify the identity information of the person to be tested: match the biometric fusion features of the person to be tested with the biometric features in the identity database to identify the identity information of the person to be tested.

[0014] Furthermore, in the gait feature extraction step, the construction of the pedestrian image sequence frame group is specifically: using a pedestrian detection frame based on pedestrian detection model reasoning, giving a boundary filling pixel size, cropping the pedestrian image area, and constructing a pedestrian image sequence frame group under a continuous monitoring video stream.

[0015] Furthermore, in the gait feature extraction step, the image alignment operation performed on the cropped pedestrian image region is specifically performed by using feature matching and pixel interpolation to perform the image alignment operation.

[0016] Furthermore, in the gait feature extraction step, the video duration occupied by the pedestrian image sequence in the "walking" action state should be no less than 4 seconds.

[0017] Furthermore, in the gait feature extraction step, the gait features include time series features, spatial features and deep image features, the time series features include gait cycle and phase information, the gait cycle includes support time and swing time, the phase information includes the duration of each stage of the gait cycle and its changes, the spatial features include stride and step width, and the deep image features include high-dimensional, implicit, deep feature vectors extracted based on convolutional neural networks.

[0018] Furthermore, in the facial feature extraction step, the facial image is enhanced by: utilizing a pixel interpolation method and a brightness, contrast, and saturation adjustment method to achieve facial image enhancement.

[0019] Furthermore, in the feature fusion step, the gait feature prominence and the facial feature prominence are calculated based on the corresponding feature approximation, model confidence, image quality score and feature area coverage, respectively.

[0020] Furthermore, the facial feature prominence calculation formula is:

[0021] face_prominence=x1*face_similarity+x2*face_confidence+x3*face_imageQuality+x4*face_featureAreaCoverage

[0022] Among them, face_prominence is the prominence of facial features, face_similarity is the similarity of facial features, face_confidence is the confidence of facial model, face_imageQuality is the facial image quality score, face_featureAreaCoverage is the facial feature area coverage, and [x1,x2,x3,x4] are parameter influencing factors;

[0023] The calculation formula for the gait feature significance is:

[0024] gait_prominence=y1*gait_similarity+y2*gait_confidence+y3*gait_imageQuality+y4*gait_featureAreaCoverage

[0025] Among them, gait_prominence is the prominence of gait features, gait_similarity is the similarity of gait features, gait_confidence is the confidence of gait model, gait_imageQuality is the gait image quality score, gait_featureAreaCoverage is the gait feature area coverage, and [y1,y2,y3,y4] are parameter influencing factors.

[0026] Furthermore, in the feature fusion step, the feature fusion weighting method is any one of linear weighted average, geometric weighting, exponential weighting, and nonlinear function weighting.

[0027] Furthermore, when the feature fusion weighting method is linear weighted average, the feature fusion vector calculation formula is:

[0028] feature_fusion=w1*gait_prominence+w2*face_prominence;

[0029] Among them, feature_fusion is the feature fusion vector, w1 and w2 are weights, and the calculation formulas of w1 and w2 are as follows

[0030] Weight w1=gait_prominence / (gait_prominence+face_prominence)

[0031] Weight w2=face_prominence / (gait_prominence+face_prominence).

[0032] Compared with the prior art, the identity recognition method integrating multimodal biometrics of the present invention can significantly improve the accuracy and high applicability of elderly identity recognition by extracting and integrating the gait features and facial features of the detected pedestrians through the steps of video detection, gait feature extraction, facial feature extraction, feature fusion, establishing a human biometric identity database, and identifying the identity information of the person to be detected. It is suitable for the following monitoring situations, such as: 1. The elderly are close to the monitoring camera, and only facial information but no gait information exists; 2. The elderly are far away from the monitoring camera, and there is no facial information but only gait information exists; 3. The elderly are at a moderate distance from the monitoring camera, and both facial information and gait information exist, so there are many applicable scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of the identity recognition method integrating multi-modal biometrics of the present invention;

[0034] Figure 2 Flowchart for gait feature extraction;

[0035] Figure 3 Extract the flow chart for facial features;

[0036] Figure 4 Schematic diagram of the human biometric identity database. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may be another intermediate component through which it is fixed. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may be another intermediate component at the same time. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be another intermediate component at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0040] like Figure 1 As shown, the present invention provides an identity recognition method integrating multi-modal biometric features, comprising the following steps:

[0041] Video detection: Obtain camera surveillance video, use the pedestrian detection model to infer each image frame of the surveillance video stream, and obtain the confidence score of the existence of pedestrians and the confidence score of the existence of faces in the image frame. When the confidence score of pedestrians and / or the confidence score of faces is greater than the threshold, it is determined that pedestrians and / or faces are detected;

[0042] Gait feature extraction: When a pedestrian is detected, the effective image area where the pedestrian exists is cropped, and a pedestrian image sequence frame group is constructed. The cropped pedestrian image area is aligned, and the pedestrian image sequence is judged using the behavior recognition model to obtain the action state of the pedestrian in each frame image. The pedestrian image sequence in the "walking" action state is selected, and the pedestrian gait features are obtained based on the pedestrian image sequence in the "walking" action state;

[0043] Facial feature extraction: When a face is detected, the effective image area where the face exists is cropped, the face image is enhanced, the face image is diffused in multiple dimensions, the data fusion of multi-angle face information is completed, and the facial features of the fused image are extracted;

[0044] Feature fusion: Calculate the gait feature significance and the facial feature significance, determine the feature fusion weighting method according to the numerical values ​​of the gait feature significance and the facial feature significance, and fuse the gait features and facial features according to the determined feature fusion weighting method to obtain a feature fusion vector;

[0045] Establishing a human biometric identity database: Collecting each person's identity information and biological information to establish a human biometric identity database, the biological information includes facial images and gait information;

[0046] Identify the identity information of the person to be tested: match the biometric fusion features of the person to be tested with the biometric features in the identity database to identify the identity information of the person to be tested.

[0047] The specific steps of video detection are as follows: collect the RGB video stream captured by the surveillance camera, and use the pedestrian detection model to infer each image frame of the video stream to obtain the confidence score of the existence of pedestrians and the confidence score of the existence of faces in the image frame. The pedestrian detection model is any one of Mask R-CNN, YOLOv5s, and EfficientDet. The above-mentioned pedestrians refer to people who are moving. The confidence score of pedestrians is CS_person, and the confidence score of faces is CS_face. Considering the pixel quality of the surveillance video and environmental interference, the confidence score threshold of the existence of pedestrians (CT_person) and the confidence score threshold of the existence of faces (CT_face) are defined. If the confidence score of pedestrians and the confidence score of faces in the model inference results exceed the corresponding thresholds, it means that pedestrians or faces are detected. If no pedestrians or faces are detected in the surveillance video, the process ends.

[0048] The specific steps of gait feature extraction are as follows: if a pedestrian is detected (i.e. CS_person>CT_person), the gait biometric features of the pedestrian are extracted. Figure 2As shown in the figure, the effective image area where pedestrians exist is cropped to improve the coverage of the gait feature area. Based on the pedestrian detection frame inferred by the pedestrian detection model, the pedestrian image area is cropped with a given boundary filling pixel size, and the pedestrian image sequence frame group is constructed under the continuous monitoring video stream. The cropped pedestrian image area is image aligned to ensure that the pedestrian feature distribution of the image sequence is similar and the image pixel size is consistent. Image alignment uses means such as feature matching and pixel interpolation. Using behavior recognition models (such as I3D, X3D, slowfast, etc.), the pedestrian image sequence is judged to obtain the action state of the pedestrian in each frame image (such as sitting, standing, lying, walking, etc.), and the pedestrian image sequence in the "walking" action state is selected. To ensure that the image sequence covers the complete gait performance and characteristics, the video length occupied by the image sequence should be no less than 4 seconds. This sequence will be used as the input sample for subsequent pedestrian gait recognition. Models such as GaitSet and GaitCycle are used to obtain the temporal features, spatial features, and deep image features of pedestrian gait. Temporal features include gait cycle, phase information, etc. The gait cycle refers to the time required for a complete step of walking, including support time and swing time, reflecting the rhythmicity of gait; phase information refers to the duration and changes of each stage of the gait cycle, reflecting the specificity of gait. Spatial features include stride length, step width, etc. Deep image features refer to high-dimensional, implicit, deep feature vectors extracted based on convolutional neural networks.

[0049] The facial feature extraction steps are as follows: Figure 3 As shown in the figure, the effective image area with face is cropped to improve the coverage of facial feature area. Based on the face detection frame inferred by the pedestrian detection model, the face image area is cropped with a given boundary filling pixel size. The face image is enhanced by using pixel interpolation method, brightness, contrast and saturation adjustment method to solve the problems of unclear face image or external environment interference and improve image quality. The Zero123++ framework is used to realize the diffusion of face images from a single perspective to multiple perspectives, complete the data fusion and enhancement of multi-angle face information, and effectively improve the significance and specificity of facial features. Finally, FaceNet, DeepFace and other models are used to extract facial features and derive facial feature vectors.

[0050] The specific steps of feature fusion are as follows: some surveillance videos have clearer face images but less gait information; some surveillance videos have complete gait information but no face information (such as walking with the back to the camera); some surveillance videos have both gait information and face information. Formulating a scientific and effective feature fusion strategy can significantly improve the accuracy and applicability of elderly identity recognition. Considering the prominence of gait features and facial features, a weighted fusion strategy is formulated. The prominence of features can be calculated based on the following indicators, including but not limited to: feature similarity, model confidence, image quality score, feature area coverage, etc. The calculation formula is as follows, gait_prominence and face_prominence represent the prominence of gait features and facial features respectively, and [x1,x2,x3,x4], [y1,y2,y3,y4] are parameter influencing factors:

[0051] face_prominence=x1*face_similarity+x2*face_confidence+x3*face_imageQuality+x4*face_featureAreaCoverage

[0052] Among them, face_prominence is the prominence of facial features, face_similarity is the similarity of facial features, face_confidence is the confidence of facial model, face_imageQuality is the facial image quality score, face_featureAreaCoverage is the facial feature area coverage, and [x1,x2,x3,x4] are parameter influencing factors;

[0053] The calculation formula of gait feature significance is:

[0054] gait_prominence=y1*gait_similarity+y2*gait_confidence+y3*gait_imageQuality+y4*gait_featureAreaCoverage

[0055] Among them, gait_prominence is the prominence of gait features, gait_similarity is the similarity of gait features, gait_confidence is the confidence of gait model, gait_imageQuality is the gait image quality score, gait_featureAreaCoverage is the gait feature area coverage, and [y1,y2,y3,y4] are parameter influencing factors.

[0056] According to the specific values ​​of gait_prominence and face_prominence, the feature fusion weighting method is set to reflect the importance of gait features and facial features in identity recognition reasoning. Weighting includes but is not limited to the following methods: linear weighted average, geometric weighting, exponential weighting, nonlinear function weighting, etc. Taking linear weighting as an example, the feature fusion vector feature_fusion calculation formula is:

[0057] feature_fusion=w1*gait_prominence+w2*face_prominence,

[0058] Weight w1=gait_prominence / (gait_prominence+face_prominence)

[0059] Weight w2=face_prominence / (gait_prominence+face_prominence).

[0060] The specific steps to establish a human biometric identity database are:

[0061] The biometric identity database is the unique and only identity authentication credential for each elderly person as the feature matching benchmark for the person to be detected. First, the identity information of all the elderly in the park is counted, including but not limited to: ID number, age, contact information, height, weight, health information, etc. Then the biometric information of the elderly is collected, which is divided into two parts: personal facial image collection and gait video collection during normal walking. The collected biometric information must be of high quality to meet the requirements of subsequent identity feature extraction. High-quality biometric information can improve the performance of the identity recognition fusion model. When collecting facial images, standard collection processes and conditions must be designed to ensure the quality of facial images, such as meeting the following conditions: good lighting conditions, multi-angle face collection, and no decorations. When collecting gait information, it is necessary to ensure that the walking time of the elderly meets specific requirements and covers multiple complete gait cycles; in addition, gait videos from multiple perspectives can be collected, such as gait videos in the sagittal and coronal planes, so as to improve the significance, difference, and robustness of gait information. Then, we extract the biometric features of the elderly, extract facial features based on models such as FaceNet and DeepFace, and obtain gait timing features based on models such as GaitSet and GaitCycle. Finally, we complete the construction of the biometric database of the elderly in the park, such as Figure 4 As shown, each instance of the identity database is the biometric identity of each elderly person, and the identity instance includes the elderly person's identity information, gait feature vector, and facial feature vector.

[0062] The identity recognition method integrating multi-modal biometrics of the present invention can significantly improve the accuracy and high applicability of elderly identity recognition through the steps of video detection, gait feature extraction, facial feature extraction, feature fusion, establishment of a human biometric identity database, and identification of identity information of the person to be detected, while extracting and integrating the gait features and facial features of the detected pedestrians. The method is suitable for the following monitoring situations, such as: 1. The elderly are close to the monitoring camera, and only facial information but no gait information exists; 2. The elderly are far away from the monitoring camera, and no facial information but only gait information exists; 3. The elderly are at a moderate distance from the monitoring camera, and both facial information and gait information exist, so there are many applicable scenarios.

[0063] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of them belong to the protection scope of the present invention.

Claims

1. An identity recognition method integrating multi-modal biometric features, characterized in that: The following steps are involved: Video detection: Obtain camera surveillance video, use the pedestrian detection model to infer each image frame of the surveillance video stream, and obtain the confidence score of the existence of pedestrians and the confidence score of the existence of faces in the image frame. When the confidence score of pedestrians and / or the confidence score of faces is greater than the threshold, it is determined that pedestrians and / or faces are detected; Gait feature extraction: When a pedestrian is detected, the effective image area where the pedestrian exists is cropped, and a pedestrian image sequence frame group is constructed. The cropped pedestrian image area is aligned, and the pedestrian image sequence is judged using the behavior recognition model to obtain the action state of the pedestrian in each frame image. The pedestrian image sequence in the "walking" action state is selected, and the pedestrian gait features are obtained based on the pedestrian image sequence in the "walking" action state; Facial feature extraction: When a face is detected, the effective image area where the face exists is cropped, the face image is enhanced, the face image is diffused in multiple dimensions, the data fusion of multi-angle face information is completed, and the facial features of the fused image are extracted; Feature fusion: Calculate the gait feature significance and the facial feature significance, determine the feature fusion weighting method according to the numerical values ​​of the gait feature significance and the facial feature significance, and fuse the gait features and facial features according to the determined feature fusion weighting method to obtain a feature fusion vector; Establishing a human biometric identity database: Collecting each person's identity information and biological information to establish a human biometric identity database, the biological information includes facial images and gait information; Identify the identity information of the person to be tested: match the biometric fusion features of the person to be tested with the biometric features in the identity database to identify the identity information of the person to be tested.

2. The identity recognition method integrating multi-modal biometric features according to claim 1 is characterized in that: In the gait feature extraction step, the construction of the pedestrian image sequence frame group is specifically: using a pedestrian detection frame based on pedestrian detection model reasoning, giving a boundary filling pixel size, cropping the pedestrian image area, and constructing a pedestrian image sequence frame group under a continuous monitoring video stream.

3. The identity recognition method integrating multi-modal biometric features according to claim 1, characterized in that: In the gait feature extraction step, the image alignment operation performed on the cropped pedestrian image region is specifically performed by using feature matching and pixel interpolation.

4. The identity recognition method integrating multi-modal biometric features according to claim 1, characterized in that: In the gait feature extraction step, the video duration occupied by the pedestrian image sequence in the "walking" action state should be no less than 4 seconds.

5. The identity recognition method integrating multi-modal biometric features according to claim 1, characterized in that: In the gait feature extraction step, the gait features include time series features, spatial features and deep image features. The time series features include gait cycle and phase information. The gait cycle includes support time and swing time. The phase information includes the duration of each stage of the gait cycle and its changes. The spatial features include stride length and step width. The deep image features include high-dimensional, implicit, deep feature vectors extracted based on convolutional neural networks.

6. The identity recognition method integrating multi-modal biometric features according to claim 1, characterized in that: In the facial feature extraction step, the facial image is enhanced by using a pixel interpolation method and a brightness, contrast, and saturation adjustment method to achieve facial image enhancement.

7. The identity recognition method integrating multi-modal biometric features according to claim 1, characterized in that: In the feature fusion step, the gait feature prominence and the facial feature prominence are calculated according to the corresponding feature approximation, model confidence, image quality score and feature area coverage.

8. The identity recognition method integrating multi-modal biometric features according to claim 7, characterized in that: The facial feature prominence calculation formula is: face_prominence=x1*face_similarity+x2*face_confidence+x3*face_imageQuality+x4*face_featureAreaCoverage Among them, face_prominence is the prominence of facial features, face_similarity is the similarity of facial features, face_confidence is the confidence of facial model, face_imageQuality is the facial image quality score, face_featureAreaCoverage is the facial feature area coverage, and [x1,x2,x3,x4] are parameter influencing factors; The calculation formula for the gait feature significance is: gait_prominence=y1*gait_similarity+y2*gait_confidence+y3*gait_imageQuality+y4*gait_featureAreaCoverage Among them, gait_prominence is the prominence of gait features, gait_similarity is the similarity of gait features, gait_confidence is the confidence of gait model, gait_imageQuality is the gait image quality score, gait_featureAreaCoverage is the gait feature area coverage, and [y1,y2,y3,y4] are parameter influencing factors.

9. The identity recognition method integrating multi-modal biometric features according to claim 8, characterized in that: In the feature fusion step, the feature fusion weighting method is any one of linear weighted average, geometric weighting, exponential weighting, and nonlinear function weighting.

10. The identity recognition method integrating multi-modal biometric features according to claim 9, characterized in that: When the feature fusion weighting method is linear weighted average, the feature fusion vector calculation formula is: feature_fusion=w1*gait_prominence+w2*face_prominence; Among them, feature_fusion is the feature fusion vector, w1 and w2 are weights, and the calculation formulas of w1 and w2 are as follows Weight w1=gait_prominence / (gait_prominence+face_prominence) Weight w2=face_prominence / (gait_prominence+face_prominence).

Images