An identity recognition method and system based on adaptive fusion of multiple biometric features

Gait features are extracted through convolutional neural networks and long-term memory networks, combined with face recognition algorithms with adaptive weighted HOG features, and SVM's adaptive weighted fusion algorithm solves the gait and face multi-biometric identity in complex backgrounds. The poor identification of problem of poor adaptability achieves higher recognition accuracy and robustness.

CN114519899BActive Publication Date: 2024-11-19YICHUN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210166085.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-23
Publication Date
2024-11-19
Estimated Expiration
2042-02-23

AI Technical Summary

Technical Problem

The prior art has poor adaptability to the identification method of gait and face multi-biometric identity in complex backgrounds, resulting in inaccurate identification results and it is difficult to maintain high accuracy when factors such as light, distance, expression, and posture change.

Method used

The gait features are extracted using convolutional neural network and long-term memory network, and combined with the face recognition algorithm of adaptive weighted HOG features, the gait and face features are fused through the SVM's adaptive weighted fusion algorithm to generate identity recognition results.

Benefits of technology

In a complex context, the accuracy and adaptability of identity recognition are improved, the robustness and recognition speed of the system are enhanced, and the influence of interference factors such as lighting, clothing changes and items carried are effectively resisted.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114519899B_ABST
    Figure CN114519899B_ABST
Patent Text Reader

Abstract

The present invention relates to an identity recognition method and system based on adaptive fusion of multiple biometric features. The method comprises: obtaining a video sequence within a gait cycle from a video sequence to be recognized to obtain a periodic video; obtaining a PAF matrix corresponding to each frame of a person to be recognized in the periodic video to obtain a walking feature vector map; inputting the walking feature vector map into a first convolutional neural network and a long short-term memory neural network to obtain a spatial feature matrix and a temporal feature matrix; fusing the spatial feature matrix and the temporal feature matrix to obtain a gait feature matrix of the person to be recognized; using an adaptive weighted HOG feature face recognition algorithm to process a face image to be recognized to obtain a face feature matrix; using an adaptive weighted fusion algorithm based on SVM to process a face, a gait feature matrix and a standard image to obtain an identity recognition result of the person to be recognized. The present invention has strong adaptability and can improve the accuracy of identity recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of identity recognition technology, and in particular to an identity recognition method and system based on adaptive fusion of multiple biological features. Background Art

[0002] Face recognition and gait recognition are currently the most popular and widely used biometric identity recognition technologies. They are similar in application scenarios and applicable conditions, so there is a premise for integration. In terms of influencing factors, face recognition is easily affected by factors such as lighting, distance, expression, posture, etc., while gait recognition is easily affected by age, walking conditions, physical condition, psychological factors, backpacks, etc. Since these two biometric influencing factors do not interfere with each other when the external environment changes. Therefore, it can be seen that the two biometric features of gait and face can not only be obtained simultaneously from a gait video sequence, but can also be applied to the same occasion and have certain complementarity. Therefore, multi-biometric identity recognition that combines face features and gait features has become a new research direction in the field of computer vision and pattern recognition.

[0003] In the existing technology, the research on multi-biometric identity recognition that integrates gait and face mostly uses frontal face and side gait images, which are two angles that are easier to extract corresponding features for recognition research. Only a few use side face and frontal gait or other biometric images of some other angles for recognition. In terms of fusion algorithms, there are four fusion algorithms: data layer, feature layer, matching layer and decision layer. The above methods are all carried out under normal circumstances and single background conditions, which makes the above-mentioned identity recognition methods have poor adaptability in complex backgrounds and inaccurate recognition results. Therefore, further research is needed to improve the adaptability of identity recognition methods in complex backgrounds to improve recognition accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide an identity recognition method and system based on adaptive fusion of multiple biometric features, which has strong adaptability and can improve the accuracy of identity recognition under complex background conditions.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An identity recognition method based on adaptive fusion of multiple biometric features, comprising:

[0007] Acquire a video sequence to be identified, and acquire a video sequence within a gait cycle from the video sequence to be identified to obtain a periodic video; the video sequence to be identified includes a person to be identified;

[0008] Obtaining the PAF matrix corresponding to each frame of the person to be identified in the periodic video to obtain a walking feature vector map; the PAF matrix includes the PAFs of the limbs and the PAF of the trunk;

[0009] Inputting the walking feature vector graph into a first convolutional neural network and a long short-term memory neural network respectively to obtain a spatial feature matrix and a temporal feature matrix;

[0010] Performing feature fusion on the spatial feature matrix and the temporal feature matrix to obtain a gait feature matrix of the person to be identified;

[0011] Acquire a face image of a person to be identified in the video sequence to be identified, and use a face recognition algorithm with adaptive weighted HOG features to process the face image of the person to be identified to obtain a face feature matrix;

[0012] The face feature matrix, the gait feature matrix and the standard image in the image database are processed by an adaptive weighted fusion algorithm based on SVM to obtain the identity recognition result of the person to be recognized.

[0013] Optionally, the method for determining the gait cycle is:

[0014] The video sequence to be identified is input into a trained second convolutional neural network to obtain the gait cycle of the person to be identified.

[0015] Optionally, before the adaptive weighted fusion algorithm is used to process the facial feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized, the method further includes:

[0016] Performing dimensionality reduction processing on the facial feature matrix to obtain a reduced-dimensional facial feature matrix.

[0017] Optionally, the step of acquiring a facial image of a person to be identified in the video sequence to be identified and processing the facial image of the person to be identified using a face recognition algorithm with adaptive weighted HOG features to obtain a facial feature matrix specifically includes:

[0018] Acquire a facial image of the person to be identified in the video sequence to be identified;

[0019] Dividing the facial image of the person to be identified into cell units of the same size;

[0020] Calculate the gradient histogram of each cell unit according to the direction and amplitude of the pixel gradient in each cell unit;

[0021] The gradient histogram of all cell units is determined as the face feature matrix.

[0022] Optionally, the adopting of an adaptive weighted fusion algorithm based on SVM to process the facial feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized specifically includes:

[0023] Projecting the gait feature matrix and the face feature matrix onto the coordinate system formed by the base image to obtain a gait projection matrix and a face projection matrix;

[0024] The gait Euclidean distances between the image in the video sequence to be identified and each standard image in the image database are calculated according to the gait projection matrix to obtain a gait Euclidean distance array; the face Euclidean distances between the image in the video sequence to be identified and each standard image in the image database are calculated according to the face projection matrix to obtain a face Euclidean distance array;

[0025] Calculating gait confidence and face confidence according to the gait Euclidean distance array, the face Euclidean distance array, the rejection rate of the gait feature matrix and the rejection rate of the face feature matrix;

[0026] Calculating a gait fusion weight according to the gait confidence, and calculating a face fusion weight according to the face confidence;

[0027] The identity recognition result of the person to be recognized is obtained according to the gait fusion weight and the face fusion weight.

[0028] An identity recognition system based on adaptive fusion of multiple biometric features, comprising:

[0029] A video sequence acquisition module is used to acquire a video sequence to be identified, and acquire a video sequence within a gait cycle from the video sequence to be identified to obtain a periodic video; the video sequence to be identified includes a person to be identified;

[0030] A PAF matrix acquisition module is used to obtain the PAF matrix corresponding to each frame of the person to be identified in the periodic video to obtain a walking feature vector map; the PAF matrix includes the PAF of the limbs and the PAF of the trunk;

[0031] A spatiotemporal feature determination module, used for inputting the walking feature vector map into a first convolutional neural network and a long short-term memory neural network to obtain a spatial feature matrix and a temporal feature matrix;

[0032] A gait feature matrix determination module, used for fusing the spatial feature matrix and the temporal feature matrix to obtain the gait feature matrix of the person to be identified;

[0033] A facial feature matrix determination module is used to obtain a facial image of a person to be identified in the video sequence to be identified and process the facial image of the person to be identified using a face recognition algorithm with adaptive weighted HOG features to obtain a facial feature matrix;

[0034] The identity recognition module is used to process the face feature matrix, the gait feature matrix and the standard image in the image database using an adaptive weighted fusion algorithm based on SVM to obtain the identity recognition result of the person to be recognized.

[0035] Optionally, the video sequence acquisition module includes:

[0036] The gait cycle determination submodule is used to input the video sequence to be identified into the trained second convolutional neural network to obtain the gait cycle of the person to be identified.

[0037] Optionally, the identity recognition system based on adaptive fusion of multiple biometric features further includes:

[0038] The dimension reduction module is used to perform dimension reduction processing on the facial feature matrix to obtain a reduced-dimensional facial feature matrix.

[0039] Optionally, the facial feature matrix determination module specifically includes:

[0040] An acquisition submodule, used for acquiring a face image of the person to be identified in the video sequence to be identified;

[0041] A cell unit determination submodule, used for dividing the face image of the person to be identified into cell units of the same size;

[0042] A gradient histogram determination submodule is used to calculate the gradient histogram of each cell unit according to the direction and amplitude of the pixel gradient in each cell unit;

[0043] The face feature matrix determination submodule is used to determine the gradient histograms of all cell units as the face feature matrix.

[0044] Optionally, the identity recognition module specifically includes:

[0045] A projection matrix determination submodule, used for projecting the gait feature matrix and the face feature matrix onto the coordinate system formed by the base image to obtain a gait projection matrix and a face projection matrix;

[0046] The Euclidean distance array determination submodule is used to calculate the gait Euclidean distance between the image in the video sequence to be identified and each standard image in the image database according to the gait projection matrix to obtain a gait Euclidean distance array; calculate the face Euclidean distance between the image in the video sequence to be identified and each standard image in the image database according to the face projection matrix to obtain a face Euclidean distance array;

[0047] A confidence determination submodule, used to calculate gait confidence and face confidence according to the gait Euclidean distance array, the face Euclidean distance array, the rejection rate of the gait feature matrix and the rejection rate of the face feature matrix;

[0048] A fusion weight determination submodule, used to calculate a gait fusion weight according to the gait confidence, and calculate a face fusion weight according to the face confidence;

[0049] The identity recognition submodule is used to obtain the identity recognition result of the person to be recognized according to the gait fusion weight and the face fusion weight.

[0050] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention obtains a video sequence within a gait cycle from a video sequence to be identified to obtain a periodic video; obtains the PAF matrix corresponding to each frame of the person to be identified in the periodic video to obtain a walking feature vector graph; inputs the feature vector graph into a first convolutional neural network and a long short-term memory neural network respectively to obtain a spatial feature matrix and a temporal feature matrix; performs feature fusion on the spatial feature matrix and the temporal feature matrix to obtain a gait feature matrix of the person to be identified; uses a face recognition algorithm with adaptive weighted HOG features to process the face image of the person to be identified to obtain a face feature matrix; uses an adaptive weighted fusion algorithm based on SVM to process the face feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be identified, and combines the HOG algorithm with the SVM so that it has strong adaptability in complex background conditions and can improve the accuracy of identity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A flowchart of an identity recognition method based on adaptive fusion of multiple biometric features provided by an embodiment of the present invention;

[0053] Figure 2A gait recognition flow chart provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a key frame of a gait cycle in a human gait sequence;

[0055] Figure 4 A waveform diagram output when determining a gait cycle provided by an embodiment of the present invention;

[0056] Figure 5 A flow chart for determining a human gait cycle provided by an embodiment of the present invention;

[0057] Figure 6 A PAF diagram after selection provided by an embodiment of the present invention;

[0058] Figure 7 A feature vector diagram within a gait cycle provided by an embodiment of the present invention;

[0059] Figure 8 A flow chart of face recognition under video surveillance provided by an embodiment of the present invention;

[0060] Fig. 9 Schematic diagram of HOG algorithm;

[0061] Fig.10 This is the basic flow chart of multi-feature fusion. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] The main research contents of the identity recognition method based on adaptive fusion of multiple biometric features provided by the embodiment of the present invention are divided into the following parts: (1) studying a gait recognition method combining a convolutional neural network and a long short-term memory network; (2) studying a face recognition method based on a HOG-NMF feature extraction algorithm; (3) extracting gait and face features, using an adaptive weighted fusion method based on a decision layer to fuse and recognize the extracted features, performing target detection based on a deep learning HOG combined with an SVM algorithm, fusing gait and face recognition, and using an adaptive dynamic weighted fusion method based on a decision layer to make the method have strong robustness, higher recognition and faster recognition speed even in complex backgrounds. The specific steps are as follows: Figure 1 As shown, including:

[0065] Step 101: obtaining a video sequence to be identified, and obtaining a video sequence within a gait cycle from the video sequence to be identified to obtain a periodic video; the video sequence to be identified includes a person to be identified.

[0066] Step 102: Obtain the PAF matrix corresponding to each frame of the person to be identified in the periodic video to obtain a walking feature vector map; the PAF matrix includes the PAFs of the limbs and the PAF of the trunk.

[0067] Step 103: Input the walking feature vector map into the first convolutional neural network and the long short-term memory neural network respectively to obtain a spatial feature matrix and a temporal feature matrix.

[0068] Step 104: Fusing the spatial feature matrix and the temporal feature matrix to obtain a gait feature matrix of the person to be identified.

[0069] Step 105: Obtain a facial image of a person to be identified in the video sequence to be identified and use a face recognition algorithm with adaptive weighted HOG features to process the facial image of the person to be identified to obtain a facial feature matrix.

[0070] Step 106: Adopting an adaptive weighted fusion algorithm based on SVM to process the facial feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized.

[0071] In practical applications, the method for determining the gait cycle is:

[0072] The video sequence to be identified is input into a trained second convolutional neural network to obtain the gait cycle of the person to be identified.

[0073] In practical applications, inputting the video sequence to be identified into the trained second convolutional neural network to obtain the gait cycle of the person to be identified specifically includes:

[0074] Extracting a pedestrian region in each frame of the video sequence to be identified;

[0075] Crop the pedestrian regions in all frame images into images of the same size;

[0076] All cropped images are input into the trained second convolutional neural network to obtain gait waveforms;

[0077] Determine the gait period based on the gait waveform.

[0078] In practical applications, before the adaptive weighted fusion algorithm is used to process the face feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized, the following steps are also included:

[0079] Performing dimensionality reduction processing on the facial feature matrix to obtain a reduced-dimensional facial feature matrix.

[0080] In practical applications, the face feature matrix is ​​subjected to dimensionality reduction processing to obtain the reduced-dimensional face feature matrix, specifically comprising:

[0081] The absolute value of the facial feature matrix is ​​taken to obtain an absolute value matrix.

[0082] The absolute value matrix is ​​subjected to rank NMF decomposition to obtain a basis matrix and a coefficient matrix.

[0083] Normalize each column vector of the basis matrix and the characteristic matrix to obtain a normalized column vector.

[0084] Cascade all normalized column vectors to obtain the reduced-dimensional face feature matrix.

[0085] In practical applications, the step of obtaining a facial image of a person to be identified in the video sequence to be identified and processing the facial image of the person to be identified using a face recognition algorithm with adaptive weighted HOG features to obtain a facial feature matrix specifically includes:

[0086] Obtain a face image of the person to be identified in the video sequence to be identified.

[0087] The facial image of the person to be identified is divided into cell units of the same size.

[0088] The gradient histogram of each cell unit is calculated according to the direction and amplitude of the pixel gradient in each cell unit.

[0089] The gradient histogram of all cell units is determined as the face feature matrix.

[0090] In practical applications, the adaptive weighted fusion algorithm based on SVM is used to process the face feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized, which specifically includes:

[0091] The gait feature matrix and the face feature matrix are respectively projected onto the coordinate system formed by the base image to obtain a gait projection matrix and a face projection matrix.

[0092] The gait Euclidean distances between the images in the video sequence to be identified and each standard image in the image database are calculated according to the gait projection matrix to obtain a gait Euclidean distance array; the face Euclidean distances between the images in the video sequence to be identified and each standard image in the image database are calculated according to the face projection matrix to obtain a face Euclidean distance array.

[0093] The gait confidence and the face confidence are calculated according to the gait Euclidean distance array, the face Euclidean distance array, the rejection rate of the gait feature matrix and the rejection rate of the face feature matrix.

[0094] A gait fusion weight is calculated according to the gait confidence, and a face fusion weight is calculated according to the face confidence.

[0095] The identity recognition result of the person to be recognized is obtained according to the gait fusion weight and the face fusion weight.

[0096] The embodiment of the present invention further provides an identity recognition system based on adaptive fusion of multiple biometric features corresponding to the above method, comprising:

[0097] The video sequence acquisition module is used to acquire a video sequence to be identified, and acquire a video sequence within a gait cycle from the video sequence to be identified to obtain a periodic video; the video sequence to be identified includes a person to be identified.

[0098] The PAF matrix acquisition module is used to obtain the PAF matrix corresponding to each frame of the person to be identified in the periodic video to obtain a walking feature vector map; the PAF matrix includes the PAF of the limbs and the PAF of the trunk.

[0099] The spatiotemporal feature determination module is used to input the walking feature vector map into the first convolutional neural network and the long short-term memory neural network to obtain a spatial feature matrix and a temporal feature matrix.

[0100] The gait feature matrix determination module is used to perform feature fusion on the spatial feature matrix and the temporal feature matrix to obtain the gait feature matrix of the person to be identified.

[0101] The facial feature matrix determination module is used to obtain the facial image of the person to be identified in the video sequence to be identified and use the face recognition algorithm of adaptive weighted HOG features to process the facial image of the person to be identified to obtain the facial feature matrix.

[0102] The identity recognition module is used to process the face feature matrix, the gait feature matrix and the standard image in the image database using an adaptive weighted fusion algorithm based on SVM to obtain the identity recognition result of the person to be recognized.

[0103] As an optional implementation, the video sequence acquisition module includes:

[0104] The gait cycle determination submodule is used to input the video sequence to be identified into the trained second convolutional neural network to obtain the gait cycle of the person to be identified.

[0105] As an optional implementation, the identity recognition system based on adaptive fusion of multiple biometric features further includes:

[0106] The dimension reduction module is used to perform dimension reduction processing on the facial feature matrix to obtain a reduced-dimensional facial feature matrix.

[0107] As an optional implementation manner, the facial feature matrix determination module specifically includes:

[0108] The acquisition submodule is used to acquire the face image of the person to be identified in the video sequence to be identified.

[0109] The cell unit determination submodule is used to divide the facial image of the person to be identified into cell units of the same size.

[0110] The gradient histogram determination submodule is used to calculate the gradient histogram of each cell unit according to the direction and amplitude of the pixel gradient in each cell unit.

[0111] The face feature matrix determination submodule is used to determine the gradient histograms of all cell units as the face feature matrix.

[0112] As an optional implementation, the identity recognition module specifically includes:

[0113] The projection matrix determination submodule is used to project the gait feature matrix and the face feature matrix onto the coordinate system formed by the base image to obtain a gait projection matrix and a face projection matrix.

[0114] The Euclidean distance array determination submodule is used to calculate the gait Euclidean distance between the image in the video sequence to be identified and each standard image in the image database according to the gait projection matrix to obtain a gait Euclidean distance array; and calculate the face Euclidean distance between the image in the video sequence to be identified and each standard image in the image database according to the face projection matrix to obtain a face Euclidean distance array.

[0115] The confidence determination submodule is used to calculate the gait confidence and the face confidence according to the gait Euclidean distance array, the face Euclidean distance array, the rejection rate of the gait feature matrix and the rejection rate of the face feature matrix.

[0116] The fusion weight determination submodule is used to calculate the gait fusion weight according to the gait confidence and calculate the face fusion weight according to the face confidence.

[0117] The identity recognition submodule is used to obtain the identity recognition result of the person to be recognized according to the gait fusion weight and the face fusion weight.

[0118] The embodiment of the present invention also provides a more specific identity recognition method based on adaptive fusion of multiple biometric features:

[0119] Step 1: Gait recognition

[0120] In order to improve the accuracy of gait recognition under the influence of many interferences such as clothing changes and the presence of objects under multiple perspectives, this embodiment adopts a gait recognition method that combines a convolutional neural network and a long short-term memory network. This method not only retains the gait space and practical information, but also avoids redundant information, which is conducive to the learning and training of gait features and improves the recognition rate, robustness and real-time performance of the detection method. The corresponding process is as follows Figure 2 As shown, gait cycle detection is obtained based on the pedestrian video sequence, and then a human walking feature vector map is obtained, and then feature extraction is performed (convolutional neural network is used for spatial feature extraction, and long short-term memory network is used for temporal feature extraction), and then gait matching results are performed, and finally fusion input is performed. The specific steps are as follows:

[0121] 1) Gait cycle detection

[0122] This embodiment uses a convolutional neural network to complete gait cycle detection. The convolutional neural network algorithm has outstanding feature learning capabilities and can automatically extract the periodic features of the gait profile sequence. Compared with the single artificial feature in the traditional method, this method can achieve better results and robustness, thereby improving the recognition accuracy of subsequent gait recognition in complex scenarios.

[0123] The definition of the gait cycle is the time from one foot touching the ground to the same foot touching the ground again. There are 4 key frames. The first frame is when the two feet are together and the left foot is ready to move. The second frame is when the two feet are together and the right foot is ready to move. The third frame is when the two feet are separated and the right foot is behind. The fourth frame is when the two feet are separated and the left foot is behind. The extraction of key frames greatly reduces the amount of calculation of subsequent gait features. Figure 3 shown.

[0124] First, the video sequence is cropped and adjusted to standardize, the purpose of which is to avoid the impact of the distance and angle between the person and the camera on the outline. Then the pedestrian area is extracted by the top and bottom pixels of the processed image sequence, and its center of gravity is calculated. Each frame is cropped into a 72×96 pedestrian outline image using the center of gravity, outline height and aspect ratio to form a gait sequence. The gait sequence is then sent to the trained second convolutional neural network, and the output waveform is finally as follows: Figure 4 As shown, filtering is performed to obtain a complete gait cycle. The specific process is as follows Figure 5 shown.

[0125] For the input gait sequence {D 1 ,D 2 ,···,D n}After standardization, we get D = {D norm1 ,D norm2 ,···,D normn} standard gait sequence, and convolve this sequence through the trained second convolutional neural network S, B = S * D to obtain the corresponding output classification result {B 1 ,B 2 ,···,B n}, where the gait cycle is modeled into 4 categories (key frames), when B 1 With B i The classification results are the same and {B 2 ,B 3 ,···,B i-1} contains other classification results in order, that is, {B 1 ,B 2 ,···,B i} is a gait cycle. Since the number of frames in the video ranges from dozens to hundreds, it is only necessary to detect a gait cycle of human motion for subsequent feature extraction. At the same time, each person uses the features of a gait cycle sequence as recognition, reducing the diversity of features and improving the recognition rate. n represents the nth pedestrian profile, D normn Indicates passing mark

[0126] The normalized nth pedestrian silhouette.

[0127] 2) Data selection and PAF extraction

[0128] Part affinity field (PAF) is a matrix that provides position and orientation information of each part of the human body. They appear in pairs: for each part, there is a PAF in the x direction, denoted as x-PAF, and a PAF in the y direction, denoted as y-PAF. PAF is a 3D matrix of size W×H×C, where W is the width, H is the height, and C is the number of layers.

[0129] In this embodiment, the width and height are both 46, and the number of layers is 57. The first 18 layers are the positions of 18 key points of the human body, and the 19th layer is the background. The above 19 layers constitute a heat map. The last 38 layers are PAFs, among which the odd-numbered layers are x-PAFs in the x direction, and the even-numbered layers are y-PAFs in the y direction. These 38 layers constitute a PAF matrix of size 46×46×38.

[0130] In gait videos, since there is subsequent face detection for the head and the head often shakes irregularly, PAF extraction of the head becomes difficult and meaningless.

[0131] This embodiment ignores the gait feature information of the head and mainly considers the limbs and torso, such as Figure 6 As shown in Figure 2, the number of layers of the head-related PAF removed from the original PAF becomes 24, and the PAF matrix is ​​reduced to 46×46×24.

[0132] 3) Formation of human walking characteristic vector diagram

[0133] The matrix size of each frame of the reduced PAF is 46×46×24, and the length of the video sequence along the time axis is T c , then the size of the walking feature vector graph is T c ×46×46×24. Figure 7 shown.

[0134] 4) Feature extraction and gait matching

[0135] Design space and time networks to learn features of walking feature vector graphs and perform gait matching. The first convolutional neural network transforms the action feature matrix M p After 4 convolutions and pooling in the first convolutional neural network branch, it is converted into a one-dimensional motion feature vector in the Flatten layer. The long short-term memory neural network branch converts the motion constraint matrix M q After passing through two layers of long short-term memory networks, it is converted into a one-dimensional motion constraint vector in the Flatten layer; finally, the two one-dimensional vectors are merged to obtain the gait matching output sequence.

[0136] ① Input layer: The input matrix size is Tc ×46×46×24.

[0137] ② The first convolutional neural network (spatial feature extraction): Input the walking feature vector map into the first convolutional neural network to obtain the spatial feature matrix, perform convolution calculation on the input, and adjust the position of batch normalization and linear rectification functions. Then perform a convolution operation to adjust the input channel size, and perform two dimensionality reductions in the 3×3 convolution layer. The output spatial feature shape is T c ×12×12×512.

[0138] ③ Long short-term memory neural network (temporal feature extraction): Input the walking feature vector map into the long short-term memory neural network to obtain the temporal feature matrix, and reshape the output of the first full connection into the input form of a single-layer unidirectional long short-term memory neural network. Suppose the number of training set samples of this network is n, and the number of hidden layer nodes is 512, then its structure is (n, T c , 512), and finally the spatial and temporal feature matrices are fused to obtain the gait feature matrix, that is, the gait matching sequence, and the output is the gait matching sequence (n, 512).

[0139] Step 2: Face recognition

[0140] A complete face recognition system based on surveillance video can be divided into three modules: face detection, feature extraction and classification recognition. The flowchart is as follows: Figure 8 As shown, firstly, video capture is performed, and then face detection is performed to determine whether there is a face. If not, the process returns to the face detection step. If so, feature extraction and dimensionality reduction are performed in sequence, and the optimal features and classification recognition are selected to obtain face features.

[0141] This embodiment proposes a face recognition algorithm with good robustness and adaptive weighted HOG features through research and analysis on the different contribution rates of HOG features and local facial features to recognition.

[0142] The main purpose of the HOG algorithm is to calculate the gradient of the image and count the gradient direction and gradient size of the image. The edge and gradient features it extracts can well capture the characteristics of the local shape, and because the image is gamma-corrected and normalized in a cell-based manner, it has good invariance to geometric and optical changes, and the transformation or rotation has little effect on sufficiently small areas. Therefore, the HOG algorithm has good robustness to linear transformations such as expression changes, lighting, and scene changes.

[0143] 1. The process of extracting facial features using the HOG algorithm is as follows:

[0144] 1) Consider the entire captured face image as a feature collection window.

[0145] 2) Set a sliding area block in this acquisition window and divide the block into several cells of uniform size, such as Fig. 9 (a) shown.

[0146] 3) The gradient histogram of the current cell is calculated according to the direction and amplitude of the gradient of each pixel in the cell unit, and then the gradient histograms of all cells are combined as the gradient histogram of the current block. In order to further reduce the influence of illumination, background, expression, posture, etc., the histogram needs to be normalized. This embodiment adopts L2-norm normalization.

[0147] 4) Then move the block horizontally and vertically along the current window at a certain step, such as Fig. 9 As shown in (b), the gradient histogram of the current block is counted every time a certain distance is translated, and finally the gradient histograms of all blocks are cascaded as the final face feature.

[0148] The steps for HOG feature extraction for a face image with a scale of 80×120 are as follows:

[0149] ① Set the size of the sliding area block to 20×20.

[0150] ② Each block is evenly divided into 16 cells, each of which is 5×5 in size. Let the grayscale value of any pixel (x, y) in the cell be H(x, y), and calculate its horizontal gradient H x (x,y) and the vertical gradient H y (x,y) then the gradient magnitude I(x,y) and gradient direction of the pixel point (x,y) They are

[0151] and

[0152] ③ Divide the gradient direction of each cell unit equally into 10 signed directions, the size of each direction is 18°, and the dimension size of each cell unit histogram is 10 dimensions.

[0153] ④ Then, the gradient histograms of the 16 cells in the current block are cascaded to obtain a 160-dimensional histogram that represents the gradient histogram features of the current block, and normalization is performed at the same time.

[0154] ⑤ Then slide the block on the face image with a horizontal and vertical step value of 20, repeat the above steps to obtain the gradient histogram of the sliding block, and finally concatenate the gradient histograms of all blocks as the final gradient histogram feature of the face. The final gradient histogram dimension size is 4×6×160=3840 dimensions.

[0155] 2. Feature extraction and dimensionality reduction processing

[0156] The dimensionality of HOG feature extraction is relatively large. For example, the feature vector formed by a typical 120×80 resolution image is 3840-dimensional. This embodiment uses non-negative matrix factorization (NMF) to reduce the dimensionality and form a HOG-NMF feature sequence to meet the real-time requirements of the system. NMF is an optimization process under a certain cost function constraint. It is a matrix decomposition method under the constraint that all elements in the matrix are non-negative numbers. Its approximate solution can be obtained through iterative operations. The specific process is as follows:

[0157] 1) Let F be a HOG feature vector of length l (the face feature matrix before dimensionality reduction), take its absolute value and convert it into an m×n matrix G, where l=m×n and m>n.

[0158] 2) According to the formula G = ZY T Perform NMF decomposition of rank r, r<<m, where Z and Y are the non-negative basis matrix and coefficient matrix of m×r and n×r respectively.

[0159] 3) For each column vector h of the Z and Y matrices i Do normalization, that is, h i =h i / ||h i ||.

[0160] 4) Finally, all normalized column vectors h i The HOG-NMF features are cascaded together to form the final face features.

[0161] This embodiment takes 120×80 resolution as an example. When HOG is used to extract features, the output is 3840 dimensions, i.e., l=3840, m=480, n=8, r=2. After NMF dimension reduction, the length of the feature becomes 986, and the feature dimension is reduced by 75%. The base matrix and coefficient matrix obtained by low-rank decomposition contain the main features of the original matrix, so the HOG-NMF feature inherits the excellent characteristics of HOG well, and greatly improves the real-time performance of the subsequent fusion algorithm.

[0162] Step 3: Identity Identification

[0163] The information fusion methods of multiple biometric features can be mainly divided into four types: data layer fusion, feature layer fusion, matching layer fusion and decision layer fusion. Fig.10 In the process of fusing multiple features at different levels, this embodiment adopts an adaptive weighted fusion method based on the decision layer. By adaptively assigning weights to the distance matching values ​​of each biometric feature at the decision layer to obtain the optimal joint matching score, the best information fusion effect can be achieved.

[0164] Decision-level image fusion is a higher level of information fusion, and its results will provide a basis for various controls or decisions. This method first processes each source data to obtain judgment and recognition results respectively. The fusion center coordinates these results according to certain criteria and the credibility of each data source decision to obtain the best decision result. The decision-level fusion method is mainly based on the cognitive model method, which requires a large database and expert decision system for analysis, reasoning, recognition and judgment. This kind of fusion has good real-time performance and certain fault tolerance.

[0165] Decision-level fusion is a high-level fusion that first processes different features independently to obtain recognition results, then uses a fusion algorithm to synthesize the multiple recognition results to obtain the final result. Decision-level fusion has the advantages of high flexibility, strong anti-interference ability, low processing cost, and less reliance on sensors.

[0166] The SVM (support vector machine) algorithm is one of the best methods currently used in intelligent fusion classification. It constructs a model with the best generalization ability based on limited sample information, that is, it seeks the best trade-off between model complexity and learning ability. Many of its unique advantages are reflected in both linear and nonlinear classification problems.

[0167] Based on the SVM algorithm, in order to improve the robustness of the fusion algorithm of the decision layer, this embodiment uses the discrimination results (distance information) of the feature layers of different modes to assist in guiding the fusion of the decision layer, introduces the rejection rate to perform weighted judgment of the confidence, and forms an adaptive weighted fusion algorithm to obtain the recognition result.

[0168] Decision layer fusion algorithm steps:

[0169] 1) The one-dimensional feature column vector c of the gait feature obtained in step 1 and the facial feature obtained in step 2 is applied to the base image (first read the training image database of the face and gait to obtain the training image matrix V = (V 1 , V 2 ....V n ), n is the number of images in the training library, column vector V iRepresents a training image. Through the NMF transformation formula V=AH, a new matrix is ​​obtained, where H is the projection coefficient matrix, and each column in the A matrix is ​​a basis image. The projection is performed on the coordinate system formed by r=c·A to obtain the projection matrix r (face projection matrix and gait projection matrix), and then the Euclidean distance (face Euclidean distance and gait Euclidean distance) between the image to be identified and all n existing training images (images or videos input into the created training library, or existing finished product libraries) is calculated according to the projection matrix, and the Euclidean distance of all existing training images is determined as a Euclidean distance array (gait Euclidean distance array and face Euclidean distance array). Among them, the gait Euclidean distance array is denoted as Ob={Ob 1 ,Ob 2 ···Ob n}, the face Euclidean distance array is recorded as Of = {Of 1 ,Of 2 ···Of n Then, the linear normalization method is used to convert the obtained gait distance matching value array and face distance matching value array into two values ​​according to the formula Normalization is performed to obtain a normalized gait distance matching value array and a normalized face distance matching value array, where: o is the Euclidean distance array, r norm is the linearly normalized Euclidean distance array. Thus, the normalized gait Euclidean distance array r is obtained. b = {r b1 ,r b2 ,···,r bn} and the normalized face Euclidean distance array r f = {r f1 ,r f2 ,···,r fn}, the Euclidean distance array can be mapped to the interval [0, 1] through the linear normalization method without changing the distribution form of the original matching data.

[0170] 2) The normalized gait distance matching value array and the normalized face distance matching value array can be used as one of the parameters for subsequent confidence calculation. However, how to allocate the weights of the sub-functions in each array to achieve the best fusion effect is the core issue of the fusion quality of the entire decision-making layer. Therefore, this embodiment introduces the rejection rate into the calculation of the decision-making layer algorithm. The rejection rate (false rejection rate, FRR) is an important parameter for measuring the classification performance of the system in system identification. Especially in the multimodal recognition process, the classification performance differences between each mode will reduce the recognition accuracy of the system. Therefore, incorporating the rejection rate into the calculation of the confidence can improve the ability of information fusion extraction and classification and enhance the robustness of the system. This embodiment calculates the confidence of each mode (gait and face) according to the normalized feature distance information (which is the normalized Euclidean distance array of gait and face) and the rejection rate:

[0171] Among them, r b,f are the normalized Euclidean distance arrays; n is the total number of values ​​in the normalized Euclidean distance array; FRR b,f They are the rejection rates of the modalities. When calculating gait confidence, the formula uses the normalized gait Euclidean distance array and the rejection rate of gait features. When calculating face confidence, the formula uses the normalized face Euclidean distance array and the rejection rate of face features. From the confidence formula, we can see that the confidence of each modality is determined by the normalized Euclidean distance array and the performance parameters of the classifier. Therefore, the weight w of each modality fusion in the decision layer can be calculated separately. b,f for: Where 0≤w b,f ≤1, m is the number of recognition modes, in this embodiment, there are two modes: gait and face, then m = 2, and The confidence is converted into the weight factor of gait and face through the exponential relationship, so as to strengthen the relationship between high confidence and high weight factor, and the resulting fusion system is more robust to noise. The modal weight value reflects the position of the sub-classification results of the two modalities in the decision-making, and therefore directly affects the quality and effect of the final decision. The higher the weight factor, the stronger the classification ability of the corresponding modality, and the greater the impact on the final decision. In this way, the complementary advantages of the two recognition modes can be well utilized to improve the recognition rate and anti-interference of the entire system.

[0172] 3) Finally, when the decision is combined, the weight factor is used as the weight of the sub-decision according to the formula The final identification result T is obtained, where In the formula, represents the configuration identification coefficient, T b,f are the recognition results of the modes respectively;i is the target category (the target category is divided into two types: collected gait and face). The decision fusion method based on feature information and rejection rate guidance proposed in this embodiment, since the normalized feature distance (normalized Euclidean distance array) comes from different modes, the information contained in its matching distance is different, so this feature distance is non-standardized and changes dynamically. The advantage of this method is that by extracting effective information on the distribution of normalized feature distances of different modes, and adding the rejection rates of different modal systems, the confidence of the mode is obtained. This confidence is mathematically mapped to obtain the corresponding weight factor, and the decision-level fusion is performed according to different weights. It is a dynamic and adaptive weighted fusion recognition algorithm.

[0173] The present invention has the following technical effects:

[0174] 1. The present invention has strong robustness under complex background conditions.

[0175] 2. Since most studies do not consider the problem of reduced recognition rate caused by interference factors such as changes in clothing and carried items, the present invention only recognizes ordinary pedestrians walking. For this purpose, a method integrating convolutional neural network and long short-term memory network is proposed, which can not only extract the dynamic and static features of the target but also reduce the impact of interference factors such as changes in clothing and carried items.

[0176] 3. Based on the SVM algorithm, in order to improve the robustness of the fusion algorithm of the decision layer, the present invention uses the discrimination results (distance information) of the feature layers of different modes to assist in guiding the fusion of the decision layer, introduces the rejection rate to make a weighted judgment of the confidence, and forms an adaptive weighted fusion algorithm to obtain the recognition result. This method makes good use of the complementary advantages of the two recognition modes at different angles and in different scenarios, and improves the recognition rate, anti-interference and scope of application of the entire system. In practical applications, the performance of the identity recognition system can be effectively improved with a small increase in system cost and user burden.

[0177] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0178] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An identity recognition method based on adaptive fusion of multiple biometric features. It is characterized in that include: Acquire a video sequence to be identified, and acquire a video sequence within a gait cycle from the video sequence to be identified to obtain a periodic video; The video sequence to be identified includes a person to be identified; Obtaining the PAF matrix corresponding to each frame of the person to be identified in the periodic video to obtain a walking feature vector map; the PAF matrix includes the PAFs of the limbs and the PAF of the trunk; Inputting the walking feature vector graph into a first convolutional neural network and a long short-term memory neural network respectively to obtain a spatial feature matrix and a temporal feature matrix; Performing feature fusion on the spatial feature matrix and the temporal feature matrix to obtain a gait feature matrix of the person to be identified; Acquire a face image of a person to be identified in the video sequence to be identified, and use a face recognition algorithm with adaptive weighted HOG features to process the face image of the person to be identified to obtain a face feature matrix; Adopting an adaptive weighted fusion algorithm based on SVM to process the facial feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized; The method of using an adaptive weighted fusion algorithm based on SVM to process the facial feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized specifically includes: Projecting the gait feature matrix and the face feature matrix onto the coordinate system formed by the base image to obtain a gait projection matrix and a face projection matrix; The gait Euclidean distances between the image in the video sequence to be identified and each standard image in the image database are calculated according to the gait projection matrix to obtain a gait Euclidean distance array; the face Euclidean distances between the image in the video sequence to be identified and each standard image in the image database are calculated according to the face projection matrix to obtain a face Euclidean distance array; Calculating gait confidence and face confidence according to the gait Euclidean distance array, the face Euclidean distance array, the rejection rate of the gait feature matrix and the rejection rate of the face feature matrix; Calculating a gait fusion weight according to the gait confidence, and calculating a face fusion weight according to the face confidence; The identity recognition result of the person to be recognized is obtained according to the gait fusion weight and the face fusion weight.

2. According to claim 1, an identity recognition method based on adaptive fusion of multiple biometric features, It is characterized in that The method for determining the gait cycle is: The video sequence to be identified is input into a trained second convolutional neural network to obtain the gait cycle of the person to be identified.

3. According to claim 1, an identity recognition method based on adaptive fusion of multiple biometric features, It is characterized in that Before the adaptive weighted fusion algorithm is used to process the facial feature matrix, the gait feature matrix and the standard image in the image database to obtain the identity recognition result of the person to be recognized, the method further includes: Performing dimensionality reduction processing on the facial feature matrix to obtain a reduced-dimensional facial feature matrix.

4. According to claim 1, an identity recognition method based on adaptive fusion of multiple biometric features, It is characterized in that The step of obtaining a facial image of a person to be identified in the video sequence to be identified and processing the facial image of the person to be identified using a face recognition algorithm with adaptive weighted HOG features to obtain a facial feature matrix specifically includes: Acquire a facial image of the person to be identified in the video sequence to be identified; Dividing the facial image of the person to be identified into cell units of the same size; Calculate the gradient histogram of each cell unit according to the direction and amplitude of the pixel gradient in each cell unit; The gradient histogram of all cell units is determined as the face feature matrix.

5. An identity recognition system based on adaptive fusion of multiple biometric features. It is characterized in that include: A video sequence acquisition module is used to acquire a video sequence to be identified, and acquire a video sequence within a gait cycle from the video sequence to be identified to obtain a periodic video; The video sequence to be identified includes a person to be identified; A PAF matrix acquisition module is used to obtain the PAF matrix corresponding to each frame of the person to be identified in the periodic video to obtain a walking feature vector map; the PAF matrix includes the PAF of the limbs and the PAF of the trunk; A spatiotemporal feature determination module, used for inputting the walking feature vector map into a first convolutional neural network and a long short-term memory neural network to obtain a spatial feature matrix and a temporal feature matrix; A gait feature matrix determination module, used for fusing the spatial feature matrix and the temporal feature matrix to obtain the gait feature matrix of the person to be identified; A facial feature matrix determination module is used to obtain a facial image of a person to be identified in the video sequence to be identified and process the facial image of the person to be identified using a face recognition algorithm with adaptive weighted HOG features to obtain a facial feature matrix; An identity recognition module, used to process the face feature matrix, the gait feature matrix and the standard image in the image database using an adaptive weighted fusion algorithm based on SVM to obtain an identity recognition result of the person to be recognized; The identity recognition module specifically includes: A projection matrix determination submodule, used for projecting the gait feature matrix and the face feature matrix onto the coordinate system formed by the base image to obtain a gait projection matrix and a face projection matrix; The Euclidean distance array determination submodule is used to calculate the gait Euclidean distance between the image in the video sequence to be identified and each standard image in the image database according to the gait projection matrix to obtain a gait Euclidean distance array; calculate the face Euclidean distance between the image in the video sequence to be identified and each standard image in the image database according to the face projection matrix to obtain a face Euclidean distance array; A confidence determination submodule, used to calculate gait confidence and face confidence according to the gait Euclidean distance array, the face Euclidean distance array, the rejection rate of the gait feature matrix and the rejection rate of the face feature matrix; A fusion weight determination submodule, used to calculate a gait fusion weight according to the gait confidence, and calculate a face fusion weight according to the face confidence; The identity recognition submodule is used to obtain the identity recognition result of the person to be recognized according to the gait fusion weight and the face fusion weight.

6. The identity recognition system based on adaptive fusion of multiple biometric features according to claim 5, It is characterized in that The video sequence acquisition module comprises: The gait cycle determination submodule is used to input the video sequence to be identified into the trained second convolutional neural network to obtain the gait cycle of the person to be identified.

7. The identity recognition system based on adaptive fusion of multiple biometric features according to claim 5, It is characterized in that Also includes: The dimension reduction module is used to perform dimension reduction processing on the facial feature matrix to obtain a reduced-dimensional facial feature matrix.

8. The identity recognition system based on adaptive fusion of multiple biometric features according to claim 5, It is characterized in that The facial feature matrix determination module specifically includes: An acquisition submodule, used for acquiring a face image of the person to be identified in the video sequence to be identified; A cell unit determination submodule, used for dividing the face image of the person to be identified into cell units of the same size; A gradient histogram determination submodule is used to calculate the gradient histogram of each cell unit according to the direction and amplitude of the pixel gradient in each cell unit; The face feature matrix determination submodule is used to determine the gradient histograms of all cell units as the face feature matrix.