Gait recognition method based on time-space correlation degree judgment

Abstract

The invention belongs to the technical field of biological recognition, and discloses a gait recognition method based on spatio-temporal correlation degree judgment, which comprises the following steps: extracting a gait sequence of a human body target from monitoring video data; the method comprises the following steps: constructing a city monitoring point location map, establishing vector representation of each node and an interaction relationship among elements of a vector based on the city monitoring point location map, obtaining spatial characteristics, obtaining time characteristics according to the moment of the current moment in one day, and generating a self-adaptive space-time representation map based on the spatial characteristics and the time characteristics; performing deep fusion on the gait feature vector and the self-adaptive spatio-temporal representation diagram to obtain a gait feature vector fused with spatio-temporal correlation; and inputting the gait feature vector fused with the spatial-temporal correlation into a gait recognition network for gait recognition to obtain gait features. According to the method, gait recognition can be carried out more accurately by capturing the spatial relationship between the urban monitoring point locations and the dynamic change of the pedestrian gaits.

Description

Technical Field

[0001] The present invention relates to the field of biometric identification technology, and in particular to a gait recognition method based on spatiotemporal correlation determination. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Gait recognition technology is an emerging biometric recognition technology that mainly identifies people by analyzing their body shape (such as height, muscles, joints and other physiological characteristics) and walking posture. Gait recognition technology integrates gait database building, gait recognition, gait retrieval, and large-scale tracking. It can support real-time intelligent interconnection of gait recognition under massive cameras, and realize instant retrieval and positioning of massive historical videos and real-time videos. It has broad application prospects in safe cities, intelligent security and other aspects.

[0004] In the field of intelligent video surveillance, gait recognition technology has more advantages than general image recognition. Since pedestrians have certain differences in center of gravity, tendon muscle strength, bone length, bone density, etc., gait recognition technology can achieve identity recognition by extracting and analyzing these unique gait features. In the actual use scenarios of gait recognition technology, applying time and space association for gait retrieval is a complex but efficient process. During the gait retrieval process, the system will use this time and space association information to narrow the search scope and improve accuracy. For example, the system may first filter out gait data that is similar to the target gait pattern in time based on the time association information, and then further filter out gait data that is also close to the target in spatial geographic location based on the spatial association information. Finally, the system will conduct a comprehensive analysis of the filtered gait data to identify the individual that best matches the target gait. This process may involve weighing and comparing multiple gait features to ensure the accuracy and reliability of the retrieval results.

[0005] By applying time and space associations to gait retrieval, gait recognition technology can quickly and accurately find individuals that match specific gait patterns in a large amount of surveillance video or image data, greatly improving the usability of gait recognition technology. This has important application value in the fields of public security criminal investigation, personnel management, etc., and can help relevant departments improve work efficiency and accuracy, and maintain social security and stability.

[0006] Gait recognition technology has a strong dependence on time and space. For example, the lighting conditions at different times will affect the gait characteristics of pedestrian gait sequences. The angles and directions of monitoring points at different locations will also affect the extracted gait characteristics, making the gait recognition process more difficult. However, existing methods have certain limitations in extracting spatiotemporal correlations in gait features. They only restrict time and space during the gait retrieval process, and do not extract deeper relationships between individual gait features, so that the deep spatiotemporal representation relationships contained therein are not fully utilized. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a gait recognition method based on spatiotemporal correlation judgment. With the help of spatiotemporal information, a graph neural network based on urban monitoring points is established for gait recognition. By capturing the spatial relationship between urban monitoring points and the dynamic changes of pedestrian gait, gait recognition can be performed more accurately.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] The present invention provides a gait recognition method based on time-space correlation determination, comprising the following steps:

[0010] Acquire surveillance video data, and extract the gait sequence of the human target from the surveillance video data;

[0011] Each monitoring point is regarded as a node, and a city monitoring point map is constructed. Based on the city monitoring point map, a vector representation of each node and the interaction relationship between each element of the vector are established to obtain the spatial characteristics of the gait sequence. According to the current time of the day, the temporal characteristics of the gait sequence are obtained. Based on the spatial characteristics and temporal characteristics, an adaptive spatiotemporal representation diagram is generated;

[0012] The gait feature vector is deeply fused with the adaptive spatiotemporal representation graph to obtain the gait feature vector integrating the spatiotemporal correlation;

[0013] The gait feature vector integrating the temporal and spatial correlation is input into the gait recognition network for gait recognition to obtain the gait features.

[0014] As an optional implementation, a spatiotemporal representation graph convolution layer is used to deeply fuse the gait feature vector with the adaptive spatiotemporal representation graph, wherein the spatiotemporal representation graph convolution layer is constructed based on a spatiotemporal representation graph convolution operator, and the spatiotemporal representation graph convolution layer is stacked with multiple layers.

[0015] As an optional implementation, skip connections are used in stacking multiple layers of spatiotemporal representation graph convolutional layers.

[0016] As an optional implementation, based on the city monitoring point map, the vector representation of each node is established as follows:

[0017] Relevant data of each monitoring point is obtained, multiple features that can represent the characteristics of each monitoring point are extracted, and multiple feature values ​​of each monitoring point are combined into a multidimensional vector.

[0018] As an optional implementation method, based on the city monitoring point map, the interactive relationship between the vector elements of each node is established, specifically:

[0019] Establish a hypothesis about the possible interaction relationship between the features of the monitoring points and quantify the defined interaction relationship into a numerical value.

[0020] As an optional implementation, according to the time of the day at which the current time is, the time feature is mapped into a one-hot encoding, and is mapped into a D-dimensional space using a fully connected network.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present disclosure proposes a gait recognition method based on spatiotemporal correlation judgment, and establishes a graph neural network based on urban monitoring points for gait recognition with the help of spatiotemporal information, which is of great significance for improving the efficiency of urban safety management and intelligent monitoring systems. The graph neural network can process complex graph structure data, and while extracting pedestrian gait features in real time, it integrates the input time features and spatial features. By capturing the spatial relationship between urban monitoring points and the dynamic changes of pedestrian gait, gait recognition can be performed more accurately. At the same time, the established graph neural network can cope with various complex scenarios, such as lighting changes, occlusion, pedestrian posture changes, etc., thereby improving the robustness of the gait recognition system. Through efficient algorithm and model design, the graph neural network can realize real-time gait recognition and provide timely and effective support for urban safety management.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a framework flow chart of a gait recognition method based on spatiotemporal correlation determination according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0028] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0029] Example 1

[0030] like Figure 1 As shown, this embodiment provides a gait recognition method based on spatiotemporal correlation determination, comprising the following steps:

[0031] S1 obtains surveillance video data and extracts the gait sequence of the human target from the surveillance video data;

[0032] S2 regards each monitoring point as a node, constructs a city monitoring point map, and based on the city monitoring point map, establishes the vector representation of each node and the interaction between the elements of the vector, obtains the spatial characteristics of the gait sequence, and obtains the time characteristics of the gait sequence according to the current time of the day. Based on the spatial characteristics and time characteristics, an adaptive spatiotemporal representation diagram is generated;

[0033] S3 deeply fuses the gait feature vector with the adaptive spatiotemporal representation graph to obtain a gait feature vector that integrates spatiotemporal correlation;

[0034] S4 inputs the gait feature vector integrating the temporal and spatial correlation into the gait recognition network for gait recognition to obtain gait features.

[0035] The purpose of the present disclosure is to capture the spatial relationship between monitoring points and the dynamic changes of pedestrian gait through a graph neural network algorithm. Specifically, the present disclosure proposes a spatiotemporal representation graph construction module, which fuses the input time features and spatial features while extracting pedestrian gait features in real time, performs unsupervised adjustment to embed the spatiotemporal representation, and generates an adaptive spatiotemporal representation graph. The real-time gait sequence extracted under monitoring is processed by a fully connected layer and input into the spatiotemporal representation graph convolution module together with the adaptive spatiotemporal representation graph. The spatiotemporal representation graph convolution module is constructed based on the spatiotemporal representation graph convolution operator and consists of multiple layers of stacked spatiotemporal representation graph convolution layers. The spatiotemporal representation graph convolution layer fuses the gait feature vector with the spatiotemporal representation. The module also uses jump connections to make full use of the depth of the neural network so that the neural network can learn deeper feature representations and improve the representation ability of the fused features. The gait feature vector that fuses the spatiotemporal features is input into a conventional gait recognition network for gait feature extraction, and the extracted gait features have a high generalization ability.

[0036] The main purpose of the spatiotemporal representation graph construction module of the present application is to simultaneously extract the dynamic temporal correlation, spatial correlation and cross-spatiotemporal correlation between pedestrian gait sequences. Among them, each monitoring point is regarded as a node, and a city monitoring point map is constructed. Based on the city monitoring point map, a spatial node embedding vector is first randomly initialized for each node, and then the spatial node embedding vector is mapped to the D-dimensional space using a fully connected layer. Since the gait sequence of pedestrians is affected by lighting at different times, the present disclosure introduces time features to construct a spatiotemporal representation embedding vector. The present disclosure considers 1 time feature, which is the time of the day at which the current time t is located. This time feature is mapped to a one-hot encoding, and a fully connected network is used to map it to the D-dimensional space. At each node at each time, the spatial node embedding vector and the time feature vector are added to obtain the spatiotemporal embedding vector of all nodes at each time, and the gait feature vector is added as input to the model training process.

[0037] A graph is established based on the city monitoring points, where each monitoring point is regarded as a node. All nodes in the graph are distributed in the same space. The vector representation of the node in this space and the interaction between the elements of the vector jointly affect the association between the nodes. Two matrices are established based on the city monitoring point map: E∈R N×D , represents the D-dimensional vector of the N nodes in the graph in this space, B∈R D×D Represents the interaction between the elements of a D-dimensional vector. T It can represent the connection relationship between nodes and is constantly adjusted during model training.

[0038] Based on the spatiotemporal embedding method, the adjacency matrix of the adaptive spatiotemporal composite graph at time t can be defined as:

[0039]

[0040]

[0041] The main purpose of the g(·) function is to discard weak connections (less than the specified threshold δ) in the adjacency matrix, and the softmax function is used for normalization. t Represents the connection relationship between N nodes at time t.

[0042] The purpose of establishing the matrix E (node ​​vector matrix): The matrix E is used to represent the position or characteristics of each node (i.e., monitoring point) in the graph in the D-dimensional space. The establishment process is as follows:

[0043] Data collection: First, it is necessary to collect relevant data about each monitoring point. This data includes geographical location (latitude and longitude), monitoring range, monitoring angle, historical monitoring records (such as pedestrian flow, vehicle flow, etc.), etc.

[0044] Feature extraction: Based on the collected data, D features that can represent the characteristics of each monitoring point are extracted. These features can be direct (such as longitude and latitude) or calculated or converted (such as activity index calculated based on historical data).

[0045] Vectorization: Combine the D eigenvalues ​​of each monitoring point into a D-dimensional vector, which constitutes the rows of the matrix E. Therefore, E is an N×D matrix, where N is the number of monitoring points and D is the number of features.

[0046] The purpose of establishing matrix B (interaction relationship matrix): Matrix B is used to represent the interaction relationship between each element (i.e., feature) in the D-dimensional vector. Its establishment process is:

[0047] Relationship definition: First, establish a hypothesis of possible interactive relationships between features. This relationship may be based on physical meaning (such as the interaction between geographic location and monitoring range) or data-driven (such as the correlation between features discovered through statistical or machine learning methods).

[0048] Quantify the relationship: Quantify the defined interaction relationship into a numerical value and fill it into the matrix B. B is a D×D matrix, whose element bij represents the interaction strength between the i-th feature and the j-th feature. This value is automatically adjusted by the optimization algorithm during the model training process.

[0049] Initialization: Before model training begins, B needs to be randomly initialized.

[0050] Matrix EBE TThe meaning of: By performing matrix multiplication of E and B, and then multiplying it by the transpose of E (i.e. EBE T ), we can get a matrix representing the connection strength between nodes. Each element of this matrix (EBE T ) ij Both reflect the strength of the connection between node i and node j, which is calculated based on their positions in the D-dimensional space (represented by E) and the interaction relationship between the features in these positions (represented by B).

[0051] During the model training process, the values ​​of E and B are continuously adjusted through the optimization algorithm to minimize the loss function (a function that reflects the difference between the actual connection between nodes and the predicted connection). As the training progresses, E and B will gradually approach the optimal value that can accurately reflect the relationship between urban monitoring points.

[0052] Based on the spatiotemporal representation graph construction module, this method also proposes a spatiotemporal representation graph convolution module. By defining an adaptive spatiotemporal representation graph convolution operator, an adaptive spatiotemporal representation graph convolution layer is constructed to deeply fuse the input pedestrian gait sequence with the spatiotemporal features.

[0053] The spatiotemporal representation graph convolution operator is defined as:

[0054] P = f(OXW+b)

[0055] Where: X and P are the input and output signals of the graph convolution operator, respectively, W is the node weight, b is the bias, f(·) represents the activation function, and O is the normalized adjacency matrix.

[0056] Based on the spatiotemporal representation graph convolution operator, a spatiotemporal representation graph convolution layer can be constructed. First, a fully connected network is used to map the input gait sequence data into a D-dimensional space, and then input into a multi-layer spatiotemporal representation graph convolution layer. The spatiotemporal representation graph convolution operator can better fuse spatiotemporal features with gait feature vectors by stacking multiple layers of spatiotemporal representation graph convolution layers based on the deeply joint spatiotemporal correlation at a certain moment. Using skip connections in the spatiotemporal representation graph convolution module not only alleviates the gradient vanishing problem, but also enables the neural network to learn a more deeply joint feature representation. By inputting this feature into the gait recognition network, gait features that fuse spatiotemporal correlation can be obtained. By searching and comparing in the gait data base, the identity landing process of the pedestrian target can be completed.

[0057] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0058] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A gait recognition method based on spatiotemporal correlation determination, characterized in that: The following steps are involved: Acquire surveillance video data, and extract the gait sequence of the human target from the surveillance video data; Each monitoring point is regarded as a node, and a city monitoring point map is constructed. Based on the city monitoring point map, a vector representation of each node and the interaction relationship between each element of the vector are established to obtain the spatial characteristics of the gait sequence. According to the current time of the day, the temporal characteristics of the gait sequence are obtained. Based on the spatial characteristics and temporal characteristics, an adaptive spatiotemporal representation diagram is generated; The gait feature vector is deeply fused with the adaptive spatiotemporal representation graph to obtain the gait feature vector integrating the spatiotemporal correlation; The gait feature vector integrating the temporal and spatial correlation is input into the gait recognition network for gait recognition to obtain the gait features.

2. A gait recognition method based on spatiotemporal correlation determination as claimed in claim 1, characterized in that: The gait feature vector is deeply fused with the adaptive spatiotemporal representation graph by using a spatiotemporal representation graph convolution layer, wherein the spatiotemporal representation graph convolution layer is constructed based on a spatiotemporal representation graph convolution operator, and the spatiotemporal representation graph convolution layer is stacked with multiple layers.

3. A gait recognition method based on spatiotemporal correlation determination as claimed in claim 2, characterized in that: Skip connections are used in stacking multiple convolutional layers of the spatiotemporal representation graph.

4. A gait recognition method based on spatiotemporal correlation determination as claimed in claim 1, characterized in that: Based on the urban monitoring point map, the vector representation of each node is established as follows: Relevant data of each monitoring point is obtained, multiple features that can represent the characteristics of each monitoring point are extracted, and multiple feature values ​​of each monitoring point are combined into a multidimensional vector.

5. A gait recognition method based on spatiotemporal correlation determination as claimed in claim 4, characterized in that: Based on the urban monitoring point map, the interactive relationship between the vector elements of each node is established, specifically: Establish a hypothesis about the possible interaction relationship between the features of the monitoring points and quantify the defined interaction relationship into a numerical value.

6. A gait recognition method based on spatiotemporal correlation determination as claimed in claim 1, characterized in that: According to the time of the day at which the current moment is, the time feature is mapped into a one-hot encoding and mapped into a D-dimensional space using a fully connected network.

Images