Training method of gait recognition model, gait recognition method, medium and controller

Abstract

The invention relates to the technical field of gait recognition, and discloses a training method of a gait recognition model, a gait recognition method, a medium and a controller, and the training method comprises the steps: obtaining a training sample set which comprises a gait silhouette sequence and a corresponding real identity tag; inputting the gait silhouette sequence in the training sample set into a gait recognition model to obtain gait features and predicted identity tags of the gait silhouette sequence; constructing a triple loss function according to the gait features, constructing a cross entropy loss function according to the predicted identity tag, and constructing a gait recognition model loss function according to the triple loss function and the cross entropy loss function; and training the gait recognition model based on the gait recognition model loss function to obtain a trained gait recognition model. The gait recognition model trained by the training method has the advantage of high gait recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of gait recognition, and in particular to a training method for a gait recognition model, a gait recognition method and medium, and a controller. Background Art

[0002] Gait recognition is a biometric technology that authenticates the subject's identity based on his or her body shape and walking style. Compared with face recognition, iris recognition, and fingerprint recognition, gait data has the advantages of being difficult to disguise, being able to be collected from a distance, and not requiring the subject's cooperation. Therefore, gait recognition has broad application prospects in video surveillance, security, and criminal investigation.

[0003] The goal of gait recognition is to learn specific identity representations from the temporal changes in human body shape. However, factors such as camera perspective, clothing, and objects carried in real scenes not only cause obvious changes in the human body contour, but also change the human body's motion characteristics to a certain extent, thus posing a huge challenge to the gait recognition task. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one object of the present invention is to provide a training method for a gait recognition model, wherein the trained gait recognition model has the advantage of high gait recognition accuracy.

[0005] The second objective of the present invention is to provide a gait recognition method.

[0006] A third object of the present invention is to provide a computer-readable storage medium.

[0007] A fourth object of the present invention is to provide a controller.

[0008] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a training method for a gait recognition model, wherein the gait recognition model comprises a gait silhouette feature extraction network, a fully connected layer and a classifier, wherein the input end of the gait silhouette feature extraction network is used to input a gait silhouette sequence, the output end of the gait silhouette feature extraction network is connected to the input end of the fully connected layer, the output end of the fully connected layer is connected to the input end of the classifier, the fully connected layer is used to output the gait features of the gait silhouette sequence, and the classifier is used to output the identity label of the gait silhouette sequence, and the training method comprises:

[0009] Acquire a training sample set, wherein the training sample set includes a gait silhouette sequence and its corresponding true identity label;

[0010] Inputting the gait silhouette sequence in the training sample set into a gait recognition model to obtain gait features and predicted identity labels of the gait silhouette sequence;

[0011] constructing a triplet loss function according to the gait feature, constructing a cross entropy loss function according to the predicted identity label and the true identity label, and constructing a gait recognition model loss function according to the triplet loss function and the cross entropy loss function;

[0012] The gait recognition model is trained based on the gait recognition model loss function to obtain a trained gait recognition model.

[0013] According to the training method of the gait recognition model of the embodiment of the present invention, the gait silhouette features extracted by the gait silhouette feature extraction network adopted by the gait recognition model are enhanced gait silhouette features, which can more effectively highlight the contour features of the human body and improve the gait recognition accuracy of the trained gait recognition model.

[0014] In addition, the training method of the gait recognition model proposed in the above embodiment of the present invention may also have the following additional technical features:

[0015] According to one embodiment of the present invention, the gait silhouette feature extraction network includes a first three-dimensional convolution layer, a first time aggregation layer, an inter-frame difference and context information feature extraction layer, a first pooling layer and a first generalized mean pooling layer connected in sequence, and the gait silhouette sequence in the training sample set is input into the gait recognition model, including:

[0016] Using the first three-dimensional convolution layer and the first time aggregation layer to extract and aggregate features of the gait silhouette sequence to obtain a first feature;

[0017] Extracting the first feature using the inter-frame difference and context information feature extraction layer to obtain a second feature;

[0018] Using the first pooling layer and the first generalized mean pooling layer to pool the second feature to obtain a gait silhouette feature;

[0019] Mapping the gait silhouette feature using the fully connected layer to obtain the gait feature;

[0020] The gait feature is classified using the classifier to obtain the predicted identity label.

[0021] According to an embodiment of the present invention, the inter-frame difference and context information feature extraction layer includes a first feature extraction module, a second feature extraction module and a third feature extraction module, the first feature extraction module, the second feature extraction module and the third feature extraction module all include a global feature extraction branch, an inter-frame difference feature extraction branch, a context information extraction branch and a feature fusion layer, the feature fusion layer includes a feature addition fusion sublayer and a feature splicing fusion sublayer, wherein the first feature extraction module and the second feature extraction module use the feature splicing fusion sublayer to perform element-level feature addition fusion, and the third feature extraction module uses the feature splicing fusion sublayer to perform channel dimension feature splicing fusion, and the inter-frame difference and context information feature extraction layer is used to extract the first feature to obtain the second feature, including:

[0022] Utilize the global feature extraction branch in the first feature extraction module to perform overall feature extraction on the first feature, the inter-frame difference feature extraction branch to perform feature extraction on the feature difference between adjacent frames in the first feature, and the context information extraction branch to perform information extraction on the feature mean between adjacent frames in the first feature, to obtain a first overall feature, a first inter-frame difference feature, and a first context feature;

[0023] Using the feature fusion layer in the first feature extraction module, element-level feature addition is performed on the first overall feature, the first inter-frame difference feature, and the first context feature to obtain a first silhouette fusion feature;

[0024] The global feature extraction branch in the second feature extraction module is used to perform overall feature extraction on the first silhouette fusion feature, the inter-frame difference feature extraction branch is used to perform feature extraction on the feature difference between adjacent frames in the first silhouette fusion feature, and the context information extraction branch is used to perform information extraction on the feature mean between adjacent frames in the first silhouette fusion feature, so as to obtain a second overall feature, a second inter-frame difference feature, and a second context feature;

[0025] Using the feature fusion layer in the second feature extraction module, perform element-level feature addition on the second overall feature, the second inter-frame difference feature, and the second context feature to obtain a second silhouette fusion feature;

[0026] The global feature extraction branch in the third feature extraction module is used to perform overall feature extraction on the second silhouette fusion feature, the inter-frame difference feature extraction branch is used to perform feature extraction on the feature difference between adjacent frames in the second silhouette fusion feature, and the context information extraction branch is used to perform information extraction on the feature mean between adjacent frames in the second silhouette fusion feature, so as to obtain a third overall feature, a third inter-frame difference feature, and a third context feature;

[0027] The feature fusion layer in the second feature extraction module is used to perform channel dimension feature splicing on the third overall feature, the third inter-frame difference feature, and the third context feature to obtain the second feature.

[0028] According to one embodiment of the present invention, the gait recognition model further includes a deformation field feature extraction network and a fusion layer, the input end of the deformation field feature extraction network is used to input the deformation field data of the gait silhouette sequence, the output end of the deformation field feature extraction network is connected to the first input end of the fusion layer, the second input end of the fusion layer is connected to the output end of the gait silhouette feature extraction network, and the output end of the fusion layer is connected to the fully connected layer; the training method further includes:

[0029] Using the deformation field feature extraction network to extract features from the deformation field data of the gait silhouette sequence to obtain deformation field features, and using the fusion layer to fuse the gait silhouette features and the deformation field features;

[0030] Mapping the gait silhouette feature by using the fully connected layer includes:

[0031] The fully connected layer is used to map the gait silhouette features fused with the deformation field features.

[0032] According to one embodiment of the present invention, the deformation field data is obtained in the following manner:

[0033] For each of the gait silhouette sequences, performing contour extraction on each gait silhouette frame in the gait silhouette sequence to obtain a human body contour;

[0034] Discretizing the human body contour to obtain discrete points of the human body contour;

[0035] The discrete points of the human body contour between adjacent frames are registered to obtain the deformation field data corresponding to each gait silhouette sequence.

[0036] According to an embodiment of the present invention, the deformation field feature extraction network branch includes a second three-dimensional convolution layer, a second time aggregation layer, a multi-scale feature extraction layer, a second pooling layer, and a second generalized mean pooling layer connected in sequence, and the deformation field feature extraction network branch is used to extract features from the deformation field data of the gait silhouette sequence, including:

[0037] Using the second three-dimensional convolution layer and the second time aggregation layer to perform feature extraction and aggregation on the deformation field data to obtain a third feature;

[0038] Using the multi-scale feature extraction layer to extract the third feature, to obtain a fourth feature;

[0039] The fourth feature is pooled by using the second pooling layer and the second generalized mean pooling layer to obtain the deformation field feature.

[0040] According to one embodiment of the present invention, the expression of the gait recognition model loss function is:

[0041] L all =L ce +L tri

[0042] Among them, L all is the loss function of the gait recognition model, L ce is the cross entropy loss function, L tri is the triple loss function;

[0043] The cross entropy loss function L ce The expression is:

[0044]

[0045] Where N is the number of identity features, y i is the identity label corresponding to the i-th sample, p i is the predicted probability of the i-th sample;

[0046] The triplet loss function L tri The expression is:

[0047]

[0048] Among them, H is the number of triplets, and the triplets are is the feature of the subject in the hth triplet, is the hth triple with Characteristics belonging to the same subject, is the hth triple with Characteristics belonging to different subjects, Features With features The Euclidean distance between Features With features The Euclidean distance between them, and m represents the boundary threshold.

[0049] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a gait recognition method, which includes: inputting a gait silhouette sequence to be identified into a trained gait recognition model to obtain the category of the gait silhouette sequence to be identified, wherein the trained gait recognition model is trained using the training method of the gait recognition model proposed in the first aspect of the present invention.

[0050] To achieve the above-mentioned objectives, the third aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the training method of the gait recognition model proposed in the first aspect of the present invention, or the gait recognition method proposed in the second aspect of the present invention.

[0051] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a controller, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the training method of the gait recognition model proposed in the first aspect of the present invention, or the gait recognition method proposed in the second aspect of the present invention.

[0052] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a method for training a gait recognition model according to an embodiment of the present invention;

[0054] Figure 2 is a network structure diagram of a gait recognition model according to an embodiment of the present invention;

[0055] Figure 3 is a processing flow chart of inputting a gait silhouette sequence into a gait recognition model according to an embodiment of the present invention;

[0056] Figure 4 is a flow chart for obtaining the second feature according to an embodiment of the present invention;

[0057] Figure 5 is a structural diagram of an inter-frame difference and context information feature extraction module according to an embodiment of the present invention;

[0058] Figure 6 is a network structure diagram of a gait recognition model according to another embodiment of the present invention;

[0059] FIG. 7( a ) is a process diagram of obtaining a human body contour deformation field from a gait silhouette image according to an embodiment of the present invention;

[0060] FIG. 7( b ) is a visualization effect diagram of human body contour registration according to an embodiment of the present invention;

[0061] Figure 8 is a flow chart of extracting features from deformation field data of a gait silhouette sequence according to an embodiment of the present invention;

[0062] Fig. 9 is a structural diagram of a multi-scale feature extraction module according to an embodiment of the present invention;

[0063] Fig.10 is a training method of a gait recognition model according to another embodiment of the present invention;

[0064] Fig.11 is a network structure diagram of a gait recognition model according to another embodiment of the present invention;

[0065] Fig.12 is a processing flow chart of inputting a gait silhouette sequence into a gait recognition model according to an embodiment of the present invention;

[0066] Fig.13 is a flow chart for obtaining the fourth feature of an embodiment of the present invention;

[0067] Fig.14 is a flow chart of feature extraction of a gait silhouette sequence according to an embodiment of the present invention;

[0068] Fig.15 This is a graph of deformation field branch network experimental results based on different numbers of human body contour points according to an embodiment of the present invention;

[0069] Fig.16 is a flow chart of a gait recognition method according to an embodiment of the present invention;

[0070] Fig.17 is a flow chart of a gait recognition method according to another embodiment of the present invention;

[0071] Fig.18 4 is a structural block diagram of a controller according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0073] It should be noted that, according to the data type of the research object, gait recognition methods can be roughly divided into two categories: model-based methods and appearance-based methods. Among them, the model-based method extracts the coordinates of key points of the human body through the human posture estimation method, and then performs explicit modeling of the human body structure and walking style to obtain gait characteristics. Although the model-based method can cope with changes in clothing and carrying objects more robustly, this type of method is greatly affected by the results of human posture estimation. The method using gait contour as appearance information shows better performance and occupies less computing resources. The appearance-based method has gradually become the mainstream, such as gait templates and gait silhouette sequences, but the appearance-based method is often more sensitive to changes in external factors such as perspective, clothing and carrying objects.

[0074] In order to solve the above problems, researchers have applied deep learning methods to the field of gait recognition. The current mainstream deep gait recognition methods can be divided into three categories. The first method treats the human body contour sequence as an unordered set of image frames, extracts frame-level features and set features respectively, and uses the horizontal pyramid mapping (HPM) module to aggregate global and local features to enhance the representation ability of gait features, such as GaitSet and GLN (Gaitlate ral network); the second method focuses on the division of gait features in spatial scale, and uses 3D CNN (Convolutionnal Nerual Network) to obtain overall spatiotemporal features and local spatiotemporal representations of each body part, such as GaitGL and GaitStrip; the third method focuses on temporal relationship modeling and motion feature capture of gait sequences, and attempts to design multi-scale temporal feature extraction methods and motion feature capture methods, such as CSTL (Context-Sensitive Temporal Feature Learning) and LagrangeGait.

[0075] Through observation and analysis, gait silhouettes can effectively reflect the shape characteristics of the human body. At the same time, as a grayscale image, the effective information in the silhouette is concentrated on the edge contour. However, the above method does not design a specific feature extraction method for the human body contour area.

[0076] To solve the above problems, the present invention provides a training method for a gait recognition model, a gait recognition method and medium, and a controller. Figure 1-18 And specific implementation methods are described in detail the training method of the gait recognition model, the gait recognition method and medium, and the controller of the embodiment of the present invention.

[0077] In one embodiment of the present invention, the gait recognition model may include a gait silhouette feature extraction network, a fully connected layer and a classifier, wherein the input end of the gait silhouette feature extraction network is used to input a gait silhouette sequence, the output end of the gait silhouette feature extraction network is connected to the input end of the fully connected layer, the output end of the fully connected layer is connected to the input end of the classifier, the fully connected layer is used to output the gait features of the gait silhouette sequence, and the classifier is used to output the identity label of the gait silhouette sequence. Figure 1 FIG. 1 is a flow chart of a method for training a gait recognition model according to an embodiment of the present invention. Figure 1 As shown, the training method of the gait recognition model may include:

[0078] S101, obtaining a training sample set, wherein the training sample set includes a gait silhouette sequence and its corresponding true identity label;

[0079] S102, inputting the gait silhouette sequence in the training sample set into the gait recognition model to obtain the gait features and predicted identity labels of the gait silhouette sequence;

[0080] S103, constructing a triplet loss function according to the gait feature, constructing a cross entropy loss function according to the predicted identity label and the true identity label, and constructing a gait recognition model loss function according to the triplet loss function and the cross entropy loss function;

[0081] S104, training the gait recognition model based on the gait recognition model loss function to obtain a trained gait recognition model.

[0082] Specifically, establish Figure 2 The gait recognition model shown. The gait recognition model includes a gait silhouette feature extraction network, a fully connected layer and a classifier connected in sequence. The embodiment of the present invention utilizes a gait silhouette feature extraction network to extract features of a gait silhouette sequence to obtain gait silhouette features. The gait silhouette features extracted by the gait silhouette feature extraction network in the embodiment of the present invention are enhanced gait silhouettes that can more effectively highlight the contour features of the human body. The fully connected layer (Separate Fully Connected Layer, Separate FC) maps the extracted gait silhouette features, and the classifier (Classifier) ​​classifies the mapped gait features to obtain identity labels for the gait silhouette sequence.

[0083] In one embodiment of the present invention, when acquiring a training sample set, gait silhouette sequences of different subjects in different states and different perspectives may be collected, and the training sample set may be obtained by marking the true identity label of each gait silhouette sequence.

[0084] In another embodiment of the present invention, when obtaining the training sample set, the CASIA-B data set can be used as the training sample set. It should be noted that the CASIA-B data set is a widely used gait recognition data set. It contains 10 groups of gait data collected by 124 subjects under 3 walking conditions, including 6 groups of normal walking (NM) data, 2 groups of backpack walking (BG) data and 2 groups of dressing walking (CL) data. In addition, each group of gait data corresponds to 11 different viewing angles (0°, 18°, ..., 90°, 108°, ..., 180°), that is, each subject has a total of 110 groups of gait sequences. Among them, the gait data of 74 subjects are used for training, and the gait data of the remaining 50 subjects are used for testing. In the test phase, the first 4 groups of data of each subject under the NM condition are the verification set (Gallery), and the remaining 6 groups of data are the probe set (Prob e), including 2 groups of NM data, 2 groups of BG data and 2 groups of CL data.

[0085] When training the gait recognition model, the gait silhouette sequence in the training sample set is input into the gait recognition model, the gait silhouette feature extraction network extracts the features of the gait silhouette sequence, the fully connected layer maps the extracted gait silhouette features, and the classifier classifies the mapped gait features to obtain the gait features of the gait silhouette sequence output by the fully connected layer and the predicted identity label output by the classifier.

[0086] The triplet loss (Triplet Loss) is calculated for the gait features output by the fully connected layer, and the cross-entropy loss (Cross-EntropyLos s) is calculated based on the predicted identity label output by the classifier and the true identity label corresponding to the gait silhouette sequence, and the gait recognition model loss function is constructed based on the triplet loss function and the cross-entropy loss function. The gait recognition model is trained based on the gait recognition model loss function to obtain a trained gait recognition model.

[0087] In one embodiment of the present invention, the gait silhouette feature extraction network may include a first three-dimensional convolution layer, a first time aggregation layer, an inter-frame difference and context information feature extraction layer, a first pooling layer and a first generalized mean pooling layer connected in sequence, such as Figure 3 As shown, inputting the gait silhouette sequence in the training sample set into the gait recognition model may include:

[0088] S201, extracting and aggregating features of a gait silhouette sequence using a first three-dimensional convolutional layer and a first temporal aggregation layer to obtain a first feature;

[0089] S202, extracting the first feature by using the inter-frame difference and context information feature extraction layer to obtain a second feature;

[0090] S203, pooling the second feature using the first pooling layer and the first generalized mean pooling layer to obtain a gait silhouette feature;

[0091] S204, mapping the gait silhouette features using a fully connected layer to obtain gait features;

[0092] S205, using a classifier to classify the gait features to obtain a predicted identity label.

[0093] Since the human body changes dynamically during walking and each subject has different body structures and behavioral habits, their gait silhouettes show different degrees of dynamic changes. The embodiment of the present invention adopts inter-frame difference and context information feature extraction layers to capture dynamic difference characteristics and obtain context information between consecutive gait frames.

[0094] Specifically, Figure 2 As shown in Figure 1, the gait silhouette sequence of the input gait silhouette feature extraction network can be defined as S. The dimension of the gait silhouette sequence S is 1×T×H×W, where T represents the number of input frames, H×W represents the height and width of the image, and the number of channels 1 represents that the gait silhouette is a grayscale image. The gait silhouette sequence S is sequentially processed through the three-dimensional convolution layer Conv3d and the temporal aggregation (TA) operation to obtain the first feature f s , the first feature f s It is sent to the inter-frame difference and context information feature extraction layer, and the second feature extracted is pooled using the first pooling layer TP (Temporal Pooling, TP) and the first generalized mean pooling layer (Generalized-Mean pooling, GeM), and the gait silhouette feature obtained by pooling is mapped using the fully connected layer to obtain the gait feature. The gait feature is classified using a classifier to obtain the predicted identity label.

[0095] In one embodiment of the present invention, the inter-frame difference and context information feature extraction layer includes a first feature extraction module, a second feature extraction module and a third feature extraction module, the first feature extraction module, the second feature extraction module and the third feature extraction module all include a global feature extraction branch, an inter-frame difference feature extraction branch, a context information extraction branch and a feature fusion layer, the feature fusion layer includes a feature addition fusion sublayer and a feature splicing fusion sublayer, wherein the first feature extraction module and the second feature extraction module use the feature splicing fusion sublayer to perform element-level feature addition fusion, and the third feature extraction module uses the feature splicing fusion sublayer to perform channel dimension feature splicing fusion, such as Figure 4 As shown, extracting the first feature using the inter-frame difference and context information feature extraction layer to obtain the second feature may include:

[0096] S301, using the global feature extraction branch in the first feature extraction module to perform overall feature extraction on the first feature, the inter-frame difference feature extraction branch to perform feature extraction on the feature difference between adjacent frames in the first feature, and the context information extraction branch to perform information extraction on the feature mean between adjacent frames in the first feature, to obtain a first overall feature, a first inter-frame difference feature, and a first context feature;

[0097] S302, using a feature fusion layer in a first feature extraction module to perform element-level feature addition on the first overall feature, the first inter-frame difference feature, and the first context feature to obtain a first silhouette fusion feature;

[0098] S303, using the global feature extraction branch in the second feature extraction module to perform overall feature extraction on the first silhouette fusion feature, the inter-frame difference feature extraction branch to perform feature extraction on the feature difference between adjacent frames in the first silhouette fusion feature, and the context information extraction branch to perform information extraction on the feature mean between adjacent frames in the first silhouette fusion feature, to obtain a second overall feature, a second inter-frame difference feature, and a second context feature;

[0099] S304, using a feature fusion layer in a second feature extraction module to perform element-level feature addition on the second overall feature, the second inter-frame difference feature, and the second context feature to obtain a second silhouette fusion feature;

[0100] S305, using the global feature extraction branch in the third feature extraction module to perform overall feature extraction on the second silhouette fusion feature, the inter-frame difference feature extraction branch to perform feature extraction on the feature difference between adjacent frames in the second silhouette fusion feature, and the context information extraction branch to perform information extraction on the feature mean between adjacent frames in the second silhouette fusion feature, to obtain a third overall feature, a third inter-frame difference feature, and a third context feature;

[0101] S306, using the feature fusion layer in the second feature extraction module to perform channel dimension feature splicing on the third overall feature, the third inter-frame difference feature, and the third context feature to obtain a second feature.

[0102] Specifically, the inter-frame difference and context information feature extraction layer may include a first feature extraction module, a second feature extraction module and a third feature extraction module. Among them, the first feature extraction module, the second feature extraction module and the third feature extraction module adopt an inter-frame difference and context information feature extraction module (Adjacent Frame Difference and Context Feature Extraction Module, ACFEM).

[0103] In one embodiment of the present invention, the inter-frame difference and context information feature extraction module ACFEM consists of three parts: a global feature extraction branch, an inter-frame difference feature extraction branch, and a context information extraction branch. The structure of the inter-frame difference and context information feature extraction module ACFEM is as follows: Figure 5 As shown, the input features can be defined as The dimension is C In ×T In ×H×W, where C In Represents the number of channels, T In Represents dimension, H×W represents the height and width of the image.

[0104] In one embodiment of the present invention, the global feature extraction branch is composed of a three-dimensional convolution Conv3d with a convolution kernel size of k = (3, 3, 3) and an activation function LeakyReLU, which is used to extract the overall features.

[0105] In the inter-frame difference feature extraction branch, in order to capture the changing characteristics of the dynamic area of ​​the human body in consecutive frames, the input features are calculated in the time dimension. The feature difference between adjacent frames in the The dimension is C In ×T In -1×H×W, use the matrix with value 0 to fill in the difference feature after filling The dimension is C In ×T In ×H×W. Difference features after padding The spatiotemporal features of the dynamic area are obtained by the three-dimensional convolution Conv 3d operation with a convolution kernel size k = (3, 3, 3) and the spatial features of the dynamic area are obtained by the convolution layer Conv3d with a convolution kernel size k = (1, 3, 3) as supplementary information. The output features of the three-dimensional convolution Conv3 d with a convolution kernel size k = (3, 3, 3) and the convolution layer Conv3d with a convolution kernel size k = (1, 3, 3) are added and the enhanced features are obtained by the activation function LeakyReLU. (Inter-frame difference feature).

[0106] In the context information extraction branch, in order to obtain the context information of the feature sequence, the input feature The features between adjacent frames in the calculation are averaged and the mean features are obtained. The dimension is C In ×T In -1×H×W, use the matrix with value 0 to fill in and get the mean feature after filling Its dimension is CIn ×T In ×H×W. The mean feature after padding The three-dimensional convolution Conv3d with a convolution kernel size of k = (3, 3, 3) is used to extract spatiotemporal features and the three-dimensional convolution Conv3d with a convolution kernel size of k = (3, 1, 1) is used to extract temporal features. The output features of the three-dimensional convolution Conv3d with a convolution kernel size of k = (3, 3, 3) and the three-dimensional convolution Conv3d with a convolution kernel size of k = (3, 1, 1) are added after the activation function LeakyReLU to obtain the contextual features.

[0107] In one embodiment of the present invention, after the first feature is processed by the inter-frame difference and context information feature extraction layer, the inter-frame difference and context information feature extraction layer outputs a second feature, wherein the second feature can be defined as After passing through the first pooling layer TP and the first generalized mean pooling layer GeM in sequence, the output gait silhouette feature is obtained

[0108] It should be noted that ACFEM adopts two feature fusion methods: element-level feature addition and channel-dimensional feature concatenation. When the output feature of ACFEM is the element-level feature addition operation, the module is defined as ACFEMA. When the output feature of ACFEM is the channel-dimensional feature concatenation, the module is defined as ACFEMB.

[0109] In one embodiment of the present invention, the first feature extraction module and the second feature extraction module adopt element-level feature addition (ACFEMA), and the third feature extraction module adopts channel-dimensional feature splicing (ACFEMB). The first feature extraction module uses the global feature extraction branch, the inter-frame difference feature extraction branch and the context information extraction branch to extract the first feature, and performs element-level feature addition and fusion on the features output by the global feature extraction branch, the inter-frame difference feature extraction branch and the context information extraction branch to obtain the first silhouette fusion feature. The second feature extraction module uses the global feature extraction branch, the inter-frame difference feature extraction branch and the context information extraction branch to extract the first silhouette fusion feature, and performs element-level feature addition and fusion on the features output by the global feature extraction branch, the inter-frame difference feature extraction branch and the context information extraction branch to obtain the second silhouette fusion feature. The third feature extraction module uses the global feature extraction branch, the inter-frame difference feature extraction branch and the context information extraction branch to extract the second silhouette fusion feature, and performs channel-dimensional feature splicing on the features output by the global feature extraction branch, the inter-frame difference feature extraction branch and the context information extraction branch.

[0110] The inter-frame difference and context information feature extraction module of the embodiment of the present invention adopts an inter-frame difference feature extraction branch to capture dynamic difference characteristics, and obtains context information between consecutive gait frames by constructing an inter-frame context information extraction branch, so as to solve the problem that the gait silhouette presents different degrees of dynamic change differences due to the differences in the body structure and behavioral habits of the subjects.

[0111] It should be noted that in view of the problem that related methods do not design specific feature extraction for the human body contour area, the GaitEdge network uses the semantic segmentation method to extract gait silhouettes from RGB images, and multiplies the edge area in the silhouette with the semantic segmentation probability of the corresponding position to obtain an edge area between 0 and 1, and then combines the edge area with the internal area of ​​the human body to obtain an edge-enhanced gait silhouette. Although the enhanced gait silhouette can more effectively highlight the human body contour features, it still lacks important explicit motion information, which results in the lack of representation of human body movement details when directly extracting spatiotemporal characteristics from the silhouette sequence using 3D CNN.

[0112] Although the gait silhouette enhanced by the gait silhouette feature extraction network can more effectively highlight the contour features of the human body, it still lacks important explicit motion information, which results in the lack of representation of human motion details when directly extracting the spatiotemporal characteristics from the silhouette sequence using 3D CNN. The embodiment of the present invention introduces a deformation field feature extraction network into the gait recognition model.

[0113] In one embodiment of the present invention, Figure 6 As shown, the gait recognition model may further include a deformation field feature extraction network and a fusion layer, the input end of the deformation field feature extraction network is used to input the deformation field data of the gait silhouette sequence, the output end of the deformation field feature extraction network is connected to the first input end of the fusion layer, the second input end of the fusion layer is connected to the output end of the gait silhouette feature extraction network, the output end of the fusion layer is connected to the fully connected layer, and the training method of the gait recognition model may also include:

[0114] The deformation field feature extraction network is used to extract the deformation field data of the gait silhouette sequence to obtain the deformation field features, and the gait silhouette features and the deformation field features are fused using the fusion layer;

[0115] The gait silhouette features are mapped using a fully connected layer, including:

[0116] The fully connected layer is used to map the gait silhouette features fused with the deformation field features.

[0117] Specifically, the deformation field data of the gait silhouette sequence is input into the deformation field feature extraction network, and the deformation field feature extraction network extracts features from the deformation field data of the gait silhouette sequence to obtain deformation field features of the gait silhouette sequence. The gait silhouette features and the deformation field features are fused using the fusion layer, and the gait silhouette features fused with the deformation field features are mapped using the fully connected layer to obtain gait features. The gait features are classified using a classifier to obtain a predicted identity label.

[0118] In one embodiment of the present invention, the network structure of the gait recognition model is as follows: Figure 6 As shown, the network structure of the gait recognition model may include a branch of a deformation field feature extraction network, a branch of a gait silhouette feature extraction network, a fully connected layer (Separate Fully Connected Layer, Separate FC) and a classifier.

[0119] Specifically, the deformation field feature extraction network uses the deformation field data obtained through point set registration to characterize the shape information and motion characteristics of the human body edge contour. The deformation field data and gait silhouette sequence are respectively passed through the feature extraction network to obtain the deformation field features and gait silhouette features, and then the final gait features are obtained through linear feature fusion. The gait features are sequentially passed through the fully connected layer and the classifier. The triplet loss (Triplet Loss) is calculated for the gait features output by the fully connected layer, and the cross-entropy loss (Cross-Entropy Loss) is calculated for the classification results output by the classifier.

[0120] In one embodiment of the present invention, the deformation field data is obtained by:

[0121] For each gait silhouette sequence, contour extraction is performed on each gait silhouette frame in the gait silhouette sequence to obtain a human body contour;

[0122] Discretize the human body contour to obtain discrete points of the human body contour;

[0123] The discrete points of the human body contour between adjacent frames are registered to obtain the deformation field data corresponding to each step silhouette sequence.

[0124] The embodiment of the present invention uses the deformation field in point set registration to characterize the displacement amount and displacement direction of the human body contour during walking. The human body contour deformation field can not only effectively retain the human body morphological information, but also make up for the problem of lack of motion information of the human body contour in the GaitEdge method, laying a good foundation for the subsequent extraction of gait motion characteristics.

[0125] Since the change of human body contour is a typical non-rigid transformation, the embodiment of the present invention uses the Thin-Plate Spline RPM (TPS-RPM) algorithm to perform non-rigid point set registration on the human body contour points of adjacent frames, so as to characterize the correspondence between the human body contour points of adjacent frames and mine the motion characteristics implied by the subject during walking. It should be noted that in the classic non-rigid point set matching algorithm, the TPS-RPM algorithm uses annealing algorithm, thin plate spline and soft allocation for spatial transformation and outlier removal, so as to obtain the correspondence and transformation relationship between point sets, and has good robustness.

[0126] The process of obtaining the human body contour deformation field from the gait silhouette image is shown in Figure 7(a). The embodiment of the present invention extracts the human body contour from the gait silhouette image, discretizes the continuous human body contour using the equal-interval sampling method, and uses TPS-RPM to align the human body contour points of adjacent frames. The visualization effect of the human body contour alignment is shown in Figure 7(b), where the gray circles represent the human body contour points of the previous frame, the white crosses represent the human body contour points of the next frame, and the gray lines represent the alignment results of two adjacent frames.

[0127] The TPS-RPM algorithm mainly includes two key steps that are iterative: calculating the correspondence relationship M between point sets and solving the non-rigid spatial transformation function f. Assume that the two human contour point sets to be registered are V = {v a , a=1,2,...,K} and U={u i , i = 1, 2, ..., N}, define X as the set of intermediate transformation points, f as the transformation function from V to X, and each point in V is mapped to a new position x through f. a =f(v a ). The goal of point set registration can be expressed as minimizing the quadratic energy function E(Z, f). The expression for minimizing the quadratic energy function E(Z, f) is:

[0128]

[0129] Where M = {m ai} is the correspondence matrix between point sets V and U, and the size of M is J×K. Each element m in M ai The value range of m is [0, 1]. ai The closer it is to 1, the higher the correspondence between the two points. λ||Lf|| 2 is the smoothness constraint on the objective function, Prevent too many points from being judged as wild points.

[0130] On this basis, the correspondence relationship (M) and transformation function (f) are iteratively updated. First, the correspondence matrix M is updated according to the annealing parameter T and the spatial transformation f until T reaches or is less than the termination temperature, and then M is normalized row by row and column by column. Then, the energy target E(f) is updated for the transformation function f according to the updated correspondence relationship M. The expression of the energy target E(f) is:

[0131]

[0132] in, Variable y a is the corresponding v calculated according to the correspondence matrix M in the point set U a point.

[0133] Finally, TPS (thin plate spline interpolation) is combined with the energy target E(f) to obtain the thin plate spline energy function E TPS (f), where the thin plate spline energy function E TPS The expression of (f) is:

[0134]

[0135] The human body contour points in adjacent frames are aligned to obtain the correspondence matrix M. The two-dimensional correspondence matrix M records the point v a Move to point u i The amount of displacement in the horizontal and vertical directions during the process.

[0136] The dimension of the gait silhouette sequence S of the embodiment of the present invention is 1×T×H×W, where T represents the number of input frames, H×W represents the height and width of the image, and the number of channels 1 represents that the gait silhouette is a grayscale image. After contour extraction, contour discretization and point set registration, the deformation field data D is output, and the dimension of D is 2×(T-1)×H×W, and the number of channels 2 represents the displacement in the horizontal and vertical directions.

[0137] The embodiment of the present invention introduces a non-rigid point set registration algorithm in the gait recognition task to solve the problem of insufficient motion information in gait silhouette data, and uses point set registration to generate deformation field data containing human body morphology and motion change information from human body contour points in adjacent frames.

[0138] In one embodiment of the present invention, the deformation field feature extraction network may include a second three-dimensional convolution layer, a second time aggregation layer, a multi-scale feature extraction layer, a second pooling layer, and a second generalized mean pooling layer connected in sequence, such as Figure 8 As shown, using the deformation field feature extraction network branch to extract features from the deformation field data of the gait silhouette sequence may include:

[0139] S401, extracting and aggregating features of the deformation field data using a second three-dimensional convolution layer and a second time aggregation layer to obtain a third feature;

[0140] S402, extracting the third feature using a multi-scale feature extraction layer to obtain a fourth feature;

[0141] S403: Pooling the fourth feature using a second pooling layer and a second generalized mean pooling layer to obtain a deformation field feature.

[0142] Specifically, the deformation field data D is subjected to three-dimensional convolution Conv3d and temporal aggregation (TA) to obtain the third feature f D , the process can be expressed as: D =F TA (F Conv3d (D)), where f D The dimension is C TA ×T TA ×H×W,C TA Indicates the number of channels, T TA represents the dimension after time compression, F Conv3d (·) represents a 3D convolution with kernel size k = (3, 3, 3), F TA (·) represents the temporal aggregation operation, which is a three-dimensional convolution Conv3d with a kernel size of k = (3, 3, 3) and a step size of k = (3, 1, 1).

[0143] Since the valid values ​​in the deformation field data D are only distributed at the positions of the human body contour points, the deformation field data has data sparsity. In view of this characteristic, the embodiment of the present invention adopts a multi-scale feature extraction layer to extract features, and pools the extracted fourth feature using the second pooling layer and the second generalized mean pooling layer to obtain the deformation field feature.

[0144] In one embodiment of the present invention, the multi-scale feature extraction layer may include a first multi-scale feature extraction module, a second multi-scale feature extraction module and a third multi-scale feature extraction module, the first multi-scale feature extraction module, the second multi-scale feature extraction module and the third multi-scale feature extraction module all include a first convolution branch, a second convolution branch, a third convolution branch and a feature fusion layer, the feature fusion layer includes a feature addition fusion sublayer and a feature splicing fusion sublayer, wherein the first multi-scale feature extraction module and the second multi-scale feature extraction module use the feature addition fusion sublayer to perform element-level feature addition fusion, the third multi-scale feature extraction module uses the feature splicing fusion sublayer to perform element-level feature splicing fusion, and the multi-scale feature extraction layer is used to extract the third feature to obtain a fourth feature, including:

[0145] Using the first convolution branch, the second convolution branch, and the third convolution branch in the first multi-scale feature extraction module to extract features of the third feature, respectively, to obtain first sub-features of three different scales;

[0146] Using the feature fusion layer in the first multi-scale feature extraction module, element-level feature addition is performed on the first sub-features of three different scales to obtain a first deformation field fusion feature;

[0147] The first convolution branch, the second convolution branch, and the third convolution branch in the second multi-scale feature extraction module are used to extract features of the first deformation field fusion feature, respectively, to obtain second sub-features of three different scales;

[0148] Using the feature fusion layer in the first multi-scale feature extraction module, element-level feature addition is performed on the second sub-features of three different scales to obtain a second deformation field fusion feature;

[0149] The first convolution branch, the second convolution branch, and the third convolution branch in the third multi-scale feature extraction module are used to extract features of the second deformation field fusion feature, respectively, to obtain third sub-features of three different scales;

[0150] The feature fusion layer in the third multi-scale feature extraction module is used to perform element-level feature concatenation on the third sub-features of three different scales to obtain the fourth feature.

[0151] Specifically, the multi-scale feature extraction layer includes a first multi-scale feature extraction module, a second multi-scale feature extraction module and a third multi-scale feature extraction module. Among them, the first multi-scale feature extraction module, the second multi-scale feature extraction module and the third multi-scale feature extraction module all adopt a multi-scale feature extraction module (Multi-Scale Feature Extraction Module, MSFEM).

[0152] In one embodiment of the present invention, the structure of the multi-scale feature extraction module MSFEM is as follows: Fig. 9 As shown. The input features can be defined as The dimension is defined as C In ×T In ×H×W. They are respectively fed into the three-dimensional convolution Conv3d (first convolution branch) with a convolution kernel size of k = (3, 3, 3), the three-dimensional convolution Conv3d (second convolution branch) with a convolution kernel size of k = (3, 5, 5), and the three-dimensional convolution Conv3d (third convolution branch) with a convolution kernel size of k = (3, 7, 7). Multi-scale feature extraction is performed. At the same time, residual connections are added to the two three-dimensional convolution Conv3d with convolution kernel sizes of k = (3, 5, 5) and k = (3, 7, 7) to enhance the representation ability of features.

[0153] It should be noted that the multi-scale feature extraction module uses two multi-scale feature fusion methods: element-level feature addition and feature splicing. When the output feature of MSFEM is an addition operation, the module is defined as MSFEMA, and when the output feature is a splicing operation, the module is defined as MSFEMB.

[0154] In one embodiment of the present invention, the first multi-scale feature extraction module and the second multi-scale feature extraction module adopt the element-level feature addition (MSFEMA), and the third multi-scale feature extraction module adopts the splicing operation (MSFEMB). The first multi-scale feature extraction module uses the first convolution branch, the second convolution branch, and the third convolution branch to extract the third feature, and performs element-level feature addition on the extracted features to obtain the first deformation field fusion feature. The second multi-scale feature extraction module uses the first convolution branch, the second convolution branch, and the third convolution branch to extract the first deformation field fusion feature, and performs element-level feature addition on the extracted features to obtain the second deformation field fusion feature. The third multi-scale feature extraction module uses the first convolution branch, the second convolution branch, and the third convolution branch to extract the second deformation field fusion feature, and performs element-level feature splicing on the extracted features to obtain the fourth feature.

[0155] The fourth characteristic can be defined as The characteristic dimension is defined as C Comb ×T Comb ×H×W. Then, After the second temporal pooling layer (Temporal Pooling, TP), the features are max-pooled in the time dimension and the deformation field feature f is obtained. TP , f TP The dimension is C Comb ×1×H×W. Furthermore, the deformation field characteristic f TP It is sent to the second generalized mean pooling layer (Generalized-Mean pooling, GeM) to achieve feature compression in the spatial dimension. The process can be expressed as: Among them, the deformation field characteristics is the final output feature of the multi-scale feature extraction layer, and its dimension is C Comb ×1×H×1, represents the average pooling operation on the W dimension, and p is a parameter that can be continuously updated as the network is trained. and have the same characteristic dimensions.

[0156] An embodiment of the present invention proposes a dual-path network model GaitDef based on deformation field and gait silhouette, which uses the deformation field to improve the perception of changes in human body movement. In the deformation field branch, since the deformation field data only has data values ​​at discrete human body contour point positions, the deformation field has data sparsity. In view of this feature, the embodiment of the present invention designs a multi-scale feature extraction module to improve the network's ability to characterize the deformation field. In the gait silhouette branch, since the human body changes dynamically during walking, and each subject has differences in body structure and behavioral habits, which causes their gait silhouettes to show different degrees of dynamic change differences, the embodiment of the present invention designs an inter-frame difference feature extraction module to capture dynamic difference characteristics, and constructs an inter-frame context feature extraction module to obtain context information between consecutive gait frames. Finally, the output features of each dual-path network are fused to obtain the final gait feature.

[0157] In another embodiment of the present invention, the gait recognition model may include a deformation field feature extraction network, a fully connected layer and a classifier, wherein the input end of the deformation field feature extraction network is used to input the deformation field data of the gait silhouette sequence, the output end of the deformation field feature extraction network is connected to the input end of the fully connected layer, the output end of the fully connected layer is connected to the input end of the classifier, the fully connected layer is used to output the gait features of the gait silhouette sequence, and the classifier is used to output the identity label of the gait silhouette sequence. Fig.10 FIG. 1 is a flow chart of a method for training a gait recognition model according to an embodiment of the present invention. Fig.10 As shown, the training method of the gait recognition model may include:

[0158] S501, obtaining a training sample set, wherein the training sample set includes deformation field data of a gait silhouette sequence and its corresponding real identity label;

[0159] S502, inputting the deformation field data of the gait silhouette sequence in the training sample set into the gait recognition model to obtain the gait features and predicted identity labels of the gait silhouette sequence;

[0160] S503, constructing a triplet loss function according to the gait feature, constructing a cross entropy loss function according to the predicted identity label and the true identity label, and constructing a gait recognition model loss function according to the triplet loss function and the cross entropy loss function;

[0161] S504, training the gait recognition model based on the gait recognition model loss function to obtain a trained gait recognition model.

[0162] Since related gait recognition models lack important explicit motion information, the spatiotemporal features directly extracted from silhouette sequences using 3D CNN lack the representation of human motion details.

[0163] Specifically, establish Fig.11 The gait recognition model shown. The gait recognition model includes a deformation field feature extraction network, a fully connected layer and a classifier connected in sequence. The embodiment of the present invention uses the deformation field feature extraction network to extract the deformation field data of the gait silhouette sequence to obtain the deformation field features. The deformation field feature extraction network of the embodiment of the present invention uses the deformation field data obtained through point set registration to characterize the shape information and motion characteristics of the human body edge contour. The fully connected layer (Separate Fully Connected Layer, Separate FC) maps the extracted deformation field features, and the classifier (Classifier) ​​classifies the mapped gait features to obtain the identity label of the gait silhouette sequence.

[0164] The triplet loss (Triplet Loss) is calculated for the gait features output by the fully connected layer, and the cross-entropy loss (Cross-EntropyLos s) is calculated based on the predicted identity label output by the classifier and the true identity label corresponding to the gait silhouette sequence, and the gait recognition model loss function is constructed based on the triplet loss function and the cross-entropy loss function. The gait recognition model is trained based on the gait recognition model loss function to obtain a trained gait recognition model.

[0165] In one embodiment of the present invention, the deformation field data is obtained by:

[0166] For each gait silhouette sequence, contour extraction is performed on each gait silhouette frame in the gait silhouette sequence to obtain a human body contour;

[0167] Discretize the human body contour to obtain discrete points of the human body contour;

[0168] The discrete points of the human body contour between adjacent frames are registered to obtain the deformation field data corresponding to each step silhouette sequence.

[0169] In one embodiment of the present invention, when acquiring a training sample set, gait silhouette sequences of different subjects in different states and different perspectives may be collected, and the true identity label of each gait silhouette sequence may be marked.

[0170] In another embodiment of the present invention, when acquiring the training sample set, the gait silhouette sequence in the CASIA-B data set and the true identity label of each gait silhouette sequence may be acquired.

[0171] The embodiment of the present invention uses the deformation field in point set registration to characterize the displacement amount and displacement direction of the human body contour during walking. The human body contour deformation field can not only effectively retain the human body morphological information, but also make up for the problem of lack of motion information of the human body contour in the GaitEdge method, laying a good foundation for the subsequent extraction of gait motion characteristics.

[0172] Since the change of human body contour is a typical non-rigid transformation, the embodiment of the present invention uses the Thin-Plate Spline RPM (TPS-RPM) algorithm to perform non-rigid point set registration on the human body contour points of adjacent frames, so as to characterize the correspondence between the human body contour points of adjacent frames and mine the motion characteristics implied by the subject during walking. It should be noted that in the classic non-rigid point set matching algorithm, the TPS-RPM algorithm uses annealing algorithm, thin plate spline and soft allocation for spatial transformation and outlier removal, so as to obtain the correspondence and transformation relationship between point sets, and has good robustness.

[0173] The process of obtaining the human body contour deformation field from the gait silhouette image is shown in Figure 7(a). The embodiment of the present invention extracts the human body contour from the gait silhouette image, discretizes the continuous human body contour using the equal-interval sampling method, and uses TPS-RPM to align the human body contour points of adjacent frames. The visualization effect of the human body contour alignment is shown in Figure 7(b), where the gray circles represent the human body contour points of the previous frame, the white crosses represent the human body contour points of the next frame, and the gray lines represent the alignment results of two adjacent frames.

[0174] The TPS-RPM algorithm mainly includes two key steps that are iterative: calculating the correspondence relationship M between point sets and solving the non-rigid spatial transformation function f. Assume that the two human contour point sets to be registered are V = {v a , a=1,2,...,K} and U={u i , i = 1, 2, ..., N}, define X as the set of intermediate transformation points, f as the transformation function from V to X, and each point in V is mapped to a new position x through f. a =f(v a ). The goal of point set registration can be expressed as minimizing the quadratic energy function E(Z, f). The expression for minimizing the quadratic energy function E(Z, f) is:

[0175]

[0176] Where M = {m ai} is the correspondence matrix between point sets V and U, and the size of M is J×K. Each element m in M ai The value range of m is [0,1], aiThe closer it is to 1, the higher the correspondence between the two points. λ||Lf|| 2 is the smoothness constraint on the objective function, Prevent too many points from being judged as wild points.

[0177] On this basis, the correspondence relationship (M) and transformation function (f) are iteratively updated. First, the correspondence matrix M is updated according to the annealing parameter T and the spatial transformation f until T reaches or is less than the termination temperature, and then M is normalized row by row and column by column. Then, the energy target E(f) is updated for the transformation function f according to the updated correspondence relationship M. The expression of the energy target E(f) is:

[0178]

[0179] in, Variable y a is the corresponding v calculated according to the correspondence matrix M in the point set U a point.

[0180] Finally, TPS (thin plate spline interpolation) is combined with the energy target E(f) to obtain the thin plate spline energy function E TPS (f), where the thin plate spline energy function E TPS The expression of (f) is:

[0181]

[0182] The human body contour points in adjacent frames are aligned to obtain the correspondence matrix M. The two-dimensional correspondence matrix M records the point v a Move to point u i The amount of displacement in the horizontal and vertical directions during the process.

[0183] The dimension of the gait silhouette sequence S of the embodiment of the present invention is 1×T×H×W, where T represents the number of input frames, H×W represents the height and width of the image, and the number of channels 1 represents that the gait silhouette is a grayscale image. After contour extraction, contour discretization and point set registration, the deformation field data D is output, and the dimension of D is 2×(T-1)×H×W, and the number of channels 2 represents the displacement in the horizontal and vertical directions.

[0184] The embodiment of the present invention introduces a non-rigid point set registration algorithm in the gait recognition task to solve the problem of insufficient motion information in gait silhouette data, and uses point set registration to generate deformation field data containing human body morphology and motion change information from human body contour points in adjacent frames.

[0185] In one embodiment of the present invention, the deformation field feature extraction network branch includes a second three-dimensional convolution layer, a second time aggregation layer, a multi-scale feature extraction layer, a second pooling layer, and a second generalized mean pooling layer connected in sequence, such as Fig.12 As shown, inputting the gait silhouette sequence in the training sample set into the gait recognition model may include:

[0186] S601, extracting and aggregating features of the deformation field data using a second three-dimensional convolution layer and a second time aggregation layer to obtain a third feature;

[0187] S602, extracting the third feature using a multi-scale feature extraction layer to obtain a fourth feature;

[0188] S603, pooling the fourth feature using a second pooling layer and a second generalized mean pooling layer to obtain a deformation field feature;

[0189] S604, mapping the deformation field features using a fully connected layer to obtain gait features;

[0190] S605: Classify the gait features using a classifier to obtain a predicted identity label.

[0191] Specifically, Fig.11 As shown in the figure, the deformation field data D is subjected to three-dimensional convolution Conv3d and temporal aggregation (TA) to obtain the third feature f D , the process can be expressed as: D =F TA (F Conv3d (D)), where f D The dimension is C TA ×TT A ×H×W,C TA Indicates the number of channels, T TA represents the dimension after time compression, F Conv3d (·) represents a 3D convolution with kernel size k = (3, 3, 3), F TA (·) represents the temporal aggregation operation, which is a three-dimensional convolution Conv3d with a kernel size of k = (3, 3, 3) and a step size of k = (3, 1, 1).

[0192] Since the valid values ​​in the deformation field data D are only distributed at the positions of the human body contour points, the deformation field data has data sparsity. In view of this characteristic, the embodiment of the present invention adopts a multi-scale feature extraction layer to extract features, and pools the extracted fourth feature using the second pooling layer and the second generalized mean pooling layer to obtain the deformation field feature.

[0193] The deformation field features obtained by pooling are mapped using a fully connected layer to obtain gait features. The gait features are classified using a classifier to obtain the predicted identity label.

[0194] In one embodiment of the present invention, the multi-scale feature extraction layer may include a first multi-scale feature extraction module, a second multi-scale feature extraction module and a third multi-scale feature extraction module, the first multi-scale feature extraction module, the second multi-scale feature extraction module and the third multi-scale feature extraction module each include a first convolution branch, a second convolution branch, a third convolution branch and a feature fusion layer, the feature fusion layer includes a feature addition fusion sublayer and a feature splicing fusion sublayer, wherein the first multi-scale feature extraction module and the second multi-scale feature extraction module use the feature addition fusion sublayer to perform element-level feature addition fusion, and the third multi-scale feature extraction module uses the feature splicing fusion sublayer to perform element-level feature splicing fusion, such as Fig.13 The method of extracting the third feature using a multi-scale feature extraction layer to obtain a fourth feature may include:

[0195] S701, using the first convolution branch, the second convolution branch, and the third convolution branch in the first multi-scale feature extraction module to extract features of the third feature respectively, to obtain first sub-features of three different scales;

[0196] S702, using a feature fusion layer in a first multi-scale feature extraction module to perform element-level feature addition on the first sub-features of three different scales to obtain a first deformation field fusion feature;

[0197] S703, using the first convolution branch, the second convolution branch, and the third convolution branch in the second multi-scale feature extraction module to respectively extract features from the first deformation field fusion feature to obtain second sub-features of three different scales;

[0198] S704, using the feature fusion layer in the first multi-scale feature extraction module to perform element-level feature addition on the second sub-features of three different scales to obtain a second deformation field fusion feature;

[0199] S705, using the first convolution branch, the second convolution branch, and the third convolution branch in the third multi-scale feature extraction module to respectively extract features of the second deformation field fusion feature to obtain third sub-features of three different scales;

[0200] S706 , using the feature fusion layer in the third multi-scale feature extraction module to perform element-level feature concatenation on the third sub-features of three different scales to obtain a fourth feature.

[0201] Specifically, the multi-scale feature extraction layer includes a first multi-scale feature extraction module, a second multi-scale feature extraction module and a third multi-scale feature extraction module. Among them, the first multi-scale feature extraction module, the second multi-scale feature extraction module and the third multi-scale feature extraction module all adopt a multi-scale feature extraction module (Multi-Scale Feature Extraction Module, MSFEM).

[0202] In one embodiment of the present invention, the structure of the multi-scale feature extraction module MSFEM is as follows: Fig. 9 As shown. The input features can be defined as The dimension is defined as C In ×T In ×H×W. They are respectively fed into the three-dimensional convolution Conv3d (first convolution branch) with a convolution kernel size of k = (3, 3, 3), the three-dimensional convolution Conv3d (second convolution branch) with a convolution kernel size of k = (3, 5, 5), and the three-dimensional convolution Conv3d (third convolution branch) with a convolution kernel size of k = (3, 7, 7). Multi-scale feature extraction is performed. At the same time, residual connections are added to the two three-dimensional convolution Conv3d with convolution kernel sizes of k = (3, 5, 5) and k = (3, 7, 7) to enhance the representation ability of features.

[0203] It should be noted that the multi-scale feature extraction module uses two multi-scale feature fusion methods: element-level feature addition and feature splicing. When the output feature of MSFEM is an addition operation, the module is defined as MSFEMA, and when the output feature is a splicing operation, the module is defined as MSFEMB.

[0204] In one embodiment of the present invention, the first multi-scale feature extraction module and the second multi-scale feature extraction module adopt the element-level feature addition (MSFEMA), and the third multi-scale feature extraction module adopts the splicing operation (MSFEMB). The first multi-scale feature extraction module uses the first convolution branch, the second convolution branch, and the third convolution branch to extract the third feature, and performs element-level feature addition on the extracted features to obtain the first deformation field fusion feature. The second multi-scale feature extraction module uses the first convolution branch, the second convolution branch, and the third convolution branch to extract the first deformation field fusion feature, and performs element-level feature addition on the extracted features to obtain the second deformation field fusion feature. The third multi-scale feature extraction module uses the first convolution branch, the second convolution branch, and the third convolution branch to extract the second deformation field fusion feature, and performs element-level feature splicing on the extracted features to obtain the fourth feature.

[0205] The fourth characteristic can be defined as The characteristic dimension is defined as C Comb ×T Comb ×H×W. Then, After the second temporal pooling layer (Temporal Pooling, TP), the features are max-pooled in the time dimension and the deformation field feature f is obtained. TP , f TP The dimension is C Comb ×1×H×W. Furthermore, the deformation field characteristic f TP It is sent to the second generalized mean pooling layer (Generalized-Mean pooling, GeM) to achieve feature compression in the spatial dimension. The process can be expressed as: Among them, the deformation field characteristics is the final output feature of the multi-scale feature extraction layer, and its dimension is C Comb ×1×H×1, represents the average pooling operation on the W dimension, and p is a parameter that can be continuously updated as the network is trained.

[0206] In one embodiment of the present invention, the gait recognition model further includes a gait silhouette feature extraction network and a fusion layer, the input end of the gait silhouette feature extraction network is used to input a gait silhouette sequence, the output end of the gait silhouette feature extraction network is connected to the first input end of the fusion layer, the second input end of the fusion layer is connected to the output end of the deformation field feature extraction network, and the output end of the fusion layer is connected to the fully connected layer; the training method further includes:

[0207] The gait silhouette feature extraction network is used to extract features from the gait silhouette sequence to obtain gait silhouette features, and the gait silhouette features and deformation field features are fused using the fusion layer;

[0208] The deformation field features are mapped using the fully connected layer, including:

[0209] The deformation field features fused with gait silhouette features are mapped using a fully connected layer.

[0210] Specifically, the gait silhouette sequence is input into the gait silhouette feature extraction network, which extracts features from the gait silhouette sequence to obtain gait silhouette features, and the gait silhouette features and deformation field features are fused using the fusion layer, and the deformation field features of the fused gait silhouette features are mapped using the fully connected layer to obtain gait features. The gait features are classified using a classifier to obtain a predicted identity label.

[0211] In another embodiment of the present invention, the network structure of the gait recognition model is as follows: Figure 6As shown, the network structure of the gait recognition model may include a branch of a deformation field feature extraction network, a branch of a gait silhouette feature extraction network, a fully connected layer (Separate Fully Connected Layer, Separate FC) and a classifier.

[0212] Specifically, the deformation field feature extraction network uses the deformation field data obtained through point set registration to characterize the shape information and motion characteristics of the human body edge contour. The deformation field data and gait silhouette sequence are respectively passed through the feature extraction network to obtain the deformation field features and gait silhouette features, and then the final gait features are obtained through linear feature fusion. The gait features are sequentially passed through the fully connected layer and the classifier. The triplet loss (Triplet Loss) is calculated for the gait features output by the fully connected layer, and the cross-entropy loss (Cross-Entropy Loss) is calculated for the classification results output by the classifier.

[0213] In one embodiment of the present invention, the gait silhouette feature extraction network includes a first three-dimensional convolution layer, a first time aggregation layer, an inter-frame difference and context information feature extraction layer, a first pooling layer and a first generalized mean pooling layer connected in sequence, such as Fig.14 As shown, using a gait silhouette feature extraction network to extract features from a gait silhouette sequence may include:

[0214] S801, extracting and aggregating features of a gait silhouette sequence using a first three-dimensional convolutional layer and a first temporal aggregation layer to obtain a first feature;

[0215] S802, extracting the first feature using a feature extraction layer to obtain a second feature;

[0216] S803: Pooling the second feature using the first pooling layer and the first generalized mean pooling layer to obtain a gait silhouette feature.

[0217] Since the human body changes dynamically during walking and each subject has different body structures and behavioral habits, their gait silhouettes show different degrees of dynamic changes. The embodiment of the present invention adopts inter-frame difference and context information feature extraction layers to capture dynamic difference characteristics and obtain context information between consecutive gait frames.

[0218] Specifically, Figure 2As shown in Figure 1, the gait silhouette sequence of the input gait silhouette feature extraction network can be defined as S. The dimension of the gait silhouette sequence S is 1×T×H×W, where T represents the number of input frames, H×W represents the height and width of the image, and the number of channels 1 represents that the gait silhouette is a grayscale image. The gait silhouette sequence S is sequentially processed through the three-dimensional convolution layer Conv3d and the temporal aggregation (TA) operation to obtain the first feature f s , the first feature f s It is sent to the inter-frame difference and context information feature extraction layer, and the extracted second feature is pooled using the first pooling layer TP (Temporal Pooling, TP) and the first generalized mean pooling layer (Generalized-Mean pooling, GeM) to obtain the gait silhouette feature.

[0219] In one embodiment of the present invention, the expression of the gait recognition model loss function is:

[0220] L all =L ce +L tri

[0221] Among them, L all is the loss function of the gait recognition model, L ce is the cross entropy loss function, L tri is the triplet loss function;

[0222] Cross entropy loss function L ce The expression is:

[0223]

[0224] Where N is the number of identity features, y i is the identity label corresponding to the i-th sample, p i is the predicted probability of the i-th sample;

[0225] Triplet loss function L tri The expression is:

[0226]

[0227] Among them, H is the number of triplets, and the triplets are is the feature a of the subject in the h-th triple, is the hth triple with The characteristics q belonging to the same subject, is the hth triple with The features p belonging to different subjects, Features With features The Euclidean distance between Features With features The Euclidean distance between them, and m represents the boundary threshold.

[0228] The embodiment of the present invention is to determine the deformation field characteristics and gait silhouette features Perform element-level feature summation and obtain fused features Then, f Comb It passes through the fully connected layer and the classifier in sequence, where the output features of the fully connected layer are used to calculate the triplet loss to shorten the distance between samples within the class and push the distance between samples in distant classes, and the output classification results of the classifier are used to calculate the cross entropy loss to constrain the network to learn identity discriminative features.

[0229] For triplet loss, assuming that a feature triplet (f a , f p , f q ), where f a and f q The characteristics belonging to the same subject, f a and f p Features belonging to different subjects, when there are H triplets When , the triplet loss can be expressed as:

[0230]

[0231] in, and represents the Euclidean distance between two features, and m represents the boundary threshold.

[0232] The cross entropy loss in the embodiment of the present invention can be expressed as:

[0233]

[0234] Where N is the number of identity features, y i is the one-hot (unique hot encoding) type identity label corresponding to the i-th sample, p i represents the predicted probability of the i-th sample.

[0235] Finally, the gait recognition model loss function L all It can be expressed as:

[0236] L all =L ce +L tri

[0237] Among them, L allis the loss function of the gait recognition model, L ce is the cross entropy loss function, L tri is the triple loss function.

[0238] In one embodiment of the present invention, non-rigid point set registration is introduced into gait recognition, and the deformation field between adjacent gait frames is used to characterize the displacement of the human body contour during walking, thereby improving the dynamic perception ability of human body morphological changes. In one embodiment of the present invention, the gait recognition model is a two-stream convolutional neural network (Gait Def) based on the deformation field of the human body contour, and the network model is composed of two feature extraction branches: the deformation field and the gait silhouette. A multi-scale feature extraction module is designed for the sparsity of the deformation field data to obtain the multi-level spatial structure information of the deformation field. A dynamic difference capture module and a context information enhancement module are proposed for the gait silhouette to capture the changing characteristics of the dynamic area and enhance the gait representation ability by using context information. The output features of the dual-branch network structure are fused to obtain the final gait representation. For the CASIA-B data set, the Rank-1 accuracy can reach 97.6%, 95.7% and 87.2% respectively in the cases of normal walking, backpacking and dressing, and the comparison experiment and ablation experiment respectively verify the effectiveness of the deformation field data and the network structure.

[0239] In one embodiment of the present invention, the gait silhouette size input to the gait recognition model is H×W=64×44. In the training phase, the length of the input gait sequence is T=30. In the test phase, all frames in the gait sequence are sent to the network model to extract gait features. The boundary threshold in the triplet loss is set to m=1.2, and the initial value of the parameter p in the GeM layer is set to 6.5. The number of feature channels output by the MSFEM and ACFEM modules in the network in the three stages (A, A, B) are (64, 128, 128), respectively. In addition, in the network training phase, Batchsize is set to 8×8=64, where the number of subjects extracted from each group of training data is set to 8, and the number of samples selected from each subject is 8. This paper uses Adam as the optimizer, the learning rate is set to 0.0001, and the number of training iterations (epoch) is set to 100K.

[0240] The hardware environment used in the embodiment of the present invention is 2 cards NVIDIA GeForce RTX 3090, and the CPU model is Intel (R) Core (TM) i9-13900KF. The experimental environment uses Ubuntu 20.04.6, Cuda version 12.2, Python version 3.9.16, and Pytorch version 1.10.2.

[0241] The embodiments of the present invention respectively verify the effectiveness of deformation field data and each network module on the CASIA-B dataset, and compare them with 7 advanced step recognition algorithms. At the same time, they significantly outperform existing methods in two setting scenarios: dressing and carrying objects.

[0242] The comparison results of GaitDef and the current mainstream gait recognition methods on the CASIA-B dataset are shown in Table 1. The compared methods include GaitSet, GaitPart, GaitGL, CSTL, Lagrange, MetaGait and GaitGCI-T. All results are the average values ​​of 11 perspectives and do not include the same perspective.

[0243] Table 1 Rank-1 accuracy (%) of CASIA-B dataset under all viewing angles and different settings, excluding the same viewing angle

[0244]

[0245]

[0246]

[0247] As can be seen from Table 1, under the conditions of NM, BG and CL, the Rank-1 accuracy of GaitDef can reach 97.6%, 95.7% and 87.2% respectively.

[0248] Among them, under the NM condition, the method proposed in the embodiment of the present invention exceeds four SOTA (State Of The Art) methods, namely GaitSet, GaitPart, GaitGL and Lagrange, and is slightly lower than CSTL, GaitGCI and MetaGait.

[0249] Under BG conditions, the method proposed in the embodiment of the present invention achieves SOTA performance, and GaitDef has obvious advantages over methods before 2021. Compared with MetaGait and GaitGCI-T, GaitDef's accuracy is also 0.5% and 0.7% higher, respectively.

[0250] Under CL conditions, the Rank-1 accuracy of GaitDef is also significantly higher than that of other compared methods, and the experimental results are 0.3% and 0.8% higher than MetaGait and GaitGCI-T, respectively.

[0251] The experimental results show that although MetaGait achieves the best results under NM conditions, GaitDef performs better overall, and GaitDef can achieve higher recognition accuracy when dealing with backpack and clothing scenarios.

[0252] Before human body contour registration, the embodiment of the present invention discretizes the human body contour by sampling at equal intervals, and the number of human body contour points in the data set is less than 140. Therefore, this paper sets four different contour point numbers when sampling contour points: 50, 80, 100 and 120. A comparative test is set to compare the experimental results of deformation field data obtained based on different sampling point numbers after passing through the deformation field network branch. Fig.15 The comparative experimental results in show that under the conditions of NM, BG and CL, the gait recognition accuracy gradually increases with the increase of the number of human contour points. Among them, under the NM condition, the highest recognition accuracy of 93.1 was achieved when the number of sampling points was 100; under the BG and CL conditions, the phenomenon of the accuracy increasing with the increase of contour points was more obvious, and reached 86.4% and 69.2% when the number of sampling points was 120. And the experimental results of 120 human contour points are 1.3%, 3.3% and 3.8% higher than the experimental results based on 50 human contour points under the conditions of NM, BG and CL, respectively. In addition, from the comparative test results, it can be found that even if there are only 50 valid data points in the deformation field, the recognition accuracy can reach 91.7%, 80.1% and 65.4% under the conditions of NM, BG and CL, respectively, which proves the effectiveness of the human contour deformation field data.

[0253] The embodiment of the present invention sets up multiple groups of ablation experiments for different branch network structures to verify the effectiveness of different branches. The ablation experiment results are shown in Table 2. The input data of the deformation field branch in the experiment is the deformation field data obtained based on 120 human body contour points.

[0254] Table 2 Rank-1 accuracy of different branch network structures on CASIA-B dataset (%)

[0255]

[0256] In the deformation field network structure, when the convolution layers with kernel sizes of 3, 5, and 7 are used separately in the MSFEM module to extract deformation field features, their average Rank-1 indexes in the three states can reach 75.7%, 81.5%, and 81.5%, respectively; when two different sizes of convolution kernels are used at the same time to extract gait features, the recognition accuracy is significantly higher than the result when only one size of convolution kernel is used; when three different sizes of convolution kernels are used at the same time to extract deformation field features, the Rank-1 accuracy can reach 93.0%, 86.4%, and 69.2% under NM, BG, and CL conditions, respectively, and the performance is optimal. The experimental results prove the effectiveness of the multi-scale convolution operation in MSFEM.

[0257] In the gait silhouette branch, ACFEM can achieve Rank-1 index of 91.7%, 92.1% and 91.4% using only the global branch, inter-frame difference feature extraction branch and context feature extraction branch respectively, among which the inter-frame difference extraction branch has the best performance. In addition, the accuracy achieved when using any two feature extraction branches at the same time is significantly higher than that of using the inter-frame difference extraction branch alone. After fusing the three feature extraction branches, ACFEM can achieve 97.5%, 95.4% and 86.6% under NM, BG and CL conditions, achieving the best results.

[0258] The embodiment of the present invention demonstrates the experimental results obtained after feature fusion of the deformation field branch and the gait silhouette branch. Its Rank-1 accuracy can reach 97.6%, 95.7% and 87.2% under NM, BG, and CL conditions, respectively, and the average Ran k-1 accuracy can reach 93.5%. This result exceeds the results of using the deformation field network branch and the gait silhouette branch alone, verifying the effectiveness of feature fusion.

[0259] The embodiment of the present invention introduces point set registration into the gait recognition task, and uses the human body contour deformation field obtained by point set registration to characterize the human body shape and the dynamic changes of the contour during walking. The comparative experiment verifies the validity of the deformation field data. On this basis, a dual-stream gait recognition network structure GaitDef based on the human body contour deformation field is proposed. A multi-scale feature extraction module is designed for the sparsity of the deformation field data, and multi-scale convolution is used to capture the spatial characteristics of the human body. An inter-frame difference capture module and a context information feature enhancement module are proposed for the gait silhouette sequence to enhance the network's ability to characterize the dynamic regional change characteristics and context features during walking. The ablation experiment verifies the validity of the network structure. Subsequent work will start from the perspective transformation problem existing in gait recognition to further improve the performance of the network model.

[0260] The invention provides a gait recognition method.

[0261] Fig.16 FIG. 1 is a flow chart of a gait recognition method according to an embodiment of the present invention. Fig.16 As shown, the gait recognition method may include:

[0262] S901, inputting the gait silhouette sequence to be identified into a trained gait recognition model to obtain the category of the gait silhouette sequence to be identified, wherein the trained gait recognition model is trained using the training method of the gait recognition model as described above.

[0263] Specifically, a gait silhouette sequence to be identified is obtained, and the gait silhouette sequence to be identified is input into the above-mentioned trained gait recognition model to obtain the category of the gait silhouette sequence to be identified, that is, identity information.

[0264] The gait recognition method of the embodiment of the present invention utilizes the trained gait recognition model as described above and determines the category, i.e., identity information, of the gait silhouette sequence to be recognized based on the gait silhouette sequence to be recognized, and has the advantage of high gait recognition accuracy.

[0265] The invention provides a gait recognition method.

[0266] Fig.17 FIG. 1 is a flow chart of a gait recognition method according to another embodiment of the present invention. Fig.17 As shown, the gait recognition method may include:

[0267] S1001, inputting deformation field data of the gait silhouette sequence to be identified into a trained gait recognition model to obtain the category of the gait silhouette sequence to be identified, wherein the trained gait recognition model is trained using the training method of the gait recognition model as described above.

[0268] Specifically, the deformation field data of the gait silhouette sequence to be identified is obtained, and the deformation field data of the gait silhouette sequence to be identified is input into the above-mentioned trained gait recognition model to obtain the category of the gait silhouette sequence to be identified, that is, identity information.

[0269] The gait recognition method of the embodiment of the present invention utilizes the trained gait recognition model as described above, and determines the category, i.e., identity information, of the gait silhouette sequence to be recognized based on the deformation field data of the gait silhouette sequence to be recognized, and has the advantage of high gait recognition accuracy.

[0270] The present invention provides a computer-readable storage medium.

[0271] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the training method of the gait recognition model as described above is implemented.

[0272] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the gait recognition method as described above is implemented.

[0273] The invention provides a controller.

[0274] In this embodiment, the controller may include a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the training method of the gait recognition model as described above is implemented.

[0275] In this embodiment, the controller may include a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the gait recognition method as described above is implemented.

[0276] Fig.18 4 is a structural block diagram of a controller according to an embodiment of the present invention.

[0277] like Fig.18 As shown, the controller 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the controller 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the controller 500 does not constitute a limitation on the embodiments of the present invention.

[0278] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0279] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.18 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0280] The memory 503 is used to store a computer program corresponding to the training method of the gait recognition model or the gait recognition method of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the content shown in the above method embodiment. Fig.18 The controller 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0281] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0282] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0283] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0284] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0285] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0286] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0287] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0288] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A training method for a gait recognition model, characterized in that: The gait recognition model includes a gait silhouette feature extraction network, a fully connected layer and a classifier, the input end of the gait silhouette feature extraction network is used to input a gait silhouette sequence, the output end of the gait silhouette feature extraction network is connected to the input end of the fully connected layer, the output end of the fully connected layer is connected to the input end of the classifier, the fully connected layer is used to output the gait features of the gait silhouette sequence, and the classifier is used to output the identity label of the gait silhouette sequence, and the training method includes: Acquire a training sample set, wherein the training sample set includes a gait silhouette sequence and its corresponding true identity label; Inputting the gait silhouette sequence in the training sample set into a gait recognition model to obtain gait features and predicted identity labels of the gait silhouette sequence; constructing a triplet loss function according to the gait feature, constructing a cross entropy loss function according to the predicted identity label and the true identity label, and constructing a gait recognition model loss function according to the triplet loss function and the cross entropy loss function; The gait recognition model is trained based on the gait recognition model loss function to obtain a trained gait recognition model.

2. The training method of the gait recognition model according to claim 1, characterized in that: The gait silhouette feature extraction network includes a first three-dimensional convolution layer, a first time aggregation layer, an inter-frame difference and context information feature extraction layer, a first pooling layer and a first generalized mean pooling layer connected in sequence, and inputs the gait silhouette sequence in the training sample set into the gait recognition model, including: Using the first three-dimensional convolution layer and the first time aggregation layer to extract and aggregate features of the gait silhouette sequence to obtain a first feature; Extracting the first feature using the inter-frame difference and context information feature extraction layer to obtain a second feature; Using the first pooling layer and the first generalized mean pooling layer to pool the second feature to obtain a gait silhouette feature; Mapping the gait silhouette feature using the fully connected layer to obtain the gait feature; The gait feature is classified using the classifier to obtain the predicted identity label.

3. The training method of the gait recognition model according to claim 2, characterized in that: The inter-frame difference and context information feature extraction layer includes a first feature extraction module, a second feature extraction module and a third feature extraction module. The first feature extraction module, the second feature extraction module and the third feature extraction module all include a global feature extraction branch, an inter-frame difference feature extraction branch, a context information extraction branch and a feature fusion layer. The feature fusion layer includes a feature addition fusion sublayer and a feature splicing fusion sublayer. The first feature extraction module and the second feature extraction module use the feature splicing fusion sublayer to perform element-level feature addition fusion, and the third feature extraction module uses the feature splicing fusion sublayer to perform channel dimension feature splicing fusion. The inter-frame difference and context information feature extraction layer is used to extract the first feature to obtain the second feature, including: Utilize the global feature extraction branch in the first feature extraction module to perform overall feature extraction on the first feature, the inter-frame difference feature extraction branch to perform feature extraction on the feature difference between adjacent frames in the first feature, and the context information extraction branch to perform information extraction on the feature mean between adjacent frames in the first feature, to obtain a first overall feature, a first inter-frame difference feature, and a first context feature; Using the feature fusion layer in the first feature extraction module, element-level feature addition is performed on the first overall feature, the first inter-frame difference feature, and the first context feature to obtain a first silhouette fusion feature; The global feature extraction branch in the second feature extraction module is used to perform overall feature extraction on the first silhouette fusion feature, the inter-frame difference feature extraction branch is used to perform feature extraction on the feature difference between adjacent frames in the first silhouette fusion feature, and the context information extraction branch is used to perform information extraction on the feature mean between adjacent frames in the first silhouette fusion feature, so as to obtain a second overall feature, a second inter-frame difference feature, and a second context feature; Using the feature fusion layer in the second feature extraction module, perform element-level feature addition on the second overall feature, the second inter-frame difference feature, and the second context feature to obtain a second silhouette fusion feature; The global feature extraction branch in the third feature extraction module is used to perform overall feature extraction on the second silhouette fusion feature, the inter-frame difference feature extraction branch is used to perform feature extraction on the feature difference between adjacent frames in the second silhouette fusion feature, and the context information extraction branch is used to perform information extraction on the feature mean between adjacent frames in the second silhouette fusion feature, so as to obtain a third overall feature, a third inter-frame difference feature, and a third context feature; The feature fusion layer in the second feature extraction module is used to perform channel dimension feature splicing on the third overall feature, the third inter-frame difference feature, and the third context feature to obtain the second feature.

4. The training method of the gait recognition model according to claim 2, characterized in that: The gait recognition model also includes a deformation field feature extraction network and a fusion layer, wherein an input end of the deformation field feature extraction network is used to input deformation field data of the gait silhouette sequence, an output end of the deformation field feature extraction network is connected to a first input end of the fusion layer, a second input end of the fusion layer is connected to an output end of the gait silhouette feature extraction network, and an output end of the fusion layer is connected to the fully connected layer; The training method further comprises: Using the deformation field feature extraction network to extract features from the deformation field data of the gait silhouette sequence to obtain deformation field features, and using the fusion layer to fuse the gait silhouette features and the deformation field features; Mapping the gait silhouette feature by using the fully connected layer includes: The fully connected layer is used to map the gait silhouette features fused with the deformation field features.

5. The training method of the gait recognition model according to claim 4, characterized in that: The deformation field data is obtained in the following manner: For each of the gait silhouette sequences, performing contour extraction on each gait silhouette frame in the gait silhouette sequence to obtain a human body contour; Discretizing the human body contour to obtain human body contour discrete points; The discrete points of the human body contour between adjacent frames are registered to obtain the deformation field data corresponding to each gait silhouette sequence.

6. The training method of the gait recognition model according to claim 4, characterized in that: The deformation field feature extraction network branch includes a second three-dimensional convolution layer, a second time aggregation layer, a multi-scale feature extraction layer, a second pooling layer, and a second generalized mean pooling layer connected in sequence, and the deformation field feature extraction network branch is used to extract features from the deformation field data of the gait silhouette sequence, including: Using the second three-dimensional convolution layer and the second time aggregation layer to perform feature extraction and aggregation on the deformation field data to obtain a third feature; Using the multi-scale feature extraction layer to extract the third feature, to obtain a fourth feature; The fourth feature is pooled by using the second pooling layer and the second generalized mean pooling layer to obtain the deformation field feature.

7. The training method of the gait recognition model according to claim 1, characterized in that: The expression of the loss function of the gait recognition model is: LallL ce +Ltr i Among them, L all is the loss function of the gait recognition model, L ce is the cross entropy loss function, L tri is the triple loss function; The cross entropy loss function L ce The expression is: Where N is the number of identity features, y i is the identity label corresponding to the i-th sample, p i is the predicted probability of the i-th sample; The triplet loss function L tri The expression is: Among them, H is the number of triplets, and the triplets are is the feature of the subject in the hth triplet, is the hth triple with Characteristics belonging to the same subject, is the hth triple with Characteristics belonging to different subjects, Features With features The Euclidean distance between Features With features The Euclidean distance between them, and m represents the boundary threshold.

8. A gait recognition method, characterized in that: The identification method comprises: The gait silhouette sequence to be identified is input into a trained gait recognition model to obtain the category of the gait silhouette sequence to be identified, wherein the trained gait recognition model is trained using the training method for the gait recognition model as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the gait recognition model as described in any one of claims 1 to 7, or the gait recognition method as described in claim 8.

10. A controller, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by the processor, the training method of the gait recognition model as described in any one of claims 1 to 7 is implemented, or the gait recognition method as described in claim 8.

Images