Training method of gait recognition model, gait recognition method, medium and controller

Abstract

The invention discloses a training method of a gait recognition model, a gait recognition method, a medium and a controller, and the training method comprises the steps: carrying out the periodic training of the gait recognition model for multiple times through a training image set, and for each training period, executing the following operations: obtaining the training image set, the training image set comprises an original gait silhouette sequence and a corresponding augmented gait silhouette sequence; inputting the original gait silhouette sequence and the augmented gait silhouette sequence into a gait recognition model to obtain a target gait feature and a classification probability corresponding to the target gait feature; obtaining a triple loss function according to the target gait feature, obtaining a cross entropy loss function according to the classification probability corresponding to the target gait feature, and obtaining a total loss function according to the triple loss function and the cross entropy loss function; and according to the total loss function, adjusting parameters of the gait recognition model to obtain a trained gait recognition model. According to the training method, the identity recognition accuracy of the gait recognition model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of gait recognition, and in particular to a training method for a gait recognition model, a gait recognition method and medium, and a controller. Background Art

[0002] Gait recognition is a technology that identifies people based on their morphology and movement patterns. In recent years, it has been widely used in many fields such as public security and finance. Gait recognition technology can acquire gait data at a long distance, in an uncontrolled and unperceptible condition, and accurately identify pedestrians by extracting the apparent features of human morphology and the dynamic features of movement patterns from gait data.

[0003] According to the data type of the research object, gait recognition can be roughly divided into model-based methods and representation-based methods. The model-based method first uses the human posture estimation method to extract the coordinates of the key points of the human body, then explicitly models the human body structure and walking style, and extracts gait features. Although the model-based method can cope with changes in perspective more robustly, it is greatly affected by the performance of the human posture estimation model and has a high computational overhead. The representation-based method directly uses a convolutional neural network to extract discriminative identity features from gait images.

[0004] In recent years, benefiting from the powerful feature extraction capability of CNN (Convolutional Neural Network), representation-based gait recognition methods are often better than model-based methods, and thus have attracted more attention from researchers. However, representation-based gait recognition methods have the problem of poor adaptability to complex scenes. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one object of the present invention is to propose a training method for a gait recognition model to improve the identity recognition accuracy of the gait recognition model.

[0006] The second objective of the present invention is to provide a gait recognition method.

[0007] A third object of the present invention is to provide a computer-readable storage medium.

[0008] A fourth object of the present invention is to provide a controller.

[0009] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a training method for a gait recognition model, wherein the gait recognition model includes a first branch network, a second branch network, a multi-stage feature aggregation network, an element-level feature summation layer, a fully connected layer and a classifier, wherein the first branch network is used to input an original gait silhouette sequence, the second branch network is used to input an augmented gait silhouette sequence, the output end of the first branch network and the output end of the second branch network are connected to the input end of the multi-stage feature aggregation network, the output end of the first branch network and the output end of the multi-stage feature aggregation network are connected to the input end of the element-level feature summation layer, the output end of the element-level feature summation layer is connected to the input end of the classifier through the fully connected layer, and the classifier is used to output the target gait. Features and their corresponding classification probabilities, the training method comprises: using a training image set to perform multiple periodic training on a gait recognition model, wherein for each training cycle, the following operations are performed: obtaining a training image set, wherein the training image set comprises an original gait silhouette sequence and its corresponding augmented gait silhouette sequence; inputting the original gait silhouette sequence and the augmented gait silhouette sequence into the gait recognition model to obtain target gait features and their corresponding classification probabilities; obtaining a triplet loss function according to the target gait features, obtaining a cross entropy loss function according to the classification probabilities corresponding to the target gait features, and obtaining a total loss function according to the triplet loss function and the cross entropy loss function; adjusting the parameters of the gait recognition model according to the total loss function to obtain a trained gait recognition model.

[0010] According to the training method of the gait recognition model of the embodiment of the present invention, the original gait silhouette sequence is amplified and an augmented gait silhouette sequence is added to provide the gait recognition model with diverse motion appearance information and pedestrian identity clues. By extracting and fusing the original gait silhouette sequence and the augmented gait features, the identity recognition accuracy of the gait recognition model is improved.

[0011] In addition, the training method of the gait recognition model proposed in the above embodiment of the present invention may also have the following additional technical features:

[0012] According to one embodiment of the present invention, the inputting of the original gait silhouette sequence and the augmented gait silhouette sequence into the gait recognition model to obtain the target gait features and their corresponding classification probabilities includes: inputting the original gait silhouette sequence into the first branch network to obtain the first layer fusion features, the first aggregation features, the third layer fusion features and the original gait features; inputting the augmented gait silhouette sequence into the second branch network to obtain the augmented gait features; inputting the first layer fusion features, the first aggregation features, the third layer fusion features and the augmented gait features into the multi-stage feature aggregation network to obtain the first fusion features; inputting the original gait features and the first fusion features into the element-level feature summation layer to obtain the second fusion features; and inputting the second fusion features into the fully connected layer and the classifier in sequence to obtain the target gait features and their corresponding classification probabilities.

[0013] According to one embodiment of the present invention, the first branch network includes a first feature extraction module, a second feature extraction module, a first time aggregation module, a third feature extraction module, a time pooling module and a space pooling module connected in sequence, and the original gait silhouette sequence is input into the first branch network to obtain the first layer of fusion features, the first aggregation features, the third layer of fusion features and the original gait features, including: using the first feature extraction module to perform dynamic change feature extraction and multi-granularity feature extraction on the original gait silhouette sequence, and performing element-level feature summation operation on the extracted first dynamic change feature and the first multi-granularity feature to obtain the first layer of fusion features; using the second feature extraction module to perform dynamic change feature extraction and multi-granularity feature extraction on the first layer of fusion features, The method comprises the following steps: extracting features, and performing an element-level feature summation operation on the extracted second dynamic change features and the second multi-granularity features to obtain the second layer fusion features; using the first time aggregation module, performing a time dimension compression aggregation operation on the second layer fusion features to obtain the first aggregation features; using the third feature extraction module, performing dynamic change feature extraction and multi-granularity feature extraction on the first aggregation features respectively, and performing an element-level feature summation operation on the extracted third dynamic change features and the third multi-granularity features to obtain the third layer fusion features; using the time pooling module to perform a time dimension feature pooling operation on the third layer fusion features to obtain the pooling features; using the space pooling module to perform space dimension feature compression on the pooling features to obtain the original gait features.

[0014] According to one embodiment of the present invention, the augmented gait silhouette sequence includes at least two of a leaning forward walking sequence, a leaning backward walking sequence, a going-upstairs walking sequence, and a going-downstairs walking sequence, the second branch network includes a shallow feature extraction module and a channel-level feature aggregation module, and the augmented gait silhouette sequence is input into the second branch network to obtain an augmented gait feature, comprising: using the shallow feature extraction module to perform feature extraction on at least two of the leaning forward walking sequence, the leaning backward walking sequence, the going-upstairs walking sequence, and the going-downstairs walking sequence, respectively, to obtain a first gait feature, a second gait feature, a third gait feature, and at least two of a fourth gait feature; using the channel-level feature aggregation module to perform feature aggregation on at least two of the first gait feature, the second gait feature, the third gait feature, and the fourth gait feature to obtain the augmented gait feature.

[0015] According to an embodiment of the present invention, the multi-stage feature aggregation network includes a first feature aggregation module, a second time aggregation module, a second feature aggregation module and a third feature aggregation module, and the first layer fusion feature, the first aggregation feature, the third layer fusion feature and the augmented gait feature are input into the multi-stage feature aggregation network to obtain a first fusion feature, including: using the first feature aggregation module to perform a channel-level feature splicing operation on the augmented gait feature and the first layer fusion feature, performing feature channel adjustment on the first splicing feature obtained by the splicing operation to obtain a first adjusted feature, performing feature channel adjustment and feature compression processing on the first splicing feature obtained by the splicing operation to obtain a first compressed feature; using the second time aggregation module to perform feature compression processing on the first adjusted feature A compression aggregation operation is performed on the time dimension of each row to obtain a second aggregation feature; the second feature aggregation module is used to perform a channel-level feature splicing operation on the first aggregation feature and the second aggregation feature, and a feature channel adjustment is performed on the second splicing feature obtained by the splicing operation to obtain a second adjusted feature, and a feature channel adjustment and feature compression processing are performed on the first splicing feature obtained by the splicing operation to obtain a second compressed feature; the third feature aggregation module is used to perform a channel-level feature splicing operation on the second adjusted feature and the third-layer fusion feature, and a feature channel adjustment and feature compression processing are performed on the third splicing feature obtained by the splicing operation to obtain a third compressed feature; the first compressed feature, the second compressed feature and the third compressed feature are subjected to a channel-level feature splicing operation to obtain a first fusion feature.

[0016] According to one embodiment of the present invention, obtaining an augmented gait silhouette sequence includes: obtaining an original gait silhouette sequence; for each gait silhouette image in the original gait silhouette sequence, evenly dividing each image in the gait silhouette image in the horizontal direction to obtain r slice images, fixing the first slice image at the top, and shifting the remaining r-1 slice images to the right by l, 2×l, ..., (r-1)×l pixels in the vertical direction in sequence to generate a forward walking sequence; or, for each gait silhouette image in the original gait silhouette sequence, evenly dividing each image in the gait silhouette image in the horizontal direction to obtain r slice images, fixing the first slice image at the top, and shifting the remaining r-1 slice images to the left by l, 2×l, ..., (r-1)×l pixels in the vertical direction in sequence. 1)×l pixels to generate a backward walking sequence; or, for each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the vertical direction to obtain r slice images, the first slice image on the leftmost side is fixed, and the remaining r-1 slice images are successively moved downward in the horizontal direction by l, 2×l, ..., (r-1)×l pixels to generate an upstairs walking sequence; or, for each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the vertical direction to obtain r slice images, the first slice image on the top is fixed, and the remaining r-1 slice images are successively moved upward in the horizontal direction by l, 2×1, ..., (r-1)×l pixels to generate a downstairs walking sequence.

[0017] According to one embodiment of the present invention, the expression of the total loss function is:

[0018] L all =L triplet +L ce

[0019] Among them, L all Represents the total loss function, L triplet represents the triple loss function, L ce represents the cross entropy loss function;

[0020]

[0021] Among them, K represents the number of triples, and the triples are (F pos , F neg , F anc ), F pos represents the positive sample feature, F neg represents the negative sample feature, F anc represents the anchor feature, For feature F pos With feature F neg The Euclidean distance between For feature Fpos With feature F anc The Euclidean distance between them, 1≤k≤K, max(·) represents the maximum value function, and margin represents the boundary threshold;

[0022]

[0023] Where M represents the number of samples, 1≤i≤M, y i Indicates the label of the one-hot encoding type of the i-th sample, p i Represents the classification result of the i-th sample.

[0024] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present invention proposes a gait recognition method, which includes: obtaining an original gait silhouette sequence to be tested, amplifying the gait silhouette sequence to be tested, and obtaining an augmented gait silhouette sequence to be tested; inputting the original gait silhouette sequence to be tested and the augmented gait silhouette sequence to be tested into a pre-trained gait recognition model to obtain target gait features and their corresponding classification probabilities, wherein the pre-trained gait recognition model is obtained using the training method of the gait recognition model described in any one of claims 1-7.

[0025] To achieve the above-mentioned objectives, the third aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the training method of the gait recognition model proposed in the first aspect of the present invention, or the gait recognition method proposed in the second aspect of the present invention.

[0026] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a controller, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the training method of the gait recognition model proposed in the first aspect of the present invention, or the gait recognition method proposed in the second aspect of the present invention.

[0027] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a structural diagram of a gait recognition model according to an embodiment of the present invention;

[0029] Figure 2 is a flow chart of a method for training a gait recognition model according to an embodiment of the present invention;

[0030] Figure 3 is a schematic diagram of generating an augmented gait silhouette sequence according to an embodiment of the present invention;

[0031] Figure 4 is a structural diagram of a gait recognition model according to a specific embodiment of the present invention;

[0032] Figure 5 is a flowchart of generating target gait features and classification probabilities according to a specific embodiment of the present invention;

[0033] Figure 6 is a flow chart of obtaining original gait features according to a specific embodiment of the present invention;

[0034] Figure 7 is a flow chart of a dynamic change perception unit processing of a specific embodiment of the present invention;

[0035] Figure 8 is a flowchart of a multi-granularity feature extraction unit process according to a specific embodiment of the present invention;

[0036] Fig. 9 is a flow chart of obtaining augmented gait features according to a specific embodiment of the present invention;

[0037] Fig.10 is a flow chart of obtaining a first fusion feature according to a specific embodiment of the present invention;

[0038] Fig.11 is a flow chart of feature aggregation module processing according to one embodiment of the present invention;

[0039] Fig.12 is a flow chart of a gait recognition method according to an embodiment of the present invention;

[0040] Fig.13 4 is a structural block diagram of a controller according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0042] It should be noted that representation-based gait recognition methods can generally be divided into three categories. The first method regards the human body contour sequence as an unordered set of image frames, extracts frame-level features and set features respectively, and uses a horizontal pyramid mapping module to aggregate global and local features to enhance the representation ability of gait features; the second method focuses on the division of gait features on a spatial scale, and uses 3D convolution to obtain the overall and local spatiotemporal representation of human gait; the third method focuses on the temporal relationship modeling and motion feature capture of the gait sequence, and attempts to design multi-scale temporal feature extraction methods and motion feature capture methods.

[0043] The above methods often focus on the extraction of spatial features or temporal features, and lack effective means of extracting highly discriminative motion surface features, which makes it difficult for these methods to fully utilize the dynamic human motion information in gait sequences, and thus have poor adaptability to complex scenes.

[0044] The training method of the gait recognition model, the gait recognition method and medium, and the controller of the embodiment of the present invention are described in detail below in conjunction with the drawings in the specification and specific implementation methods.

[0045] like Figure 1 As shown, the gait recognition model in the embodiment of the present invention may include a first branch network, a second branch network, a multi-stage feature aggregation network, an element-level feature summation layer, a fully connected layer and a classifier, wherein the first branch network is used to input the original gait silhouette sequence, the second branch network is used to input the augmented gait silhouette sequence, the output end of the first branch network and the output end of the second branch network are connected to the input end of the multi-stage feature aggregation network, the output end of the first branch network and the output end of the multi-stage feature aggregation network are connected to the input end of the element-level feature summation layer, the output end of the element-level feature summation layer is connected to the input end of the classifier through the fully connected layer, and the classifier is used to output the target gait features and their corresponding classification probabilities.

[0046] In order to improve the adaptability of the gait recognition model to complex scenes, the gait recognition model in the embodiment of the present invention is provided with a first branch network for extracting original gait silhouette sequence features, a second branch network for extracting augmented gait silhouette sequence features, and a multi-stage feature aggregation network that fuses the first branch network for extracting original gait silhouette sequence features and the second branch network for extracting augmented gait silhouette sequence features, and identity recognition is performed based on the features output by the first branch network and the multi-stage feature aggregation network.

[0047] It should be noted that the augmented gait silhouette sequence is obtained based on the original gait silhouette sequence, and when the gait recognition model does not perform identity recognition, it can fully utilize the dynamic human motion information in the original gait silhouette sequence.

[0048] Figure 2FIG. 1 is a flow chart of a method for training a gait recognition model according to an embodiment of the present invention. Figure 2 As shown, the training method of the gait recognition model may include:

[0049] The gait recognition model is trained multiple times using the training image set, where for each training cycle, the following operations are performed:

[0050] S101, obtaining a training image set, wherein the training image set includes an original gait silhouette sequence and a corresponding augmented gait silhouette sequence;

[0051] S102, inputting the original gait silhouette sequence and the augmented gait silhouette sequence into a gait recognition model to obtain target gait features and their corresponding classification probabilities;

[0052] S103, obtaining a triplet loss function according to the target gait feature, obtaining a cross entropy loss function according to the classification probability corresponding to the target gait feature, and obtaining a total loss function according to the triplet loss function and the cross entropy loss function;

[0053] S104, adjusting the parameters of the gait recognition model according to the total loss function to obtain a trained gait recognition model.

[0054] Specifically, when acquiring a gait silhouette image sequence When capturing gait silhouette data of different persons, electronic devices such as cameras can be used, and then the collected gait silhouette data can be processed to obtain a gait silhouette image sequence. The gait silhouette image sequence can also be directly obtained from the CASIA-B dataset and the CCPG dataset. It should be noted that the CASIA-B dataset contains a gait silhouette image sequence of 124 pedestrians, and the CCPG dataset contains a gait silhouette image sequence of 200 pedestrians. The embodiment of the present invention does not limit the method for obtaining the gait silhouette image sequence.

[0055] In a specific embodiment of the present invention, the image size during gait recognition model training is set to H×W=64×44, that is, the image height is 64, the width is 44, the input sequence length T is 30 frames, and the input channel C=1, indicating that the image is a grayscale image.

[0056] The original gait silhouette sequence obtained Data augmentation is performed to segment and translate the silhouette images in the original gait silhouette sequence to generate an augmented gait silhouette sequence. The augmented gait silhouette sequence in the embodiment of the present invention at least includes a forward leaning walking sequence D lf , backward walking sequence D bb , Walking upstairs sequence D gu and downstairs walking sequence D gd Both of them.

[0057] The original gait silhouette sequence D and its corresponding augmented gait silhouette sequence in the training image set are input into the gait recognition model. The first branch network extracts features from the original gait silhouette sequence D, and the second branch network extracts features from the augmented gait silhouette sequence. The multi-stage feature aggregation network fuses the features extracted by the first branch network and the second network. The element-level feature summation layer performs element-level feature summation on the features extracted by the first branch network and the features extracted by the multi-stage feature aggregation network. The features obtained by summing the element-level features are sequentially input into the fully connected layer and the classifier, and the classifier outputs the target gait feature F. fc and its corresponding classification probability P.

[0058] For the target gait feature F fc Calculate the triple loss function L triplet , calculate the cross entropy loss function L for the classification probability P ce Therefore, the total loss function L of the gait recognition model is all It can be expressed as: all =L triplet +L ce The gait recognition model can be trained using the gradient descent method, and the total loss function L can be calculated. all To update the network parameters, when the number of training iterations reaches the set number or the total loss function L all When convergence occurs, the training stops, thereby obtaining a trained gait recognition model (optimal gait recognition model). The trained gait recognition model is used to perform identity recognition on the gait silhouette sequence to obtain an identity recognition result.

[0059] In one embodiment of the present invention, Figure 3 As shown, obtaining the augmented gait silhouette sequence includes:

[0060] Obtain the original gait silhouette sequence D;

[0061] For each gait silhouette image in the original gait silhouette sequence D, each image in the gait silhouette image is evenly divided in the horizontal direction to obtain r slices, the first slice at the top is fixed, and the remaining r-1 slices are shifted to the right in the vertical direction by 1, 2×1, ..., (r-1)×l pixels in sequence to generate the forward walking sequence D lf ;or,

[0062] For each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the horizontal direction to obtain r slices, the first slice at the top is fixed, and the remaining r-1 slices are shifted leftward in the vertical direction by 1, 2×1, ..., (r-1)×l pixels in sequence to generate the backward walking sequence D bb ;or,

[0063] For each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the vertical direction to obtain r slice images, the first slice image on the left is fixed, and the remaining r-1 slice images are shifted downward in the horizontal direction by 1, 2×1, ..., (r-1)×l pixels in sequence to generate the upstairs walking sequence D gu ;or,

[0064] For each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly segmented in the vertical direction to obtain r slices. The first slice at the top is fixed, and the remaining r-1 slices are moved upward in the horizontal direction by 1, 2×1, ..., (r-1)×l pixels in sequence to generate the downstairs walking sequence D. gd .

[0065] The embodiment of the present invention performs an operation on the original gait silhouette sequence Data augmentation is performed. First, each gait silhouette image in the original gait silhouette sequence D is segmented and translated.

[0066] Specifically, for each gait silhouette image, the image is evenly divided into r parts in the horizontal dimension of the gait silhouette image (the height dimension of the gait silhouette image), the position of the top local image block is fixed, and the remaining r-1 parts are respectively shifted to the right by 1, 2×l, ..., (r-1)×l pixels in the horizontal direction to generate a gait image similar to forward walking, see Figure 3 . Move the remaining r-1 parts horizontally to the left by 1, 2×1, …, (r-1)×l pixels, respectively, to generate a gait image similar to backward walking, see Figure 3 .

[0067] For each gait silhouette image, the image is evenly divided into r parts in the vertical dimension of the gait silhouette image (the width dimension of the gait silhouette image), the position of the leftmost local image block is fixed, and the remaining r-1 parts are moved downward in the height direction by 1, 2×1, ..., (r-1)×l pixels respectively to generate a gait image similar to walking upstairs, see Figure 3 . Move the remaining r-1 parts upward by 1, 2×1, ..., (r-1)×l pixels in the height direction to generate a gait image similar to walking downstairs, see Figure 3 .

[0068] By cutting and filling the images of each gait silhouette image in the original gait silhouette sequence D after segmentation and translation operations, four types of gait sequences can be generated: forward walking sequence D lf , backward walking sequence D bb , Walking upstairs sequence D guand downstairs walking sequence D gd .

[0069] The embodiment of the present invention adopts a motion pattern augmentation strategy to perform data augmentation on the original gait silhouette sequence D, generating four types of new walking data, providing diverse motion appearance information and pedestrian identity clues for the gait recognition network. In addition, the gait features of the augmented data are combined to enhance the discrimination ability of the gait features.

[0070] In one embodiment of the present invention, Figure 4 and Figure 5 As shown, the original gait silhouette sequence and the augmented gait silhouette sequence are input into the gait recognition model to obtain the target gait features and their corresponding classification probabilities, which may include:

[0071] S201, inputting the original gait silhouette sequence into the first branch network to obtain the first layer fusion feature, the first aggregation feature, the third layer fusion feature and the original gait feature;

[0072] S202, inputting the augmented gait silhouette sequence into the second branch network to obtain augmented gait features;

[0073] S203, inputting the first layer fusion feature, the first aggregation feature, the third layer fusion feature and the augmented gait feature into a multi-stage feature aggregation network to obtain a first fusion feature;

[0074] S204, inputting the original gait feature and the first fused feature into an element-level feature summation layer to obtain a second fused feature;

[0075] S205, inputting the second fusion feature into the fully connected layer and the classifier in sequence to obtain the target gait feature and its corresponding classification probability.

[0076] Specifically, the first branch network extracts features from the original gait silhouette sequence D. The first branch network (spatial-temporal feature extraction network) of the embodiment of the present invention may include a first feature extraction module, a second feature extraction module, a first time aggregation module, a third feature extraction module, a time pooling module, and a spatial pooling module connected in sequence. The first feature extraction module outputs the first layer of fusion features The first time aggregation module outputs the first aggregation feature The third feature extraction module outputs the third layer of fusion features The spatial pooling module outputs the original gait feature F sp .

[0077] The second branch network is used to generate the augmented gait silhouette sequence (forward walking sequence D) corresponding to the original gait silhouette sequence D lf , backward walking sequence D bb , Walking upstairs sequence D gu and downstairs walking sequence Dgd ) to extract features and obtain the augmented gait feature F da .

[0078] Multi-stage feature aggregation network for the first layer fusion feature First Aggregate Feature The third layer fusion features and augmented gait features F da Fusion is performed to obtain the first fusion feature F ms .

[0079] The first fusion feature F ms Compared with the original gait feature F sp Through the element-level feature summation operation, the final fusion feature (second fusion feature) is obtained. The final fusion feature F fusion Through the fully connected layer and the classifier in turn, the target gait features F are output respectively fc And classification probability P.

[0080] In a specific embodiment of the present invention, the classifier may be composed of a fully connected layer and a batch normalization layer.

[0081] In one embodiment of the present invention, Figure 4 and Figure 6 As shown, the first branch network includes a first feature extraction module, a second feature extraction module, a first time aggregation module, a third feature extraction module, a time pooling module and a space pooling module connected in sequence, and the original gait silhouette sequence is input into the first branch network to obtain the first layer fusion feature, the first aggregation feature, the third layer fusion feature and the original gait feature, which may include:

[0082] S301, using a first feature extraction module, respectively performing dynamic change feature extraction and multi-granularity feature extraction on the original gait silhouette sequence, and performing an element-level feature summation operation on the extracted first dynamic change feature and the first multi-granularity feature to obtain a first layer of fusion features;

[0083] S302, using the second feature extraction module, respectively performing dynamic change feature extraction and multi-granularity feature extraction on the first layer fusion feature, and performing element-level feature summation operation on the extracted second dynamic change feature and the second multi-granularity feature to obtain a second layer fusion feature;

[0084] S303, using a first time aggregation module, performing a compression aggregation operation on the second layer fusion features in the time dimension to obtain a first aggregation feature;

[0085] S304, using a third feature extraction module, respectively performing dynamic change feature extraction and multi-granularity feature extraction on the first aggregated feature, and performing an element-level feature summation operation on the extracted third dynamic change feature and the third multi-granularity feature to obtain a third layer of fusion features;

[0086] S305, using a time pooling module to perform a time dimension feature pooling operation on the third layer fusion features to obtain pooling features;

[0087] S306, using a spatial pooling module to perform spatial dimension feature compression on the pooled features to obtain original gait features.

[0088] In this embodiment, the first feature extraction module may include a first-layer dynamic change perception unit, a first-layer multi-granularity feature extraction unit, and a first-layer element-level feature summation unit. The first-layer dynamic change perception unit and the first-layer multi-granularity feature extraction unit extract dynamic change features from the original gait silhouette sequence D respectively. and multi-granularity features The first dynamic change feature and the first multi-granularity feature After element-level feature summation, the first layer of fusion features is generated

[0089] In this embodiment, the second feature extraction module may include a second-layer dynamic change perception unit, a second-layer multi-granularity feature extraction unit, and a second-layer element-level feature summation unit. The second-layer dynamic change perception unit and the second-layer multi-granularity feature extraction unit perform fusion feature extraction. Extract dynamic change features separately and multi-granularity features The second dynamic change feature and the second multi-granularity feature After element-level feature summation, the second layer of fusion features is generated

[0090] In this embodiment, the first time aggregation module aggregates the second layer fusion features Perform compression aggregation operation on the time dimension to generate the first aggregation feature

[0091] In a specific embodiment of the present invention, the temporal aggregation module is composed of a 3D convolution with a convolution kernel of (3, 3, 3), a step size of (3, 1, 1), and an activation function LeakyReLU.

[0092] In this embodiment, the third feature extraction module may include a third-layer dynamic change perception unit, a third-layer multi-granularity feature extraction unit, and a third-layer element-level feature summation unit. The third-layer dynamic change perception unit and the third-layer multi-granularity feature extraction unit extract the first aggregate feature. Extract dynamic change features separately and multi-granularity features The third dynamic change feature and the third multi-granularity feature After element-level feature summation, the third layer of fusion features is generated

[0093] In this embodiment, the time pooling module performs the third layer fusion feature Perform feature pooling operations in the time dimension to generate pooling features The spatial pooling module pools the feature F tp Perform feature compression operations in the spatial dimension to generate original gait features

[0094] In a specific embodiment of the present invention, the temporal pooling operation may refer to a 3D maximum pooling layer. The spatial pooling operation may be expressed as: in, represents a 3D average pooling layer with a pooling kernel of (1, 1, W), and p is a learnable network parameter.

[0095] In a specific embodiment of the present invention, Figure 7 As shown in the figure, the dynamic change perception unit consists of a global feature extraction branch and a dynamic difference feature extraction branch.

[0096] In this embodiment, the global feature extraction branch is composed of a local feature extractor consisting of a 3D convolution with a convolution kernel of (3, 3, 3) and an activation function LeakyReLU. This branch extracts the input features of the dynamic change perception unit. Perform feature extraction to obtain features

[0097] In this embodiment, the dynamic difference feature extraction branch is also composed of a 3D convolution with a convolution kernel of (3, 3, 3) and an activation function LeakyReLU. This branch first performs an operation on Ti n The input features of the frame are used to calculate the difference between the features of adjacent frames, and the length is T. in -1 frame difference sequence. For the convenience of subsequent calculation, we copy the Tth in -1 frame difference feature, we can get a length of T in The difference feature sequence of the frame Then use 3D convolution and activation function to extract features and get the features

[0098] The dynamic change perception unit detects the feature F global and feature F difBy performing element-level feature summation, we can obtain the output feature F of this module. dvp .

[0099] In a specific embodiment of the present invention, Figure 8 As shown in Figure 1, the multi-granularity feature extraction unit is composed of multiple local feature extractors whose parameters are not shared. The multi-granularity feature extraction unit first converts the input feature It is divided into multiple local features along the height dimension. For different layers of multi-granularity feature extraction units, the number of local feature divisions is set as follows: the first layer of multi-granularity feature extraction units contains 3 local features, the second layer of multi-granularity feature extraction units contains 5 local features, and the third layer of multi-granularity feature extraction units contains 9 local features.

[0100] In the first layer of multi-granularity feature extraction unit, there are three groups of local feature extractors consisting of 3D convolution kernel (3, 3, 3) and activation function LeakyReLU. Each of them consists of input feature After segmentation along the height dimension, three local features can be obtained: and Then, these local features are passed through the local feature extractor to obtain the corresponding convolution local features. and Then, and After the high-dimensional feature concatenation operation again, the output features of the first-layer multi-granularity feature extraction unit are obtained. In the second-layer multi-granularity feature extraction unit and the third-layer multi-granularity feature extraction unit, 5 groups and 9 groups of local feature extractors are respectively included for extracting local features.

[0101] As a specific example, in a three-layer multi-granularity feature extraction unit, the first layer extracts the input features according to the height. and Divide into 3 parts of local features; the second layer divides the input features into three parts according to the height and Divide into 5 parts of local features; the third layer divides the input features into high and Divided into 9 parts of local features.

[0102] Each layer of multi-granularity feature extraction unit first divides the input feature into multiple local features along the height dimension, then uses the local feature extractor to perform convolution and nonlinear activation operations on each local feature, and then performs feature splicing operations on the output features of each local feature extractor along the height dimension to obtain the output features of the multi-granularity feature extraction unit.

[0103] In the first branch network of the embodiment of the present invention, a dynamic change perception unit is used to calculate the dynamic changes between adjacent frames in the gait sequence, and the motion features of the dynamic area are extracted from the dynamic change sequence; a multi-granularity feature extraction unit is used to divide the input features into multiple local blocks along the height dimension in different network layers based on prior knowledge of the human body, and extract local spatiotemporal features. As the number of network layers increases, the number of local feature divisions in the multi-granularity feature extraction unit gradually increases, thereby obtaining local gait features containing coarse-grained and fine-grained features. The dynamic change perception unit and the multi-granularity feature extraction unit can provide key dynamic difference change features and multi-granularity local gait representations, which can effectively improve the representation ability of the model.

[0104] In one embodiment of the present invention, Figure 4 and Fig. 9 As shown, the augmented gait silhouette sequence includes at least two of a forward leaning walking sequence, a backward leaning walking sequence, an upstairs walking sequence, and a downstairs walking sequence. The second branch network includes a shallow feature extraction module and a channel-level feature aggregation module. The augmented gait silhouette sequence is input into the second branch network to obtain augmented gait features, which may include:

[0105] S401, using a shallow feature extraction module to extract features from at least two of the forward leaning walking sequence, the backward leaning walking sequence, the upstairs walking sequence, and the downstairs walking sequence, to obtain at least two of the first gait feature, the second gait feature, the third gait feature, and the fourth gait feature;

[0106] S402: Use a channel-level feature aggregation module to perform feature aggregation on at least two of the first gait feature, the second gait feature, the third gait feature, and the fourth gait feature to obtain an augmented gait feature.

[0107] The second branch network of the embodiment of the present invention may include a shallow feature extraction module and a channel-level feature aggregation module.

[0108] Specifically, a shallow feature extraction module is constructed for the four augmented gait sequences: forward walking sequence D lf , backward walking sequence D bb , Walking upstairs sequence D gu and downstairs walking sequence D gd , extract their respective gait features and get the first gait feature Second gait characteristics The third gait characteristic and the fourth gait characteristic Design channel-level feature aggregation operations, and perform the first step of the state feature F on the four types of features. lf , the second gait feature F bb , the third gait feature Fgu and the fourth gait feature F gd Perform feature aggregation and obtain augmented gait features

[0109] In a specific embodiment of the present invention, the shallow feature extractor is composed of a 3D convolution with a convolution kernel of (3, 3, 3) and an activation function LeakyReLU.

[0110] In one embodiment of the present invention, Figure 4 and Fig.10 As shown, the multi-stage feature aggregation network includes a first feature aggregation module, a second time aggregation module, a second feature aggregation module and a third feature aggregation module, and the first layer fusion feature, the first aggregation feature, the third layer fusion feature and the augmented gait feature are input into the multi-stage feature aggregation network to obtain the first fusion feature, which may include:

[0111] S501, using a first feature aggregation module, performing a channel-level feature splicing operation on the augmented gait feature and the first-layer fusion feature, performing feature channel adjustment on the first splicing feature obtained by the splicing operation to obtain a first adjusted feature, and performing feature channel adjustment and feature compression processing on the first splicing feature obtained by the splicing operation to obtain a first compressed feature;

[0112] S502, using a second time aggregation module to perform a compression aggregation operation on the first adjustment feature in the time dimension to obtain a second aggregation feature;

[0113] S503, using a second feature aggregation module, performing a channel-level feature splicing operation on the first aggregated feature and the second aggregated feature, performing feature channel adjustment on the second spliced ​​feature obtained by the splicing operation to obtain a second adjusted feature, and performing feature channel adjustment and feature compression processing on the first spliced ​​feature obtained by the splicing operation to obtain a second compressed feature;

[0114] S504, using a third feature aggregation module, performing a channel-level feature splicing operation on the second adjustment feature and the third-layer fusion feature, and performing feature channel adjustment and feature compression processing on the third splicing feature obtained by the splicing operation to obtain a third compressed feature;

[0115] S505, performing a channel-level feature concatenation operation on the first compression feature, the second compression feature, and the third compression feature to obtain a first fusion feature.

[0116] In the embodiment of the present invention, a multi-stage feature aggregation network is used to augment the gait feature F da , the first layer fusion features First Aggregate Feature And the third layer fusion features To merge.

[0117] The multi-stage feature aggregation network in the embodiment of the present invention may include a first feature aggregation module, a second time aggregation module, a second feature aggregation module and a third feature aggregation module. It should be noted that the first time aggregation module and the second time aggregation module in the embodiment of the present invention have the same structure and function.

[0118] In this embodiment, the first feature aggregation module includes a channel-level feature concatenation operation, a first-layer feature compression unit, and a first-layer channel mapping unit. Fig.11 As shown, the augmented gait feature F da (Input feature 1) and the first layer fusion feature (Input feature 2) First, the channel-level feature concatenation operation is performed to obtain the first concatenated feature Then, the first splicing feature The first layer of feature compression units is used to adjust feature channels and compress features. The first layer of feature compression units includes a 3D convolution with a convolution kernel of (3, 3, 3), an activation function LeakyReLU, a temporal pooling layer, and a spatial pooling layer. Through the 3D convolution with the convolution kernel (3, 3, 3) and the LeakyReLU activation function, we get Then, Features can be obtained through the time pooling layer The first compression feature can be obtained through the spatial pooling layer

[0119] In addition, the first splicing feature The feature channel is adjusted through the first layer channel mapping unit. The channel mapping module includes a 3D convolution with a convolution kernel of (3, 3, 3) and LeakyReLU to obtain the first adjusted feature

[0120] First adjustment feature The second aggregation feature is obtained by the second time aggregation module

[0121]

[0122] In this embodiment, the second feature aggregation module includes a channel-level feature concatenation operation, a second-layer feature compression unit, and a second-layer channel mapping unit. Fig.11 As shown, the second aggregation feature (Input feature 1) and the first aggregate feature (Input feature 2) First, the channel-level feature concatenation operation is performed to obtain the second concatenated feature The structure and function of the second layer feature compression unit and the first layer feature compression unit are the same as those of the second layer feature compression unit. The second layer feature compression unit is used to adjust the feature channel and compress the feature to obtain the second compressed feature. Second splicing feature The feature channel is adjusted through the second layer channel mapping unit to obtain the second adjusted feature

[0123] In this embodiment, the third feature aggregation module includes a channel-level feature concatenation operation and a third-layer feature compression unit. (Input feature 1) and the third layer fusion feature (Input feature 2) Similarly, first perform channel-level feature concatenation to obtain the third concatenated feature The third splicing feature Only feature channel adjustment and feature compression are performed through the third-layer feature compression unit to obtain the third compressed feature

[0124] For the three compressed features generated in the multi-stage feature aggregation network: the first compressed feature Second splicing feature and the third compression feature Through the channel-level feature concatenation operation, the first fusion feature is obtained Among them, by setting 3C fccm =C s3 , we can get the first fusion feature F with the same feature dimension ms Compared with the original gait feature F sp .

[0125] In a specific embodiment of the present invention, the feature compression unit in the feature aggregation module includes a 3D convolution with a convolution kernel of (3, 3, 3), an activation function LeakyReLU, a temporal pooling layer, and a spatial pooling layer. The channel mapping unit is composed of a 3D convolution with a convolution kernel of (3, 3, 3) and LeakyReLU.

[0126] The embodiment of the present invention realizes multi-stage feature fusion of original gait features and augmented gait features of the input original gait silhouette sequence by constructing a multi-stage feature aggregation network. The fused features are sent to the subsequent multi-stage feature aggregation network through the channel mapping module, and the compressed features of the feature aggregation modules of each stage are output through the feature compression module. The gait features of multiple stages are used to improve the accuracy of model identity recognition.

[0127] In one embodiment of the present invention, the expression of the total loss function is:

[0128] L all =L triplet +L ce

[0129] Among them, L all Represents the total loss function, L triplet represents the triple loss function, L ce represents the cross entropy loss function;

[0130]

[0131] Among them, K represents the number of triples, and the triples are (F pos , F neg , F anc ), F pos represents the positive sample feature, F neg represents the negative sample feature, F anc represents the anchor feature, For feature F pos With feature F neg The Euclidean distance between For feature F pos With feature F anc The Euclidean distance between them, 1≤k≤K, max(·) represents the maximum value function, and margin represents the boundary threshold;

[0132]

[0133] Where M represents the number of samples, 1≤i≤M, y i Indicates the label of the one-hot encoding type of the i-th sample, p i Represents the classification result of the i-th sample.

[0134] In an embodiment of the present invention, in the triple loss function L triplet In the example, given a feature triple (F pos , F neg , F anc ), where F pos represents the positive sample feature, F neg represents the negative sample feature, F anc Represents the anchor feature. Given K sets of feature triplets, the triplet loss can be expressed as:

[0135]

[0136] Specifically, the gait recognition model based on motion pattern augmentation is trained using the gradient descent method, and the total loss function L is calculated. all To update the model parameters, when the number of training iterations reaches the set number or the total loss function L all When convergence occurs, the training stops, and a trained gait recognition model is obtained. The model is used to perform identity recognition on the gait silhouette sequence to obtain the identity recognition result.

[0137] It is feasible to use the Adam optimizer for a total of 100,000 iterations of training, with the initial learning rate set to 0.0001. When the training iteration is 80,000, the learning rate is reduced to 0.00001.

[0138] The augmented gait silhouette sequence in the embodiment of the present invention provides a variety of motion appearance information and pedestrian identity clues for the gait recognition model. The gait recognition model extracts features based on the original gait silhouette sequence and its corresponding augmented gait silhouette sequence. The original gait features are combined with the gait features of the augmented gait silhouette sequence (augmented gait features) to enhance the gait feature discrimination ability of the gait recognition model.

[0139] The first branch network in the embodiment of the present invention uses a dynamic change perception unit to calculate the dynamic changes between adjacent frames in a gait sequence, and extracts motion features of dynamic areas from the dynamic change sequence; using a multi-granularity feature extraction unit, in different network layers, based on prior knowledge of the human body, the input features are divided into multiple local blocks along the height dimension, and each local spatiotemporal feature is extracted. As the number of network layers increases, the number of local feature divisions in the multi-granularity feature extraction unit gradually increases, thereby obtaining local gait features containing coarse-grained and fine-grained features. The dynamic change perception unit and the multi-granularity feature extraction unit can provide key dynamic difference change features and multi-granularity local gait representations, which can effectively improve the representation ability of the model.

[0140] The multi-stage feature aggregation network in the embodiment of the present invention realizes the multi-stage feature fusion of the original gait features and the augmented gait features of the input original gait silhouette sequence. The fused features are sent to the subsequent multi-stage feature aggregation network through the channel mapping module, and the compressed features of the feature aggregation modules of each stage are output through the feature compression module. The gait features of multiple stages are used to improve the accuracy of model identity recognition.

[0141] The training method of the gait recognition model of the embodiment of the present invention amplifies the original gait silhouette sequence and adds an augmented gait silhouette sequence to provide the gait recognition model with diverse motion appearance information and pedestrian identity clues, and improves the identity recognition accuracy of the gait recognition model by extracting and fusing the original gait silhouette sequence and the augmented gait features.

[0142] The invention provides a gait recognition method.

[0143] Fig.12 FIG. 1 is a flow chart of a gait recognition method according to an embodiment of the present invention. Fig.12 As shown, the gait recognition method may include:

[0144] S601, obtaining an original gait silhouette sequence to be tested, and amplifying the gait silhouette sequence to be tested to obtain an augmented gait silhouette sequence to be tested;

[0145] S602, inputting the original gait silhouette sequence to be tested and the augmented gait silhouette sequence to be tested into a pre-trained gait recognition model to obtain target gait features and their corresponding classification probabilities, wherein the pre-trained gait recognition model is obtained using the training method of the gait recognition model as described above.

[0146] The gait recognition model in the embodiment of the present invention may include a first branch network, a second branch network, a multi-stage feature aggregation network, an element-level feature summation layer, a fully connected layer and a classifier, wherein the first branch network is used to input the original gait silhouette sequence, the second branch network is used to input the augmented gait silhouette sequence, the output end of the first branch network and the output end of the second branch network are connected to the input end of the multi-stage feature aggregation network, the output end of the first branch network and the output end of the multi-stage feature aggregation network are connected to the input end of the element-level feature summation layer, the output end of the element-level feature summation layer is connected to the input end of the classifier through the fully connected layer, and the classifier is used to output the target gait features and their corresponding classification probabilities.

[0147] The embodiment of the present invention is based on a gait recognition method based on motion pattern augmentation, which performs data augmentation on the input original gait silhouette sequence to be tested, and generates gait sequences under four different walking modes through the motion pattern augmentation strategy. The input original gait silhouette sequence to be tested is passed through the first branch network to obtain the original gait features; the four types of gait sequences generated by data augmentation are passed through the second branch network to obtain augmented gait features. The original gait features and the augmented gait features are passed through a multi-stage feature aggregation network to obtain a fused gait feature. The original gait features and the fused gait are subjected to a feature fusion operation to obtain the final gait features.

[0148] The gait recognition model trained in the embodiment of the present invention mines effective motion appearance features and discriminative identity features from diverse motion expressions, and combines the first branch network, the second branch network and the multi-stage feature aggregation network to improve the accuracy of gait recognition.

[0149] The gait recognition method of the embodiment of the present invention mines effective motion appearance features and discriminative identity features from diverse motion expressions, and uses a trained gait recognition model to recognize the original gait silhouette sequence to be tested and the augmented gait silhouette sequence to be tested, thereby improving the accuracy of gait recognition.

[0150] The present invention provides a computer-readable storage medium.

[0151] In one embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the training method of the gait recognition model as described above is implemented.

[0152] In another embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the gait recognition method as described above is implemented.

[0153] The invention provides a controller.

[0154] In one embodiment, the controller includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the training method of the gait recognition model as described above is implemented.

[0155] In one embodiment, the controller includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the gait recognition method as described above is implemented.

[0156] Fig.13 4 is a structural block diagram of a controller according to an embodiment of the present invention.

[0157] like Fig.13 As shown, the controller 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the controller 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the controller 500 does not constitute a limitation on the embodiments of the present invention.

[0158] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0159] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0160] The memory 503 is used to store a computer program corresponding to the training method of the gait recognition model or the gait recognition method of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the content shown in the above method embodiment.

[0161] Among them, the controller 500 includes but is not limited to: mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The controller 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0162] The computer-readable storage medium and controller of the embodiment of the present invention utilize the above gait recognition model training method to train the above gait recognition model, thereby improving the identity recognition accuracy of the trained gait recognition model.

[0163] The computer-readable storage medium and controller of the embodiments of the present invention utilize the above gait recognition method to mine effective motion appearance features and discriminative identity features from a variety of motion expressions, and utilize a trained gait recognition model to identify the original gait silhouette sequence to be tested and the augmented gait silhouette sequence to be tested, thereby improving the accuracy of gait recognition.

[0164] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0165] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0166] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0167] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0168] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0169] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0170] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0171] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A training method for a gait recognition model, characterized in that: The gait recognition model includes a first branch network, a second branch network, a multi-stage feature aggregation network, an element-level feature summation layer, a fully connected layer and a classifier, wherein the first branch network is used to input an original gait silhouette sequence, the second branch network is used to input an augmented gait silhouette sequence, the output end of the first branch network and the output end of the second branch network are connected to the input end of the multi-stage feature aggregation network, the output end of the first branch network and the output end of the multi-stage feature aggregation network are connected to the input end of the element-level feature summation layer, the output end of the element-level feature summation layer is connected to the input end of the classifier through the fully connected layer, and the classifier is used to output the target gait feature and its corresponding classification probability, and the training method includes: The gait recognition model is trained multiple times using the training image set, where for each training cycle, the following operations are performed: Acquire a training image set, wherein the training image set includes an original gait silhouette sequence and a corresponding augmented gait silhouette sequence; Inputting the original gait silhouette sequence and the augmented gait silhouette sequence into a gait recognition model to obtain target gait features and their corresponding classification probabilities; Obtain a triplet loss function according to the target gait feature, obtain a cross entropy loss function according to the classification probability corresponding to the target gait feature, and obtain a total loss function according to the triplet loss function and the cross entropy loss function; According to the total loss function, the parameters of the gait recognition model are adjusted to obtain a trained gait recognition model.

2. The training method of the gait recognition model according to claim 1, characterized in that: The step of inputting the original gait silhouette sequence and the augmented gait silhouette sequence into a gait recognition model to obtain target gait features and their corresponding classification probabilities includes: Inputting the original gait silhouette sequence into the first branch network to obtain the first layer fusion feature, the first aggregation feature, the third layer fusion feature and the original gait feature; Inputting the augmented gait silhouette sequence into the second branch network to obtain augmented gait features; Inputting the first-layer fusion features, the first aggregated features, the third-layer fusion features, and the augmented gait features into the multi-stage feature aggregation network to obtain a first fusion feature; Inputting the original gait feature and the first fused feature into the element-level feature summation layer to obtain a second fused feature; The second fusion feature is sequentially input into the fully connected layer and the classifier to obtain the target gait feature and its corresponding classification probability.

3. The training method of the gait recognition model according to claim 2, characterized in that: The first branch network includes a first feature extraction module, a second feature extraction module, a first time aggregation module, a third feature extraction module, a time pooling module and a space pooling module connected in sequence, and the original gait silhouette sequence is input into the first branch network to obtain a first layer of fusion features, a first aggregation feature, a third layer of fusion features and original gait features, including: Using the first feature extraction module, respectively performing dynamic change feature extraction and multi-granularity feature extraction on the original gait silhouette sequence, and performing element-level feature summation operation on the extracted first dynamic change feature and first multi-granularity feature to obtain the first layer of fusion features; Using the second feature extraction module, respectively performing dynamic change feature extraction and multi-granularity feature extraction on the first layer fusion features, and performing element-level feature summation operation on the extracted second dynamic change features and second multi-granularity features to obtain a second layer fusion feature; Using the first time aggregation module, performing a compression aggregation operation on the second layer fusion features in the time dimension to obtain the first aggregation features; Using the third feature extraction module, respectively performing dynamic change feature extraction and multi-granularity feature extraction on the first aggregated features, and performing element-level feature summation operation on the extracted third dynamic change features and third multi-granularity features to obtain the third layer fusion features; Using the time pooling module to perform a feature pooling operation in the time dimension on the third layer fusion features to obtain pooling features; The spatial pooling module is used to perform feature compression of the spatial dimension on the pooled features to obtain the original gait features.

4. The training method of the gait recognition model according to claim 2, characterized in that: The augmented gait silhouette sequence includes at least two of a forward leaning walking sequence, a backward leaning walking sequence, an upstairs walking sequence, and a downstairs walking sequence. The second branch network includes a shallow feature extraction module and a channel-level feature aggregation module. The augmented gait silhouette sequence is input into the second branch network to obtain an augmented gait feature, including: Using the shallow feature extraction module, feature extraction is performed on at least two of the forward leaning walking sequence, the backward leaning walking sequence, the upstairs walking sequence, and the downstairs walking sequence to obtain at least two of the first gait feature, the second gait feature, the third gait feature, and the fourth gait feature; The channel-level feature aggregation module is used to perform feature aggregation on at least two of the first gait feature, the second gait feature, the third gait feature, and the fourth gait feature to obtain the augmented gait feature.

5. The training method of the gait recognition model according to claim 2, characterized in that: The multi-stage feature aggregation network includes a first feature aggregation module, a second time aggregation module, a second feature aggregation module and a third feature aggregation module. The first layer fusion feature, the first aggregation feature, the third layer fusion feature and the augmented gait feature are input into the multi-stage feature aggregation network to obtain a first fusion feature, including: Using the first feature aggregation module, performing a channel-level feature splicing operation on the augmented gait feature and the first-layer fusion feature, performing feature channel adjustment on the first splicing feature obtained by the splicing operation to obtain a first adjusted feature, and performing feature channel adjustment and feature compression processing on the first splicing feature obtained by the splicing operation to obtain a first compressed feature; Using the second time aggregation module, performing a compression aggregation operation in the time dimension on the first adjustment feature to obtain a second aggregation feature; Using the second feature aggregation module, performing a channel-level feature splicing operation on the first aggregated feature and the second aggregated feature, performing feature channel adjustment on the second spliced ​​feature obtained by the splicing operation to obtain a second adjusted feature, and performing feature channel adjustment and feature compression processing on the first spliced ​​feature obtained by the splicing operation to obtain a second compressed feature; Using the third feature aggregation module, perform a channel-level feature splicing operation on the second adjustment feature and the third-layer fusion feature, and perform feature channel adjustment and feature compression processing on the third splicing feature obtained by the splicing operation to obtain a third compressed feature; A channel-level feature concatenation operation is performed on the first compression feature, the second compression feature, and the third compression feature to obtain a first fusion feature.

6. The training method of the gait recognition model according to claim 1, characterized in that: Obtaining an augmented gait silhouette sequence includes: Obtaining the original gait silhouette sequence; For each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the horizontal direction to obtain r slice images, the first slice image at the top is fixed, and the remaining r-1 slice images are shifted to the right in the vertical direction by l, 2×l, ..., (r-1)×l pixels in sequence to generate a forward walking sequence; or, For each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the horizontal direction to obtain r slice images, the first slice image at the top is fixed, and the remaining r-1 slice images are shifted leftward in the vertical direction by l, 2×l, ..., (r-1)×l pixels in sequence to generate a backward walking sequence; or, For each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the vertical direction to obtain r slice images, the first slice image on the left is fixed, and the remaining r-1 slice images are shifted downward in the horizontal direction by l, 2×l, ..., (r-1)×l pixels in sequence to generate a stair-walking sequence; or, For each gait silhouette image in the original gait silhouette sequence, each image in the gait silhouette image is evenly divided in the vertical direction to obtain r slice images, the first slice image at the top is fixed, and the remaining r-1 slice images are shifted upward in the horizontal direction by l, 2×l, ..., (r-1)×l pixels in sequence to generate a downstairs walking sequence.

7. The training method of the gait recognition model according to claim 1, characterized in that: The expression of the total loss function is: L all =L triplet +L ce Among them, L all Represents the total loss function, L triplet represents the triple loss function, L ce represents the cross entropy loss function; Among them, K represents the number of triples, and the triples are (F pos ,F neg ,F anc ), F pos represents the positive sample feature, F neg represents the negative sample feature, F anc represents the anchor feature, For feature F pos With feature F neg The Euclidean distance between For feature F pos With feature F anc The Euclidean distance between them, 1≤k≤K, max(·) represents the maximum value function, and margin represents the boundary threshold; Where M represents the number of samples, 1≤i≤M, y i Indicates the label of the one-hot encoding type of the i-th sample, p i Represents the classification result of the i-th sample.

8. A gait recognition method, characterized in that: The identification method comprises: Acquire an original gait silhouette sequence to be tested, and amplify the gait silhouette sequence to be tested to obtain an augmented gait silhouette sequence to be tested; The original gait silhouette sequence to be tested and the augmented gait silhouette sequence to be tested are input into a pre-trained gait recognition model to obtain target gait features and their corresponding classification probabilities, wherein the pre-trained gait recognition model is obtained using the training method of the gait recognition model described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the gait recognition model as described in any one of claims 1 to 7, or the gait recognition method as described in claim 8.

10. A controller, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by the processor, the training method of the gait recognition model as described in any one of claims 1 to 7 is implemented, or the gait recognition method as described in claim 8.

Images