A cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction

Abstract

The present invention relates to a cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction, including: backbone network construction: extracting shallow, middle and deep features of basic gait; motion feature mixing module construction: fusing gait information between adjacent frames and extracting timing information; fine-grained multi-stage feature extraction module construction: horizontally dividing features to extract preliminary spatial features. A branch is introduced for further horizontal division, adding a spatial attention mechanism to extract more fine-grained features; overall framework training; cross-view gait recognition. The present invention can effectively utilize frame-level time information in a gait profile sequence, as well as fine-grained spatial information, and is more effective in feature extraction.

Description

Technical Field

[0001] The present invention relates to a cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction, and belongs to the technical field of deep learning and pattern recognition. Background Art

[0002] Gait refers to the changes in posture and behavioral characteristics of a person when walking. Gait recognition refers to determining the identity of a pedestrian by his or her gait. Most of the current gait recognition technologies use the gait characteristics of the human body collected by long-distance sensors to identify pedestrians. Compared with the more mature biometric recognition technologies, such as face, voiceprint, fingerprint, DNA and other biometric recognition technologies, gait recognition technology has the characteristics of being uncontrolled and highly concealed. Based on the above advantages, gait recognition technology can be widely used in various security fields. However, gait recognition technology is affected by many covariate factors in practical applications, such as clothing, carrying objects and changes in perspective. Among them, changes in perspective can cause a huge change in the appearance of gait, so it has become the most important factor affecting gait recognition performance. Therefore, effectively solving the problem of cross-perspective gait recognition is the key to further developing gait recognition technology.

[0003] Current gait recognition technologies can be divided into two categories, namely, model-based methods and appearance-based methods. Model-based cross-view gait recognition methods model human gait sequences in different ways, extract human structure, motion parameters or other information from the model as features for cross-view gait recognition. This type of method consumes a lot of computing resources when modeling the human body, or requires high-resolution images when extracting key points of the human skeleton. The input data of the appearance-based method is generally the gait contour, which is a binary image obtained after segmentation, cropping, binarization and other processing of the original gait RGB image. The appearance-based method can be further refined into a method based on energy-like graphs and a method based on gait contour sequences. The method based on energy-like graphs superimposes the contour sequence according to certain rules, thereby aggregating the spatiotemporal information of the gait into one image. Although this method saves computing resources, it ignores the extraction of temporal information. The method based on gait contour sequence directly uses the entire gait contour sequence as input data. Compared with energy-like graphs, the gait contour sequence contains rich spatiotemporal information. Methods based on gait profile sequences generally extract gait features with view invariance through artificial neural networks. However, the current methods based on gait profile sequences cannot well extract the rich spatiotemporal information contained in gait features.

[0004] Therefore, how to extract gait features that are view-invariant and effectively utilize the rich spatiotemporal information of gait sequences is the key to improving the accuracy of cross-view gait recognition. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction. Summary of the invention:

[0007] A cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction, including backbone network construction, motion feature mixing module construction, fine-grained multi-stage feature extraction module construction, overall framework training and cross-view gait recognition.

[0008] In order to extract basic gait features, a backbone network is designed to extract features from input data. In order to obtain rich temporal information, a motion feature mixing module is constructed to mix features between adjacent frames. In order to extract detailed spatial information, a fine-grained multi-stage feature extraction module is constructed to enable the network to discover more fine-grained gait features and assign different weights to gait feature information in space. In order to improve the discrimination ability of the entire framework structure, the entire model is trained by combining triplet loss and cross entropy loss. Finally, the trained network model is used for cross-view gait recognition.

[0009] Terminology explanation:

[0010] 1. Registration data set (gallery): refers to the data set consisting of gait samples with known labels that are entered in advance.

[0011] 2. Query dataset (probe): refers to the dataset consisting of gait samples to be identified and used as test input.

[0012] 3. Gait contour sequence: refers to the binary image sequence obtained after segmentation, cropping, binarization and other processing of the original gait RGB image sequence.

[0013] 4. Gait recognition: refers to comparing each sample in the query dataset with all samples in the registration dataset, and identifying the sample in the query dataset as the sample label closest to the registration dataset.

[0014] 5. Cross-view gait recognition: refers to the gait samples in the registration dataset and the query dataset being collected under different viewpoints. The gait samples under a single viewpoint known in the registration dataset are used for modeling to realize the recognition of the query dataset collected under different viewpoints.

[0015] The technical solution of the present invention is as follows:

[0016] A cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction includes the following steps:

[0017] (1) Construct the network architecture of the feature extraction model and perform training; including:

[0018] A. Backbone network construction

[0019] The backbone network includes several convolutional layers and several pooling layers. The input gait profile sequence is extracted step by step in the backbone network to extract gait features at three stages: shallow, middle and deep.

[0020] B. Construction of motion feature hybrid module

[0021] The motion feature mixing module is used to extract the time sequence information of the gait feature; specifically, the motion feature mixing module is used to further extract the gait feature extracted in step A, and the gait information between adjacent frames is fused to extract the time sequence information;

[0022] C. Construction of fine-grained multi-stage feature extraction module

[0023] The fine-grained multi-stage feature extraction module is used to: horizontally divide the gait features including time sequence information obtained in step A and step B to achieve preliminary spatial feature extraction; after obtaining the preliminary spatial features, introduce a branch for further horizontal division, and on this basis, add a spatial attention mechanism to extract more fine-grained features;

[0024] D. Overall framework training

[0025] The network architecture of the feature extraction model is a cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, including a backbone network, a motion feature mixing module, and a fine-grained multi-stage feature extraction module;

[0026] The finer-grained features obtained by the fine-grained multi-stage feature extraction module are concatenated as the final feature representation, and the entire cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction is trained using a combination of triplet loss and cross entropy loss.

[0027] (2) Cross-view gait recognition, including: sending the gait profile sequence of the query dataset into a trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction to extract gait features, and completing the identity recognition of the query sample by comparing the feature similarity with the registered dataset.

[0028] Preferably, according to the present invention, in step A, backbone network construction includes:

[0029] The backbone network includes sequentially connected convolutional layers Conv-1, Conv-2, Conv-3, pooling layers Maxpool-1, Conv-4, Conv-5, pooling layers Maxpool-2, Conv-6, and Conv-7; after inputting the gait profile sequence, the convolutional layer Conv-3 outputs the gait features of the shallow stage, the convolutional layer Conv-5 outputs the gait features of the middle stage, and the convolutional layer Conv-7 outputs the gait features of the deep stage.

[0030] Preferably, according to the present invention, in step B, the motion feature mixing module is constructed, including:

[0031] The motion feature mixing module includes two 2D convolution layers with a convolution kernel size of 1×1, a 2D convolution layer with a parameter-sharing convolution kernel size of 3×3, a splicing unit, a Sigmoid activation function layer, a point multiplication and addition operation unit, and a maximum pooling layer;

[0032] The output feature maps of different stages are obtained through step A Among them, s represents the sequence length of a gait image sequence sample, c represents the number of output channels of the corresponding stage of the backbone network, and h and w represent the length and width of each frame feature map respectively;

[0033] First, the output feature maps x of different stages are n After a 2D convolution layer with a convolution kernel size of 1×1, the output channel is reduced by α times, where α is a hyperparameter;

[0034] Then, for each frame output feature map All subsequent frame features are passed through a parameter-shared 2D convolution layer with a convolution kernel size of 3×3, and adjacent frames are added together, as shown in formula (I):

[0035]

[0036] In formula (I), Conv 3×3 (·) represents a 2D convolutional layer with a kernel size of 3×3; represents the feature map of the t+1th frame after the frame-level information is integrated, x t ' +1 Represents x t ′’s next frame feature map;

[0037] Again, after obtaining the feature map after integrating the frame-level information, all frame-level feature maps are concatenated through the concatenation unit and restored to the feature map of the original size through a 2D convolution layer with a convolution kernel of 1×1, as shown in formula (II):

[0038]

[0039] In formula (II), Conv is the feature map after fusing the information of adjacent frames. 1×1 (·) represents a 2D convolutional layer with a convolution kernel of 1×1, and Concat(·) represents the concatenation operation along the temporal dimension; It represents the feature map of the tth frame after the frame-level information is fused;

[0040] Finally, after obtaining the feature map after fusing the information of adjacent frames, it is passed through the Sigmoid activation function of the Sigmoid activation function layer, and combined with the original feature map x through the point multiplication and addition operation unit. n Perform pixel-level dot multiplication and addition operations to obtain the output of the motion feature mixing module As shown in formula (III):

[0041]

[0042] And through the maximum pooling layer x out Perform pixel-level maximum pooling operation, as shown in formula (IV):

[0043] f n =FrameMax(x out ) (IV)

[0044] Combine the sequence into a multi-stage feature map That is, the extracted gait features including temporal information, and FrameMax(·) represents the pixel-level maximum pooling operation.

[0045] More preferably, α is 16.

[0046] Preferably, in step C, the fine-grained multi-stage feature extraction module is constructed, including:

[0047] The granular multi-stage feature extraction module includes an upper branch and a lower branch; the upper branch includes a pooling layer and a fully connected layer; the lower branch includes a clipping operation, a pooling layer, a spatial attention module and a fully connected layer; among them, the spatial attention module includes a fully connected layer, a ReLU activation function, a fully connected layer, and a Sigmoid activation function;

[0048] In extracting the multi-stage feature map f n After that, the features of each stage are horizontally divided into k blocks to obtain Perform maximum and mean pooling in the w dimension and obtain a part of the feature map through a fully connected layer As shown in formula (V) and (VI):

[0049] x reshape=Reshape h (f n ) (V)

[0050]

[0051] Among them, Reshape h (·) represents the deformation operation in the horizontal direction, Avgpooling w (·) and Maxpooling w (·) represent mean pooling and maximum pooling in w dimension respectively, FC1(·) represents the fully connected layer;

[0052] In getting x reshape After that, it is further divided along the h dimension according to the fine-grained block ratio r to obtain As shown in formula (VII):

[0053] x slice =Slice h (x reshape ) (VII)

[0054] In formula (VII), Slice h (·) indicates h-dimensional segmentation operation;

[0055] x slice Perform maximum and mean pooling in the w dimension, and then pass through a fully connected layer to obtain fine-grained features As shown in formula (VIII):

[0056] x r =FC2(Avgpooling w (x slice )+Maxpooling w (x slice )) (VIII)

[0057] The fine-grained feature x r Send it to the spatial attention module; x r The mean and maximum pooling are performed along the channel dimension and the last dimension, and then through a fully connected layer and a ReLU activation function. Finally, a fully connected layer and a Sigmoid activation function are used to obtain different weight coefficients on different blocks and compared with the original feature x. r After multiplication, we get the weighted fine-grained block features As shown in formula (IX):

[0058]

[0059] Among them, SA(·) represents the spatial attention module;

[0060] Get weighted fine-grained block features After that, the maximum pooling and mean pooling operations are performed along the horizontal block dimension to restore the number of blocks to k and obtain the feature map

[0061] Mapping the two features and Concatenate and pass through a fully connected layer to obtain the final feature map of each stage

[0062] Finally, the three final features extracted from the shallow, middle and deep stages, i.e., the more fine-grained feature maps, are concatenated to obtain the final pedestrian gait feature representation, i.e., the final feature representation f.

[0063] Preferably, in step D, the overall framework training includes:

[0064] When using the triplet loss function, the triplet loss function is shown in formula (X).

[0065]

[0066] In formula (X), is the anchor sample gait feature of the i-th triplet in the batch, represents the gait features of the positive sample with the same identity as the anchor sample, Represents the gait features of negative samples that are different from the anchor sample identity, N Tri Indicates the total number of triplets that can be formed in a batch, a i represents the anchor sample of the i-th triplet, p i represents the positive sample of the i-th triplet, n i Represents the negative sample of the i-th triple;

[0067] When using the cross entropy loss function, the subject's final feature representation f is passed through a fully connected layer with the number of output channels equal to the total number of identities N; next, the value of each element in f is converted to f by the Softmax activation function. i Normalized to an identity probability value between [0,1] It represents the probability of predicting the pedestrian identity as i, as shown in formula (XI):

[0068]

[0069] Finally, the predicted label distribution of the pedestrian features of the sample is obtained For the true label of the pedestrian sample, a one-hot vector is used to represent it, that is, for a pedestrian with identity k, a one-hot vector is constructed Let its kth element be 1, that is The values ​​of other elements are 0; then the cross entropy calculation is shown in formula (XII):

[0070]

[0071] In formula (XII), represents the true label distribution of pedestrians with identity k, represents the predicted label distribution of pedestrian with identity k, express The i-th element in express The i-th element in ;

[0072] Assuming that the number of samples in a batch is B, the cross entropy loss function is defined as formula (XIII):

[0073]

[0074] The loss function used when training the overall framework As shown in formula (XIV).

[0075]

[0076] By minimizing the combination of triplet loss and cross entropy loss To complete the training of the entire cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction.

[0077] Preferably, in step E, cross-view gait recognition includes:

[0078] The registered data set is input into the trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, and the output features of the trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction are used as the overall feature representation of each gait profile sequence, and finally the feature database of the registered data set is obtained;

[0079] The samples in the query data set to be identified are fed into a trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, and the final feature representation of the query data set is output;

[0080] The Euclidean distance between the final feature representation of each gait sample in the query data set and all the features in the obtained registration data set is calculated, and finally the query sample is identified as the label of the feature with the smallest Euclidean distance in the registration data set, and the identity label of the query sample is output to complete cross-view gait recognition.

[0081] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction are implemented.

[0082] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction.

[0083] The beneficial effects of the present invention are:

[0084] 1. The motion feature mixing module proposed in the present invention can realize frame-level feature mixing, so that the gait feature of each frame can fuse the features between adjacent frames, so as to extract motion information and enrich the temporal features.

[0085] 2. The fine-grained multi-stage feature extraction module proposed in the present invention realizes the refinement of gait features in the spatial dimension, and adds a spatial attention mechanism to the module, which can focus on more useful fine-grained gait features. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 A schematic diagram of the network architecture of the motion feature mixing module in the present invention;

[0087] FIG2( a ) is a schematic diagram of the network architecture of the fine-grained multi-stage feature extraction module in the present invention;

[0088] FIG2( b ) is a schematic diagram of the network architecture of the spatial attention module in the fine-grained multi-stage feature extraction module of the present invention;

[0089] Figure 3 Schematic diagram of the overall framework of the cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction proposed in the present invention. DETAILED DESCRIPTION

[0090] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0091] Example 1

[0092] A cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction, such as Figure 3 As shown, the steps include:

[0093] (1) Construct the network architecture of the feature extraction model and perform training; including:

[0094] A. Backbone network construction

[0095] The backbone network includes several convolutional layers and several pooling layers. The input gait profile sequence is extracted step by step in the backbone network to extract gait features at three stages: shallow, middle and deep.

[0096] B. Construction of motion feature hybrid module

[0097] The motion feature mixing module is used to extract the time sequence information of the gait feature; specifically, the motion feature mixing module is used to further extract the gait feature extracted in step A, and the gait information between adjacent frames is fused to extract the time sequence information;

[0098] C. Construction of fine-grained multi-stage feature extraction module

[0099] The fine-grained multi-stage feature extraction module is used to: horizontally divide the gait features including time sequence information obtained in step A and step B to achieve preliminary spatial feature extraction; after obtaining the preliminary spatial features, introduce a branch for further horizontal division, and on this basis, add a spatial attention mechanism to extract more fine-grained features;

[0100] D. Overall framework training

[0101] The network architecture of the feature extraction model is a cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, including a backbone network, a motion feature mixing module, and a fine-grained multi-stage feature extraction module;

[0102] The finer-grained features obtained by the fine-grained multi-stage feature extraction module are spliced ​​as the final feature representation, and the entire cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction is trained using a combination of triplet loss and cross entropy loss to obtain a feature extraction model with good fitting ability and strong discriminability;

[0103] (2) Cross-view gait recognition, including: sending the gait profile sequence of the query dataset into a trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction to extract gait features, and completing the identity recognition of the query sample by comparing the feature similarity with the registered dataset.

[0104] Example 2

[0105] The cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction described in Example 1 is different in that:

[0106] In step A, the backbone network is constructed, including:

[0107] The backbone network consists of a series of convolutional layers and pooling layers. The input gait profile sequence extracts the gait features of the shallow, middle and deep stages in the backbone network step by step. Its network structure and parameters are shown in Table 1. The backbone network includes sequentially connected convolutional layers Conv-1, Conv-2, Conv-3, Maxpool-1, Conv-4, Conv-5, Maxpool-2, Conv-6 and Conv-7. After the gait profile sequence is input, two pooling layers (Maxpool-1 and Maxpool-2) are applied to the corresponding convolutional layers, which reduces the size of the feature map and divides the backbone network into three stages. The feature map also has more advanced semantic information according to different stages. Convolutional layer Conv-3 outputs the gait features of the shallow stage, convolutional layer Conv-5 outputs the gait features of the middle stage, and convolutional layer Conv-7 outputs the gait features of the deep stage. The outputs of the convolutional layers Conv-3, Conv-5, and Conv-7 are used as inputs for subsequent steps. The characteristic of this operation is that it can extract both low-level and high-level features of the gait sequence. The shallower stages extract more features such as image texture and contour, while the deeper stages focus more on high-level and more abstract semantic information. This enables the subsequent network to process richer information and makes the network more scalable.

[0108] Table 1

[0109]

[0110]

[0111] In step B, a motion feature mixing module is constructed, and the motion feature mixing module focuses on extracting the temporal information of gait features. The gait features extracted in step A are further extracted using the motion feature mixing module, which can fuse the gait information between adjacent frames to achieve the extraction of temporal information, including:

[0112] The motion feature mixing module includes two 2D convolution layers with a convolution kernel size of 1×1, a 2D convolution layer with a parameter-sharing convolution kernel size of 3×3, a splicing unit, a Sigmoid activation function layer, a point multiplication and addition operation unit, and a maximum pooling layer; the network architecture of the motion feature mixing module is as follows: Figure 1 As shown;

[0113] The output feature maps of different stages are obtained through step A (i.e., gait features of the shallow, middle, and deep layers), where s represents the sequence length of a gait image sequence sample, c represents the number of output channels of the corresponding stage of the backbone network, and h and w represent the length and width of each frame feature map, respectively;

[0114] First, the output feature maps x of different stages are n After a 2D convolution layer with a convolution kernel size of 1×1, the output channel is reduced by α times, where α is a hyperparameter and α is set to 16.

[0115] Then, for each frame output feature map All subsequent frame features are passed through a parameter-shared 2D convolution layer with a convolution kernel size of 3×3, and adjacent frames are added together, as shown in formula (I):

[0116]

[0117] In formula (I), Conv 3×3 (·) represents a 2D convolutional layer with a kernel size of 3×3; represents the feature map of the t+1th frame after the frame-level information is integrated, x t ' +1 Represents x t ′’s next frame feature map;

[0118] Again, after obtaining the feature map after integrating the frame-level information, all frame-level feature maps are concatenated through the concatenation unit and restored to the feature map of the original size through a 2D convolution layer with a convolution kernel of 1×1, as shown in formula (II):

[0119]

[0120] In formula (II), Conv is the feature map after fusing the information of adjacent frames. 1×1 (·) represents a 2D convolutional layer with a convolution kernel of 1×1, and Concat(·) represents the concatenation operation along the temporal dimension; It represents the feature map of the tth frame after the frame-level information is fused;

[0121] Finally, after obtaining the feature map after fusing the information of adjacent frames, it is passed through the Sigmoid activation function of the Sigmoid activation function layer, and combined with the original feature map x through the point multiplication and addition operation unit. n Perform pixel-level dot multiplication and addition operations to obtain the output of the motion feature mixing module As shown in formula (III):

[0122]

[0123] And through the maximum pooling layer x out Perform pixel-level maximum pooling operation, as shown in formula (IV):

[0124] f n=FrameMax(x out ) (IV)

[0125] Combine the sequence into a multi-stage feature map That is, the extracted gait features including temporal information, FrameMax(·) represents the pixel-level maximum pooling operation. It can not only extract the most obvious temporal features, but also reasonably reduce the amount of calculation, which is convenient for subsequent network operations;

[0126] In step C, a fine-grained multi-stage feature extraction module is constructed. The fine-grained multi-stage feature extraction module horizontally divides the gait features of multiple stages obtained through steps A and B to achieve preliminary spatial feature extraction; then, after obtaining the horizontally divided features, a branch is introduced to further horizontally divide the features, and on this basis, a spatial attention mechanism is added to extract more fine-grained features, including:

[0127] The granular multi-stage feature extraction module includes an upper branch and a lower branch; the upper branch includes a pooling layer and a fully connected layer; the lower branch includes a clipping operation, a pooling layer, a spatial attention module and a fully connected layer; among them, the spatial attention module includes a fully connected layer, a ReLU activation function, a fully connected layer, and a Sigmoid activation function;

[0128] The fine-grained multi-stage feature extraction module is shown in Figure 2(a). n After that, the features of each stage are divided into k blocks horizontally. The features of the shallow stage contain less information, so the features of the whole and the blocks with fewer blocks are extracted, and the feature map is divided into k = 2 blocks horizontally; the deep stage contains more information, so the detailed features of the blocks with more blocks are extracted, and the feature map is divided into k = 16 blocks horizontally; the middle stage divides the feature map into k = 4 blocks horizontally, and obtains Perform maximum and mean pooling in the w dimension and obtain a part of the feature map through a fully connected layer As shown in formula (V) and (VI):

[0129] x reshape =Reshape h (f n ) (V)

[0130]

[0131] Among them, Reshape h (·) represents the deformation operation in the horizontal direction, Avgpooling w (·) and Maxpooling w(·) represent mean pooling and maximum pooling of w dimensions respectively, FC1(·) represents a fully connected layer; in the present invention, the hidden layer of the fully connected layer is set to 256 dimensions;

[0132] In order to discover more fine-grained feature information, after obtaining x reshape After that, it is further divided along the h dimension according to the fine-grained block ratio r to obtain As shown in formula (VII):

[0133] x slice =Slice h (x reshape ) (VII)

[0134] In formula (VII), Slice h (·) indicates h-dimensional segmentation operation;

[0135] x slice Perform maximum and mean pooling in the w dimension, and then pass through a fully connected layer to obtain fine-grained features As shown in formula (VIII):

[0136] x r =FC2(Avgpooling w (x slice )+Maxpooling w (x slice )) (VIII)

[0137] The fine-grained feature x r Send it to the spatial attention module; x r The mean and maximum pooling are performed along the channel dimension and the last dimension, and then through a fully connected layer and a ReLU activation function. Finally, a fully connected layer and a Sigmoid activation function are used to obtain different weight coefficients on different blocks and compared with the original feature x. r After multiplication, we get the weighted fine-grained block features As shown in formula (IX):

[0138]

[0139] Wherein, SA(·) represents the spatial attention module, and the spatial attention module is shown in FIG2(b);

[0140] Get weighted fine-grained block features After that, the maximum pooling and mean pooling operations are performed along the horizontal block dimension to restore the number of blocks to k and obtain the feature map

[0141] Mapping the two features and Concatenate and pass through a fully connected layer to obtain the final feature map of each stage

[0142] Finally, the three final features extracted from the shallow, middle and deep stages, i.e., the more fine-grained feature maps, are concatenated to obtain the final pedestrian gait feature representation, i.e., the final feature representation f.

[0143] In step D, the overall framework training is performed, and the features obtained through the fine-grained multi-stage feature extraction module are spliced ​​as the final feature representation. The entire gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction is trained using a combination of triplet loss and cross entropy loss to obtain a feature extraction model with good fitting ability and strong discriminability. Including:

[0144] When using the triplet loss function, the triplet loss function is shown in formula (X).

[0145]

[0146] In formula (X), is the anchor sample gait feature of the i-th triplet in the batch, represents the gait features of the positive sample with the same identity as the anchor sample, Represents the gait features of negative samples that are different from the anchor sample identity, N Tri Indicates the total number of triplets that can be formed in a batch, a i represents the anchor sample of the i-th triplet, p i represents the positive sample of the i-th triplet, n i represents the negative sample of the i-th triple;

[0147] When using the cross entropy loss function, the subject's final feature representation f is passed through a fully connected layer with the number of output channels equal to the total number of identities N; next, the value of each element in f is converted to f by the Softmax activation function. i Normalized to an identity probability value between [0,1] It represents the probability of predicting the pedestrian identity as i, as shown in formula (XI):

[0148]

[0149] Finally, the predicted label distribution of the pedestrian features of the sample is obtained For the true label of the pedestrian sample, a one-hot vector is used to represent it, that is, for a pedestrian with identity k, a one-hot vector is constructed Let its kth element be 1, that is The values ​​of other elements are 0; then the cross entropy calculation is shown in formula (XII):

[0150]

[0151] In formula (XII), represents the true label distribution of pedestrians with identity k, represents the predicted label distribution of pedestrian with identity k, express The i-th element in express The i-th element in ;

[0152] Assuming that the number of samples in a batch is B, the cross entropy loss function is defined as formula (XIII):

[0153]

[0154] The loss function used when training the overall framework As shown in formula (XIV).

[0155]

[0156] By minimizing the combination of triplet loss and cross entropy loss To complete the training of the cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, this can ensure that the feature extraction model can converge quickly and achieve better recognition performance.

[0157] In step E, cross-view gait recognition includes:

[0158] The registered data set is input into the trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, and the output features of the trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction are used as the overall feature representation of each gait profile sequence, and finally the feature database of the registered data set is obtained;

[0159] The samples in the query data set to be identified are fed into a trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, and the final feature representation of the query data set is output;

[0160] The Euclidean distance between the final feature representation of each gait sample in the query data set and all the features in the obtained registration data set is calculated, and finally the query sample is identified as the label of the feature with the smallest Euclidean distance in the registration data set, and the identity label of the query sample is output to complete cross-view gait recognition.

[0161] In this embodiment, the size of the gait profile sequence is 64×44. In the training stage, 30 frames of each gait sequence are taken, and in the test stage, all frames of the gait sequence are taken. In terms of hyperparameter setting, the number of multi-stage pyramid blocks k is set to: k=2 blocks in the shallow layer, k=4 blocks in the middle layer, and k=16 blocks in the deep layer, and the fine-grained block ratio r is set to: r1=4 in the shallow layer, r1=2 in the middle layer, and r1=1 in the deep layer. The margin of the triple loss is fixed to 0.2, and the network is trained using the SGD optimizer, and the initial learning rate is set to 0.1. During the training process, the learning rate is reduced to 0.01 at the 8kth time and to 0.001 at the 16kth time, and a total of 30k training times.

[0162] The CASIA-B gait dataset was created by the Institute of Automation, Chinese Academy of Sciences. The gait data of 124 subjects were collected in a laboratory environment. Each subject contains three types of sequences: normal type (NM), bagging type (BG), and clothing type (CL). Among them, each subject has six normal type gait sequences: NM-01, NM-02, NM-03, NM-04, NM-05, NM-06; two bagging type gait sequences: BG-01, BG-02; two clothing type gait sequences: CL-01, CL-02. For each group of different types of gait samples, different situations under 11 viewing angles (0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 154°, 162°, 180°) are included. In summary, the CASIA-B gait dataset contains a total of about 124×10×11×13640 gait sequences. This embodiment experiment uses all samples of the first 74 people when training the network model on the CASIA-B dataset, and the test set uses samples of the last 50 people in the dataset. In the test set, NM-01, NM-02, NM-03, and NM-04 are taken as registration samples, NM-05 and NM-06 are taken as normal type query samples, BG-01 and BG-02 are taken as backpack type query samples, and CL-01 and CL-02 are taken as clothing type query samples. Set each batch size to 128, that is, select 8 subjects with different identities, and select 16 sequences for each subject. In order to verify the advancedness of the cross-view gait recognition method based on spatiotemporal information enhancement and multi-scale salient feature extraction proposed in this invention, the present invention is compared with 7 existing advanced gait recognition methods on the CASIA-B dataset, including GEInet, CNN-LB, ACL, GaitPart, GaitSet, GCEM, and GaitSlice. Table 2 shows the average Rank-1 accuracy (%) of cross-view recognition at different query perspectives in the CASIA-B database.

[0163] Table 2

[0164]

[0165]

[0166] It can be seen from Table 2 that the method of the present invention achieves the best recognition effect under all walking conditions. Under the three types of walking conditions, the recognition rate of the method of the present invention exceeds all other advanced comparison methods, reaching 97.7%, 93.5% and 81.8% respectively.

[0167] Example 3

[0168] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction of embodiment 1 or 2 are implemented.

[0169] Example 4

[0170] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction in embodiment 1 or 2.

Claims

1. A cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction, characterized in that: The steps include: (1) Construct the network architecture of the feature extraction model and perform training; including: A. Backbone network construction The backbone network includes several convolutional layers and several pooling layers. The input gait profile sequence is extracted step by step in the backbone network to extract gait features at three stages: shallow, middle and deep. B. Construction of motion feature hybrid module The motion feature mixing module is used to extract the timing information of the gait features; specifically, the motion feature mixing module is used to further extract the gait features extracted in step A, and the gait information between adjacent frames is fused to extract the timing information; the motion feature mixing module includes two 2D convolution layers with a convolution kernel size of 1×1, a 2D convolution layer with a parameter-shared convolution kernel size of 3×3, a splicing unit, a Sigmoid activation function layer, a point multiplication and addition operation unit, and a maximum pooling layer; C. Construction of fine-grained multi-stage feature extraction module The fine-grained multi-stage feature extraction module is used to: horizontally divide the gait features including time sequence information obtained in step A and step B to achieve preliminary spatial feature extraction; after obtaining the preliminary spatial features, introduce a branch for further horizontal division, and on this basis, add a spatial attention mechanism to extract more fine-grained features; D. Overall framework training The network architecture of the feature extraction model is a cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, including a backbone network, a motion feature mixing module, and a fine-grained multi-stage feature extraction module; The finer-grained features obtained by the fine-grained multi-stage feature extraction module are concatenated as the final feature representation, and the entire cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction is trained using a combination of triplet loss and cross entropy loss. (2) Cross-view gait recognition, including: sending the gait profile sequence of the query dataset into a trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction to extract gait features, and completing the identity recognition of the query sample by comparing the feature similarity with the registered dataset.

2. According to claim 1, a cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction is characterized in that: In step A, the backbone network is constructed, including: The backbone network includes sequentially connected convolutional layers Conv-1, Conv-2, Conv-3, pooling layers Maxpool-1, Conv-4, Conv-5, pooling layers Maxpool-2, Conv-6, and Conv-7; after inputting the gait profile sequence, the convolutional layer Conv-3 outputs the gait features of the shallow stage, the convolutional layer Conv-5 outputs the gait features of the middle stage, and the convolutional layer Conv-7 outputs the gait features of the deep stage.

3. The cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction according to claim 1 is characterized in that: In step B, a motion feature mixing module is constructed, including: The output feature maps of different stages are obtained through step A Among them, s represents the sequence length of a gait image sequence sample, c represents the number of output channels of the corresponding stage of the backbone network, and h and w represent the length and width of each frame feature map respectively; First, the output feature maps x of different stages are n After a 2D convolution layer with a convolution kernel size of 1×1, the output channel is reduced by α times, where α is a hyperparameter; Then, for each frame output feature map All subsequent frame features are passed through a parameter-shared 2D convolution layer with a convolution kernel size of 3×3, and adjacent frames are added together, as shown in formula (I): In formula (I), Conv 3×3 (·) represents a 2D convolutional layer with a kernel size of 3×3; represents the feature map of the t+1th frame after the frame-level information is integrated, x t ' +1 Represents x t ′’s next frame feature map; Again, after obtaining the feature map after integrating the frame-level information, all frame-level feature maps are concatenated through the concatenation unit and restored to the feature map of the original size through a 2D convolution layer with a convolution kernel of 1×1, as shown in formula (II): In formula (II), Conv is the feature map after fusing the information of adjacent frames. 1×1 (·) represents a 2D convolutional layer with a convolution kernel of 1×1, and Concat(·) represents the concatenation operation along the temporal dimension; It represents the feature map of the tth frame after the frame-level information is fused; Finally, after obtaining the feature map after fusing the information of adjacent frames, it is passed through the Sigmoid activation function of the Sigmoid activation function layer, and combined with the original feature map x through the point multiplication and addition operation unit. n Perform pixel-level dot multiplication and addition operations to obtain the output of the motion feature mixing module As shown in formula (III): And through the maximum pooling layer x out Perform pixel-level maximum pooling operation, as shown in formula (IV): f n =FrameMax(x out )(IV) Combine the sequence into a multi-stage feature map That is, the extracted gait features including temporal information, and FrameMax(·) represents the pixel-level maximum pooling operation.

4. The cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction according to claim 3 is characterized in that: α is set to 16.

5. The cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction according to claim 1 is characterized in that: In step C, a fine-grained multi-stage feature extraction module is constructed, including: The fine-grained multi-stage feature extraction module includes an upper branch and a lower branch; the upper branch includes a pooling layer and a fully connected layer; the lower branch includes a cropping operation, a pooling layer, a spatial attention module and a fully connected layer; among them, the spatial attention module includes a fully connected layer, a ReLU activation function, a fully connected layer, and a Sigmoid activation function; In extracting the multi-stage feature map f n After that, the features of each stage are horizontally divided into k blocks to obtain Perform maximum and mean pooling in the w dimension and obtain a part of the feature map through a fully connected layer As shown in formula (V) and (VI): x reshape =Reshape h (f n )(V) Among them, Reshape h (·) represents the deformation operation in the horizontal direction, Avgpooling w (·) and Maxpooling w (·) represent mean pooling and maximum pooling in w dimension respectively, FC1(·) represents the fully connected layer; In getting x reshape After that, it is further divided along the h dimension according to the fine-grained block ratio r to obtain As shown in formula (VII): x slice =Slice h (x reshape )(VII) In formula (VII), Slice h (·) indicates h-dimensional segmentation operation; x slice Perform maximum and mean pooling in the w dimension, and then pass through a fully connected layer to obtain fine-grained features As shown in formula (VIII): x r =FC2(Avgpooling w (x slice )+Maxpooling w (x slice ))(VIII) The fine-grained feature x r Send it to the spatial attention module; x r The mean and maximum pooling are performed along the channel dimension and the last dimension, and then through a fully connected layer and a ReLU activation function. Finally, a fully connected layer and a Sigmoid activation function are used to obtain different weight coefficients on different blocks and compared with the original feature x. r After multiplication, we get the weighted fine-grained block features As shown in formula (IX): Among them, SA(·) represents the spatial attention module; Get weighted fine-grained block features After that, the maximum pooling and mean pooling operations are performed along the horizontal block dimension to restore the number of blocks to k and obtain the feature map Mapping the two features and Concatenate and pass through a fully connected layer to obtain the final feature map of each stage Finally, the three final features extracted from the shallow, middle and deep stages, i.e., the more fine-grained feature maps, are concatenated to obtain the final pedestrian gait feature representation, i.e., the final feature representation f.

6. The cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction according to claim 1 is characterized in that: In step D, the overall framework training includes: When using the triplet loss function, the triplet loss function is shown in formula (X): In formula (X), is the anchor sample gait feature of the i-th triplet in the batch, represents the gait features of the positive sample with the same identity as the anchor sample, Represents the gait features of negative samples that are different from the anchor sample identity, N Tri Indicates the total number of triplets that can be formed in a batch, a i represents the anchor sample of the i-th triplet, p i represents the positive sample of the i-th triplet, n i Represents the negative sample of the i-th triple; When using the cross entropy loss function, the subject's final feature representation f is passed through a fully connected layer with the number of output channels equal to the total number of identities N; next, the value of each element in f is converted to f by the Softmax activation function. i Normalized to an identity probability value between [0,1] It represents the probability of predicting the pedestrian identity as i, as shown in formula (XI): Finally, the predicted label distribution of the pedestrian features of the sample is obtained For the true label of the pedestrian sample, a one-hot vector is used to represent it, that is, for a pedestrian with identity k, a one-hot vector is constructed Let its kth element be 1, that is The values ​​of other elements are 0; then the cross entropy calculation is shown in formula (XII): In formula (XII), represents the true label distribution of pedestrians with identity k, represents the predicted label distribution of pedestrian with identity k, express The i-th element in express The i-th element in ; Assuming that the number of samples in a batch is B, the cross entropy loss function is defined as formula (XIII): The loss function used when training the overall framework As shown in formula (XIV): By minimizing the combination of triplet loss and cross entropy loss To complete the training of the entire cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction.

7. A cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction according to any one of claims 1-6, characterized in that: In step E, cross-view gait recognition includes: The registered data set is input into the trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, and the output features of the trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction are used as the overall feature representation of each gait profile sequence, and finally the feature database of the registered data set is obtained; The samples in the query data set to be identified are fed into a trained cross-view gait recognition network based on motion feature mixing and fine-grained multi-stage feature extraction, and the final feature representation of the query data set is output; The Euclidean distance between the final feature representation of each gait sample in the query data set and all the features in the obtained registration data set is calculated, and finally the query sample is identified as the label of the feature with the smallest Euclidean distance in the registration data set, and the identity label of the query sample is output to complete cross-view gait recognition.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction described in any one of claims 1-7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-view gait recognition method based on motion feature mixing and fine-grained multi-stage feature extraction described in any one of claims 1 to 7 are implemented.

Images