Multi-branch pedestrian gait recognition method and device based on multi-frame rate and position coding

Abstract

In order to improve the accuracy of gait recognition, especially in combination with the use of the overall sequence and local details of gait, a multi-branch pedestrian gait recognition method based on multi-frame rate and position coding is provided.The input of the method is the multi-frame continuous gait profile sequence of the queried personnel in the probe set of the test set, and the output is the identity ID of the personnel identified in the test set gallery.The application constructs a multi-frame branch gait recognition model MGRF (Multi-branch Gait Recognition for multi-Frame-rate) based on deep learning, realizes feature extraction of the pedestrian gait profile sequence based on position coding, feature fusion and attention mechanism, and realizes the cross-angle identity recognition task of the pedestrian through the ternary loss function and the cross-entropy function with label smoothing.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-branch pedestrian gait recognition method and apparatus based on multi-frame rate and position coding. Background Technology

[0002] Gait recognition identifies a person by analyzing their walking patterns. While traditional identity recognition methods rely on facial features and appearance, their performance is unsatisfactory under conditions of long shooting distances, low resolution, long time intervals, and complex lighting. Compared to other biometrics, gait features offer advantages such as long-distance acquisition, no need for cooperation, and difficulty in forgery. Even in challenging situations, gait contour sequences can be segmented to extract effective gait features, providing a feasible method for locating target individuals in surveillance videos. Based on these advantages, gait retrieval technology is of significant importance for various applications, including security monitoring, criminal investigations, tracking suspects, and identifying missing children.

[0003] With the development of artificial intelligence, deep learning-based methods have been widely applied to gait recognition tasks. Existing deep learning-based gait retrieval techniques mostly employ gait features based on gait templates or gait sequences for training and testing.

[0004] The most common gait template is the Gait Energy Image (GEI), which is obtained by aligning, cropping, adding, and averaging gait contour sequences. Its calculation is simple and it can effectively represent gait features. However, current gait retrieval methods using GEI have the following problems: The addition and averaging operations in the calculation process of GEI compress temporal information, making it difficult to effectively extract continuous temporal gait features between gait contour images.

[0005] Current methods using gait contour sequences typically employ frame sampling during data loading to reduce computational costs, decreasing the temporal dimension of the input sequence to a predetermined number of frames. This process reduces the number of frames in the gait sequence to half or even less. However, gait features represent temporal micro-motion patterns, and gait contour maps from the same gait sequence are closely related temporally. Clearly, frame sampling removes adjacent frames, significantly reducing the effective fine-grained micro-motion information of the gait. Summary of the Invention

[0006] To address the problem of low gait recognition accuracy in existing technologies, which is due to the difficulty in effectively extracting continuous temporal gait features between gait contour maps and the inability to retain and extract fine-grained gait features from adjacent frames, this invention proposes a multi-branch pedestrian gait recognition method based on multi-frame rate and position coding. This method targets the gait feature extraction network for the overall motion pattern and local motion details of multi-frame gait sequences, and combines temporal sampling, position coding, and attention mechanisms to propose a multi-branch pedestrian gait recognition method to improve the accuracy of gait recognition.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] The first aspect discloses a multi-branch pedestrian gait recognition method based on multi-frame rate and position coding, including:

[0009] S1: Obtain the original dataset, crop and resize the images in the original dataset, and divide the dataset into a training set and a test set with mutually exclusive identity labels. Then, divide the training set and the test set into a probe set and an image library, respectively. Process the complete gait contour sequence into three sets of input data with frame rates from low to high and resolutions from high to low, according to different frame sampling strategies and resolution reduction operations. The input data specifically includes low frame rate high resolution input, medium frame rate medium resolution input, and high frame rate low resolution input.

[0010] S2: Construct a deep learning-based multi-frame branch gait recognition model. The overall network architecture of the multi-frame branch gait recognition model is a three-branch structure. The first, second, and third branches handle low frame rate high-resolution input, medium frame rate medium-resolution input, and high frame rate low-resolution input, respectively. Each branch contains a position encoding module, a specific convolution module, a lateral feature set fusion connection, a specific convolution module, a lateral feature set scatter fusion connection, a block convolution module, and a lateral feature set fusion connection. The specific convolution modules of the first, second, and third branches adopt two-dimensional convolution, BP3D convolution, and three-dimensional convolution, respectively. The features after the last lateral feature set fusion connection are then processed by a self-attention module, GeM pooling, a horizontal pyramid, a block fully connected layer, and BNNeck.

[0011] S3: Train the multi-frame branch gait recognition model using the training set;

[0012] S4: Input the test set into the multi-frame branch gait recognition model trained on the test set, obtain the block gait features of the probe set and the image library on the test set, calculate the Euclidean distance between the block gait features of the probe set and the image library and sort them, and return the identity ID of the first ranked one as the recognition result.

[0013] In one implementation, step S1 includes:

[0014] S1.1: Crop and resize the image, specifically including: for each gait contour image in the gait contour sequence, identify the top, bottom, left and right ends of the human body, crop it into a rectangle containing only the human body, and adjust the image size to a uniform height H0 and width W0.

[0015] S1.2: Divide the training set and the test set into two disjoint sets;

[0016] S1.3: Divide the dataset into probe set and image set. The dataset structure is as follows: each identity ID folder contains different clothing types, and each clothing type contains several gait sequences from all angles in different rounds of shooting; divide the rounds in each type into probe set and test set according to a certain ratio.

[0017] S1.4: The complete gait contour sequence is processed according to different frame sampling strategies and resolution reduction operations, resulting in three sets of input data X with frame rates ranging from low to high and resolutions from high to low. f0 X s0 X t0 These are used as inputs for the three branches, where the subscripts f, s, and t represent the first branch, the second branch, and the third branch, respectively.

[0018] In one implementation, step S3 includes:

[0019] S3.1: Input three sets of data with frame rates from low to high and resolutions from high to low into the three branches of the three-branch network, respectively. After passing through the position encoding module, data with added height, width, and temporal coordinates are obtained. Input the three sets of data with added coordinates into their respective convolution modules, and use 2D convolution, BP3D, and 3D convolution respectively according to the frame rate from low to high to obtain their respective first-level gait features.

[0020] S3.2: The three sets of primary gait features are fused together, and the features of the upper and lower branches are converged into the middle branch. Specifically, the primary gait features with low frame rate and high resolution are spatially pooled, and the primary gait features with high frame rate and low resolution are spatially upsampled and temporally pooled, and then added element by element to the temporal position corresponding to the primary gait features with medium frame rate and medium resolution.

[0021] S3.3: The first-level gait features of the upper and lower branches and the fused features of the middle branch are input into their respective convolutional modules to obtain their respective second-level gait features; the three sets of second-level gait features are then fused in a feature-dispersed manner, specifically by adding the features of the middle branch to the upper and lower branches after performing different processing.

[0022] S3.4: Perform horizontal block convolution on the secondary gait features of the middle branch and the fusion features of the upper and lower branches according to different settings to obtain the tertiary gait features of the three branches;

[0023] S3.5: The three sets of three-level gait features are fused together. Specifically, the features of the upper and lower branches are converged into the middle branch to obtain the preliminary frame-level gait features; the preliminary frame-level gait features are processed by the self-attention module to obtain the enhanced frame-level gait features; and the enhanced frame-level gait features are then processed by GeM temporal pooling to converge the information in the time dimension to obtain the sequence-level gait features.

[0024] S3.6: The sequence-level gait features are passed through a horizontal pyramid pooling layer, and then horizontally segmented, shaped, and stitched together to obtain pyramid multi-scale features;

[0025] S3.7: The multi-scale features of the pyramid are mapped to each block through a block-based fully connected layer to obtain block gait features, which are used to calculate the triplet loss. The block gait features are then passed through a BNNeck layer to obtain classification probabilities, which are used together with the real identity labels to calculate the labeled smooth cross-entropy loss. During the training phase, the loss of the entire network is a weighted sum of the triplet loss and the labeled smooth cross-entropy loss, and is optimized using the AdamW optimizer.

[0026] In one implementation, step S3.1 includes:

[0027] S3.1.1: The position encoding module performs absolute position encoding based on the height H0 and width W0 of the input data. Floating-point coordinates with a range of [-1, 1] and intervals of 2 / W0 and 2 / H0 are generated from the height and width values, respectively. After adding dimensions, these coordinates are concatenated at the end of the channel dimension. The dimensions of the three sets of gait sequence data with added coordinates are increased by 2 compared to the previous channel dimension.

[0028] S3.1.2: Concatenate the absolute position coordinates to the end of the input channel dimension to obtain three sets of gait sequence data with additional coordinates;

[0029] S3.1.3: The first branch pairs the low frame rate, high resolution gait contour sequence X. f0 Two-dimensional convolution is performed and then activated by the LeakyReLU function to obtain local details in its spatial form, resulting in the first-branch first-level gait feature X. f1 =Conv 2×3 (X f0 );

[0030] S3.1.4: Second branch aligns frame rate with medium resolution gait contour sequence X s0BP3D convolution is performed in two paths. First, a two-dimensional convolution is performed on the height and width planes, followed by a convolution on the temporal dimension. The other path performs a three-dimensional convolution on the input. The results of the two convolutions are element-wise summed and then passed through the Leaky ReLU activation function to obtain its spatial and temporal features, resulting in the second-branch first-level gait feature X. s1 , represented as: X s1 =Conv3d 3×3×3 (X s0 )+Conv3d 1×3×3 (X s0 )+Conv3d 3 ×1×1 (X s0 );

[0031] S3.1.5: The third branch pairs high frame rate, low resolution gait contour sequences X t0 Perform global 3D convolution and pass it through the Leaky ReLU activation function, focusing on the global motion pattern in the temporal direction, to obtain the third branch first-level gait feature X. t1 =Conv3d 3×3×3 (X t0 ).

[0032] In one implementation, step S3.2 includes:

[0033] S3.2.1: First-order gait features X of the first branch f1 Spatial pooling is performed, using max pooling to reduce the width and height to one-rth of their original values, so that they are related to the X value of the second branch. s1 The height and width are the same, and then they are added together according to the time and position coordinates to the first-level gait feature X of the second branch. s1 At the corresponding time position;

[0034] S3.2.2: First-order gait features X of the third branch t1 Perform an upsampling operation in space, increasing the width and height to r times their original values, and then compare this to the X value of the second branch. s1 With consistent height and width, max pooling is then performed on several adjacent frames in the time dimension, reducing the number of frames to r times the original value. These frames are then added together according to their temporal position coordinates to the first-level gait feature X in the second branch. s1 At the corresponding time position;

[0035] S3.2.3: After changing the temporal frame number and spatial height and width, the features of the upper and lower branches are combined with the first-level gait feature X of the second branch. s1 Add them together to obtain the fused feature X. p1 =X s1 MaxPooling r×r (Xf1 )+MaxPooling r×1×1 (UpSample r×r (X t1 )).

[0036] In one implementation, step S3.3 includes:

[0037] S3.3.1: For the first-level gait feature X of the first branch f1 Continue with the two-dimensional convolution and LeakyReLU activation function processing in S3.1.3 to obtain the first branch secondary gait feature X. f2 =Conv 3×3 (X f1 );

[0038] S3.3.2: Feature X after centralized fusion of the second branch p1 Continue with the BP3D convolution processing in S3.1.4 to obtain the second branch secondary gait feature X. s2 , represented as: X s2 =Conv3d 3×3×3 (X p1 )+Conv3d 1×3×3 (X p1 )+Conv3d 3×1×1 (X p1 );

[0039] S3.3.3: For the first-level gait feature X of the third branch t1 Continuing with the 3D convolution and LeakyReLU activation function processing in S3.1.5, the third branch secondary gait feature X is obtained. t2 =Conv3d 3×3×3 (X t1 );

[0040] S3.3.4: For the second branch secondary gait feature X s2 In spatial terms, an upsampling operation is performed, increasing the width and height to r times their original values, and then comparing them with the X of the first branch. f2 With consistent height and width, max pooling is then performed on several adjacent frames in the time dimension, reducing the number of frames to one-rth of the original. The frames are then added to the second-level gait feature X of the first branch according to their temporal position coordinates. f2 At the corresponding time position;

[0041] S3.3.5: For the second branch secondary gait feature X s2 In spatial pooling, max pooling reduces the width and height to one-rth of their original values, making them comparable to the third branch's X. t2The height and width are the same, and then they are added together according to the time and position coordinates to the secondary gait feature X of the third branch. t2 At the corresponding time position;

[0042] S3.3.6: Secondary gait features X of the first branch f2 The second branch secondary gait features X after upsampling and temporal pooling s2 Adding them together yields the first branch fusion feature X. q1 =X f2 MaxPooling r×1×1 (UpSample r×r (X s2 ));

[0043] S3.3.7: Secondary gait features X of the third branch t2 The second branch secondary gait feature X after spatial max pooling s2 Adding them together yields the third branch fusion feature X. q2 =X t2 MaxPooling r×r (X s2 ).

[0044] In one implementation, step S3.4 includes:

[0045] S3.4.1: For the fusion feature X of the first branch q1 Divide its height into b along the horizontal direction. 2 The first branch's third-level feature X is obtained by performing a 3D convolution on each small block and activating it with Leaky ReLU, and then concatenating the results of each block along the height direction. f3 b is the base number of blocks;

[0046] S3.4.2: For the second branch secondary gait feature X s2 The height is divided into b blocks along the horizontal direction. A 3D convolution is performed on each block and activated by Leaky ReLU. The results of each block are then concatenated along the height direction to obtain the second branch's third-level feature X. s3 ;

[0047] S3.4.3: Fusion feature X for the third branch q2 Without any partitioning, a 3D convolution is performed on the entire feature and activated by Leaky ReLU to obtain the third branch tertiary feature X. t3 .

[0048] Based on the same inventive concept, a second aspect of the present invention provides a multi-branch pedestrian gait recognition device based on multi-frame rate and position coding, comprising:

[0049] The data preprocessing module is used to acquire the original dataset, crop and resize the images in the original dataset, and divide the dataset into training and test sets with non-overlapping identity labels. Then, probe sets and image libraries are further divided on the training and test sets respectively. The complete gait contour sequence is processed into three sets of input data with frame rates from low to high and resolutions from high to low according to different frame sampling strategies and resolution reduction operations. Specifically, the input data includes low frame rate high resolution input, medium frame rate medium resolution input, and high frame rate low resolution input.

[0050] The model building module is used to construct a deep learning-based multi-frame branch gait recognition model. The overall network architecture of the multi-frame branch gait recognition model is a three-branch structure. The first, second, and third branches handle low frame rate high-resolution input, medium frame rate medium-resolution input, and high frame rate low-resolution input, respectively. Each branch contains a position encoding module, a specific convolution module, a lateral feature set fusion connection, a lateral feature set fusion connection, a block convolution module, and a lateral feature set fusion connection. The specific convolution modules of the first, second, and third branches use 2D convolution, BP3D convolution, and 3D convolution, respectively. The features after the last lateral feature set fusion connection are then processed by a self-attention module, GeM pooling, a horizontal pyramid, a block fully connected layer, and BNNeck.

[0051] The model training module is used to train a multi-frame branch gait recognition model using the training set;

[0052] The gait recognition module is used to input the test set into the multi-frame branch gait recognition model trained on the test set, obtain the block gait features of the probe set and the image library on the test set, calculate the Euclidean distance between the block gait features of the probe set and the image library and sort them, and return the identity ID of the first ranked one as the recognition result.

[0053] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-branch pedestrian gait recognition method based on multi-frame rate and position coding described in the first aspect.

[0054] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-branch pedestrian gait recognition method based on multi-frame rate and position coding described in the first aspect.

[0055] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0056] 1. This invention innovatively performs temporal sampling and changes the spatial resolution of gait contour sequences, using three sets of gait contour sequences with different frame rates and resolutions as input gait representation methods for gait recognition tasks. Specifically, the first branch is low frame rate and high resolution, the second branch is medium frame rate and medium resolution, and the third branch is high frame rate and low resolution.

[0057] 2. This invention innovatively proposes a three-branch network structure, applying different convolutional networks to three sets of gait contour sequences with different frame rates and resolutions. One proposed method is BP3D convolution, which decomposes classic 3D convolution into spatial and temporal convolutions and then combines them. The low frame rate, high resolution branch extracts fine-grained spatial features of gait characteristics; the high frame rate, low resolution branch ensures effective capture of temporal features within the complete gait sequence cycle, while the low resolution reduces computational costs; the medium frame rate, medium resolution branch serves as a balance between the two, preserving spatial information and information from shorter time periods to some extent, and more importantly, acting as a bridge between the upper and lower branches.

[0058] 3. This invention innovatively proposes centralized feature fusion and decentralized feature fusion methods among the three branches, so as to gather features at different levels into different branches, achieve the effect of mutual propagation of features and sharing of learned features among different branches, and better utilize the unique information of the three sets of gait features at different frame rates and resolutions. Therefore, the multi-frame branch gait recognition model based on deep learning constructed by this invention can improve the accuracy of gait recognition. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 The network framework diagram of the multi-frame branch gait recognition model MGRF constructed in this invention is shown.

[0061] Figure 2 This is a schematic diagram of the composition of BP3D convolution in an embodiment of the present invention. Detailed Implementation

[0062] In existing technologies, the gait features used in training and testing of deep learning-based gait retrieval techniques are mostly based on gait templates or gait sequences. Among these, gait template-based methods compress temporal information through the summation and averaging operations during gait energy map calculation, making it difficult to effectively extract continuous temporal gait features between gait contour maps. Gait contour sequence-based methods, in order to reduce computational costs, typically use frame sampling operations during data loading, reducing the temporal dimension of the input sequence to a set number of frames. This process reduces the number of frames in the gait sequence to half or even less than the original. Therefore, how to preserve and extract fine-grained gait features from adjacent frames is crucial for subsequent tasks involving gait feature recognition or retrieval.

[0063] To improve the accuracy of gait recognition, this invention proposes a multi-branch pedestrian gait recognition method based on multi-frame rate and position coding, particularly by combining the utilization of the overall gait sequence and local details. The proposed method takes as input a multi-frame continuous gait contour sequence of the person being queried from the probe set of the test set, and outputs the person's identity ID identified in the test set gallery.

[0064] This invention constructs a deep learning-based multi-branch gait recognition model MGRF (Multi-branchGait Recognition for multi-Frame-rate), which uses position encoding, feature fusion, and attention mechanisms to extract features from pedestrian gait contour sequences. It is then optimized using a triplet loss function and a labeled smooth cross-entropy function to achieve cross-angle pedestrian identification.

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1

[0067] This invention discloses a multi-branch pedestrian gait recognition method based on multi-frame rate and position coding, including:

[0068] S1: Obtain the original dataset, crop and resize the images in the original dataset, and divide the dataset into a training set and a test set with mutually exclusive identity labels. Then, divide the training set and the test set into a probe set and an image library, respectively. Process the complete gait contour sequence into three sets of input data with frame rates from low to high and resolutions from high to low, according to different frame sampling strategies and resolution reduction operations. The input data specifically includes low frame rate high resolution input, medium frame rate medium resolution input, and high frame rate low resolution input.

[0069] S2: Construct a deep learning-based multi-frame branch gait recognition model. The overall network architecture of the multi-frame branch gait recognition model is a three-branch structure. The first, second, and third branches handle low frame rate high-resolution input, medium frame rate medium-resolution input, and high frame rate low-resolution input, respectively. Each branch contains a position encoding module, a specific convolution module, a lateral feature set fusion connection, a specific convolution module, a lateral feature set scatter fusion connection, a block convolution module, and a lateral feature set fusion connection. The specific convolution modules of the first, second, and third branches adopt two-dimensional convolution, BP3D convolution, and three-dimensional convolution, respectively. The features after the last lateral feature set fusion connection are then processed by a self-attention module, GeM pooling, a horizontal pyramid, a block fully connected layer, and BNNeck.

[0070] S3: Train the multi-frame branch gait recognition model using the training set;

[0071] S4: Input the test set into the multi-frame branch gait recognition model trained on the test set, obtain the block gait features of the probe set and the image library on the test set, calculate the Euclidean distance between the block gait features of the probe set and the image library and sort them, and return the identity ID of the first ranked one as the recognition result.

[0072] Overall, step S1 is data acquisition and preprocessing, S2 is model construction, S3 is model training, and S4 is model testing and application.

[0073] In a specific embodiment, the input data uses the publicly available gait dataset CASIA-B released by the Institute of Automation, Chinese Academy of Sciences as an example. CASIA-B is a large-scale multi-view gait dataset containing 124 individuals (IDs from 001 to 124). Each individual's gait data was collected from 11 perspectives (0, 18, 36, ..., 180°) and three clothing conditions (normal condition NM, wearing a coat CL, carrying a package BG). Each condition contains 6, 2, and 2 rounds of walking data, respectively. The dataset includes video and gait sequence formats.

[0074] The specific implementation methods of each step of the present invention are described in detail below.

[0075] The specific implementation steps of dataset preprocessing, frame sampling strategy, and resolution reduction in step S1 are as follows:

[0076] S1.1, Cropping and Resizing Preprocessing: For each gait contour image in the gait contour sequence, the top, bottom, left, and right edges of the human body are identified, and the image is cropped into a rectangle containing only the human body. The OpenCV library's `resize` function is then used to resize the image to a uniform height H0 and width W0. Several temporally consecutive gait contour images constitute a gait contour sequence. The complete gait contour sequence X0 dimension is represented as C0×T0×H0×W0, where C0 is the number of channels in each gait contour image, and T0 is the number of gait contour images contained in the gait contour sequence, i.e., the number of frames. Taking CASIA-B as an example, the complete gait contour sequence X0 dimension is 1×T0×64×44. Note that the training batch is omitted in the dimension representation.

[0077] Step S1.2: Divide the dataset into training and testing sets. Let there be M individuals with different IDs in the gait dataset, and let d... i Let i∈{1,2,…,M} represent the identity ID. Divide the training set and the test set into two disjoint sets. Let there be N people in the training set and MN people in the test set. Then the identity ID in the training set is represented as d. i For i∈{1,2,…,N}, the identity ID in the test set is represented as d. i , i∈{M-N+1,M-N+2,…,M}; Taking CASIA-B as an example, the gait data of 124 people in CASIA-B are divided into training set and test set, where the training set uses the first 74 people and the test set uses the last 50 people.

[0078] S1.3, Divide the dataset into probe set and image set. The dataset structure is as follows: each identity ID folder contains different clothing types (taking CASIA-B as an example, it includes three types: NM, BG, and CL, representing normal, backpack, and overcoat, respectively). Each clothing type contains several gait sequences from all angles (from 0° to 180°, in 18° intervals, for a total of 11 angles) across different rounds of shooting. The rounds under each type are then divided into the probe set and the test set according to a certain ratio. (The dataset folder contains several individual ID folders, each ID contains folders for different clothing types, and each type contains folders for various angles.)

[0079] S1.4, the topmost first branch, performs temporal sampling on the input complete gait contour sequence. If the temporal sampling interval is r, where r is an integer, usually a multiple of 2, then the temporal dimension of the first branch is reduced to r times the original value. 2 One-third, with height and width remaining unchanged, the gait profile sequence X of the first branch. f0 Dimensions In the example, r is 2;

[0080] S1.5, the second branch in the middle, performs temporal sampling and resolution reduction on the input complete gait contour sequence. The temporal dimension of the second branch is reduced to 1 / r of the original, and the height and width are also reduced to 1 / r of the original. The gait contour sequence X of the second branch... s0 Dimensions

[0081] S1.6, the bottom third branch, performs a resolution reduction operation on the input complete gait contour sequence, reducing the height and width of the third branch to r of the original value. 2 One-third, the gait contour sequence X of the third branch t0 Dimensions

[0082] S1.7, after frame sampling strategy and resolution downsampling, the three sets of input data X, with frame rate from low to high and resolution from high to low, are... f0 X s0 X t0 Input the three corresponding branches respectively, with subscripts f, s, and t representing the first, second, and third branches respectively.

[0083] Please see Figure 1 This is a network framework diagram of the multi-frame branch gait recognition model MGRF constructed in this invention.

[0084] Step S3 includes:

[0085] S3.1: Input three sets of data with frame rates from low to high and resolutions from high to low into the three branches of the three-branch network, respectively. After passing through the position encoding module, data with added height, width, and temporal coordinates are obtained. Input the three sets of data with added coordinates into their respective convolution modules, and use 2D convolution, BP3D, and 3D convolution respectively according to the frame rate from low to high to obtain their respective first-level gait features.

[0086] S3.2: The three sets of primary gait features are fused together, and the features of the upper and lower branches are converged into the middle branch. Specifically, the primary gait features with low frame rate and high resolution are spatially pooled, and the primary gait features with high frame rate and low resolution are spatially upsampled and temporally pooled, and then added element by element to the temporal position corresponding to the primary gait features with medium frame rate and medium resolution.

[0087] S3.3: The first-level gait features of the upper and lower branches and the fused features of the middle branch are input into their respective convolutional modules to obtain their respective second-level gait features; the three sets of second-level gait features are then fused in a feature-dispersed manner, specifically by adding the features of the middle branch to the upper and lower branches after performing different processing.

[0088] S3.4: Perform horizontal block convolution on the secondary gait features of the middle branch and the fusion features of the upper and lower branches according to different settings to obtain the tertiary gait features of the three branches;

[0089] S3.5: The three sets of three-level gait features are fused together. Specifically, the features of the upper and lower branches are converged into the middle branch to obtain the preliminary frame-level gait features; the preliminary frame-level gait features are processed by the self-attention module to obtain the enhanced frame-level gait features; and the enhanced frame-level gait features are then processed by GeM temporal pooling to converge the information in the time dimension to obtain the sequence-level gait features.

[0090] S3.6: The sequence-level gait features are passed through a horizontal pyramid pooling layer, and then horizontally segmented, shaped, and stitched together to obtain pyramid multi-scale features;

[0091] S3.7: The multi-scale features of the pyramid are mapped to each block through a block-based fully connected layer to obtain block gait features, which are used to calculate the triplet loss. The block gait features are then passed through a BNNeck layer to obtain classification probabilities, which are used together with the real identity labels to calculate the labeled smooth cross-entropy loss. During the training phase, the loss of the entire network is a weighted sum of the triplet loss and the labeled smooth cross-entropy loss, and is optimized using the AdamW optimizer.

[0092] The position encoding module and the three-branch convolution operation used in step S3.1 are as follows:

[0093] S3.1.1, position encoding is performed by absolute position encoding based on the height H0 and width W0 of the input gait data. The height and width values ​​generate floating-point coordinates with a range of [-1,1] and an interval of 2 / W0 and 2 / H0, respectively. After adding dimensions, they are concatenated at the end of the channel dimension. The three sets of gait sequence data with additional coordinates have 2 more dimensions than before in the channel dimension.

[0094] In the specific implementation process, absolute position encoding is performed based on the height and width of the input gait data. This is done by encoding the height H0 and width W0 of the input gait data, generating floating-point coordinates with a range of [-1, 1] and intervals of 2 / W0 and 2 / H0, respectively, from the height and width values. Taking CASIA-B as an example, the generated y-coordinate corresponding to a height of 64 is divided into 64 equal intervals from [-1, 1], tiled along the x-axis perpendicular to the y-coordinate, and then repeated along the time and channel dimensions. Similarly, the generated x-coordinate corresponding to a width of 44 is divided into 44 equal intervals from [-1, 1], tiled along the y-axis perpendicular to the x-coordinate, and then repeated along the time and channel dimensions. This results in absolute position coordinates with 2 channels and time, height, and width consistent with the input.

[0095] S3.1.2, the absolute position coordinates are concatenated to the end of the input channel dimension to obtain three sets of gait sequence data with additional coordinates;

[0096] S3.1.3, the first branch pairs the low frame rate, high resolution gait contour sequence X. f0 Two-dimensional convolution is performed and then activated by the LeakyReLU function to obtain local details in its spatial form, resulting in the first-branch first-level gait feature X. f1 =Conv 3×3 (X f0 In a specific embodiment, the step size and padding are set to 1, the number of channels C1 = 32, and the output gait feature X is... f1 Dimension That is, 32×T0 / 4×64×44;

[0097] S3.1.4, Second branch aligns with frame rate mid-resolution gait contour sequence X s0 BP3D convolution is performed in two paths. First, a 2D convolution is performed on the height and width planes, followed by a convolution on the temporal dimension. The other path performs a 3D convolution on the input (stride and padding are set to 1, and the number of channels C1 = 32). The results of the two convolutions are summed element-wise and then passed through the Leaky ReLU activation function to obtain its spatial and temporal features, resulting in the first-level gait features of the second branch, denoted as X. s1 =Conv3d 3×3×3 (X 30 )+Conv3d 1×3×3 (X s0 )+Conv3d 3×1×1 (X s0 Output gait features X s1 Dimension That is

[0098] S3.1.5, the third branch pairs high frame rate low resolution gait contour sequences X. t0 A global 3D convolution is performed with stride and padding set to 1, and the number of channels C1 = 32. The convolution is then applied using the Leaky ReLU activation function, focusing on the global motion pattern in the temporal direction to obtain the third-branch first-level gait feature X. t1 =Conv3d 3×3×3 (X t0 Output gait features X t1 Dimension That is, 32×T0×16×11.

[0099] The BP3D convolution used in S3.1.4, its composition can be found in [link to relevant documentation]. Figure 2 .

[0100] The specific calculation steps for feature set fusion in step S4 are as follows:

[0101] Step S3.2.1, for the first branch, low frame rate, high resolution first-level gait features X f1 Spatial pooling is performed, specifically max pooling, reducing the width and height to one-rth of their original values, so that they are related to the x-th value of the second branch. s1 The height and width are the same, and then they are added together according to the time and position coordinates to the first-level gait feature x of the second branch. s1 At the corresponding time position;

[0102] Step S3.2.2, for the third branch high frame rate low resolution first-level gait features x t1 In spatial terms, an upsampling operation is performed, increasing the width and height to r times their original values ​​(r = 2), which corresponds to the X value of the second branch. s1 With consistent height and width, max pooling is then performed on several adjacent frames in the time dimension, reducing the number of frames to r times the original value. These frames are then added together according to their temporal position coordinates to the first-level gait feature X in the second branch. s1 At the corresponding time position;

[0103] Step S3.2.3: After changing the temporal frame number and spatial height and width of the features of the upper and lower branches, they are combined with the first-level gait features x of the second branch. s1 Addition, i.e., the feature x after centralized fusion p1 =x s1 MaxPooling r×r (x f1 )+MaxPooling r×1×1 (UpSample r×r (X t1 )).

[0104] The specific steps of the convolution and feature dispersion fusion in the three branches of step S3.3 are as follows:

[0105] S3.3.1, for the first branch, low frame rate, high resolution first-level gait feature X f1 Continuing with the two-dimensional convolution and other processing described in step S3.1.3, the first branch secondary gait feature X is obtained. f2 =Conv 3×3 (X f1 );

[0106] S3.3.2, for the concentrated fusion of frame rate and resolution features X in the second branch p1 Continue with the BP3D convolution processing as described in step S3.1.4 to obtain the second branch secondary gait feature X. s2=Conv3d 3×3×3 (X p1 )+Conv3d 1 ×3×3 (X p1 )+Conv3d 3×1×1 (X p1 );

[0107] S3.3.3, for the first-level gait feature X of the third branch with high frame rate and low resolution. t1 Continue with the three-dimensional convolution and other processing described in step S3.1.5 to obtain the third branch secondary gait feature X. t2 =Conv3d 3×3×3 (X t1 ).

[0108] S3.3.4, for the second branch secondary gait feature X s2 In spatial terms, an upsampling operation is performed, increasing the width and height to r times their original values, which is the same as the X value of the first branch. f2 With consistent height and width, max pooling is then performed on several adjacent frames in the time dimension, reducing the number of frames to one-rth of the original. The frames are then added to the second-level gait feature X of the first branch according to their temporal position coordinates. f2 At the corresponding time position;

[0109] S3.3.5, for the second branch secondary gait feature X s2 Spatial pooling is performed, specifically max pooling, reducing the width and height to one-rth of their original values, so that they are related to the third branch X. t2 The height and width are the same, and then they are added together according to the time and position coordinates to the secondary gait feature X of the third branch. t2 At the corresponding time position;

[0110] S3.3.6, Secondary gait features X of the first branch f2 The second branch secondary gait features X after upsampling and temporal pooling s2 Adding them together yields the first branch fusion feature X. q1 =X f2 MaxPooling r×1×1 (UpSample r×r (X s2 )).

[0111] S3.3.7, Secondary gait features X of the third branch t2 The second branch secondary gait feature X after spatial max pooling s2 Adding them together yields the third branch fusion feature X. q2 =X t2 MaxPooling r×r(X s2 ).

[0112] The block convolution operation in step S3.4, assuming the cardinality of the number of blocks is b (4 in this example), has the following specific calculation steps and partitioning logic:

[0113] S3.4.1, for the fusion feature X of the first branch q1 Divide its height into b along the horizontal direction. 2 The first branch's third-level feature X is obtained by performing a 3D convolution on each small block and activating it with Leaky ReLU, and then concatenating the results of each block along the height direction. f3 Since the first branch has the highest resolution and its features are fine-grained, it is divided into the most blocks to focus more on the local fine details of the space.

[0114] S3.4.2, for the second branch secondary gait feature X s2 The height is divided into b blocks along the horizontal direction. A 3D convolution is performed on each block and activated by Leaky ReLU. The results of each block are then concatenated along the height direction to obtain the second branch's third-level feature X. s3 The second branch features encompass both a certain temporal span and a certain spatial range, and are therefore divided into larger parts to focus on the feature changes of a certain region over a shorter interval.

[0115] S3.4.3, for the fusion feature X of the third branch q2 Without any partitioning, a 3D convolution is performed on the entire feature and activated by Leaky ReLU to obtain the third branch tertiary feature X. t3 The third branch features have the highest temporal frame rate and are better able to extract the motion patterns of the entire gait cycle, so they are not divided to focus on the global temporal motion patterns.

[0116] The specific steps of step S3.5 are as follows:

[0117] S3.5.1, the feature fusion calculation is basically the same as in step S3.2 above, and the fused feature X p2 =X s3 MaxPooling r×r (X f3 )+MaxPooling r×1×1 (UpSample r×r (X t3 That is, the preliminary frame-level gait features extracted by the network;

[0118] S3.5.2, Preliminary frame-level gait features X p2 Enhanced frame-level gait features X are obtained after processing by a self-attention module.at =C out ×T out ×H out ×W out , where C out T out H out W out These represent the number of output channels, the number of frames, the height, and the width, respectively.

[0119] S3.5.3, Temporal GeM pooling operation, refers to calculating u to the power of time, then performing average pooling, and finally taking 1 / u to obtain the sequence-level gait feature X. seq =C out ×H out ×W out The formula for calculating sequential GeM pooling is:

[0120]

[0121] The specific steps of the horizontal pyramid pooling layer in step S3.6 are as follows:

[0122] S3.6.1, the sequence-level gait features X seq Divide the height into 1, 2, 4, 8, and 16 blocks horizontally, with each block having dimensions of [dimensions missing]. Where b represents the number of blocks at different levels;

[0123] S3.6.2, each X block block The pyramid multi-scale feature X is output by reshaping and then performing a 1×1 convolution that only changes the number of channels. py =C py ×B, where The sum of all the number of blocks is 31.

[0124] The specific calculation steps for the block-based fully connected layer, triplet loss, BNNeck layer, labeled smoothing cross-entropy loss, and total training loss in step S3.7 are as follows:

[0125] In S3.7.1, the block-based fully connected layer maps each block of the pyramid's multi-scale features separately to obtain the block-based gait features X. fc =C fc ×B;

[0126] S3.7.2, Calculate the triplet loss L tr: A batch is formed by randomly sampling P categories (personal identification IDs), and then K gait profile images are randomly selected from a certain gait sequence of each category (identity ID), resulting in a batch of PK images; if the batch size is set to 32, then P = 4, K = 8; for each sample a in the batch, when forming triples, positive samples p and negative samples n in the batch are selected, and the triplet loss L tr The calculation is as follows:

[0127]

[0128] Where, x i k The image corresponds to the k-th gait profile of the i-th person in this batch, where m represents the boundary margin value, and f(x) i k ) represents the k-th feature of sample i, that is, the feature corresponding to the k-th gait contour image of the i-th person in this batch. D(·) represents the distance between two sample features, D(f(x) i a ),f(x i p )) represents the distance between different gait profiles of the same person, i.e., the intra-class distance with the positive sample p, D(f(x) i a ),f(x j n The distance between samples from different individuals is represented by ), which is the inter-class distance with the negative sample n. The distance metric used is Euclidean distance.

[0129] It should be noted that a gait profile sequence consists of several gait profile images, and a sample usually refers to a gait profile sequence. i represents a gait profile sequence belonging to the i-th person, and k represents the k-th gait profile image in a given gait profile sequence. In the formula, j≠i, representing samples from different people; p≠a, representing other positive samples p that are different from sample a.

[0130] S3.7.3, the BNNeck layer includes a BatchNorm layer and a fully connected layer. The number of output channels of the fully connected layer is the total number of categories, that is, the total number of identity IDs. In CASIA-B, it is set to 74. Finally, it is activated by Sigmoid to obtain the classification probability.

[0131] S3.7.4, calculate the labeled smoothed cross-entropy loss L by combining the classification probability with the real identity label. ce The calculation formula is:

[0132]

[0133] Where y represents the real identity label, p iLet q represent the log probability of the predicted identity for the i-th class. i The definition is as follows:

[0134]

[0135] Where N is the total number of people in the training set, and ε is a small parameter used to reduce the model's confidence in the training set and thus alleviate the overfitting problem.

[0136] S3.7.5, the loss of the entire network during the training phase is defined as:

[0137] L gait =L tr +λL ce

[0138] Where λ is the weight hyperparameter of the labeled smooth cross-entropy loss, set to 0.1.

[0139] In the training phase of step S3, the block gait features of the gait sequences of the probe set and image library on the training set are calculated using the weights of the trained network model. The Euclidean distance is calculated, and the evaluation metrics mAP and Rank-1 Accuracy are calculated based on the distance ranking results. If the evaluation metrics are the current best, the network model weights are saved.

[0140] In the testing phase of step S4, the saved network model weights are used to obtain the block gait features of the probe set and the image library on the test set, calculate their Euclidean distance, and calculate the evaluation metrics mAP and Rank-1Accuracy based on their distance ranking results. The identity ID ranked first is then returned as the recognition result.

[0141] In summary, this method constructs a deep learning-based multi-frame branch gait recognition model, MGRF, using pedestrian gait contour sequence data. Based on frame sampling, position encoding, feature fusion, attention mechanism, and multi-branch structure, it achieves feature extraction of pedestrian gait contour sequences. Furthermore, it optimizes the model through a triplet loss function and a labeled smooth cross-entropy function to achieve pedestrian gait recognition and improve retrieval accuracy.

[0142] Example 2

[0143] Based on the same inventive concept, this embodiment discloses a multi-branch pedestrian gait recognition device based on multi-frame rate and position coding, comprising:

[0144] The data preprocessing module is used to acquire the original dataset, crop and resize the images in the original dataset, and divide the dataset into training and test sets with non-overlapping identity labels. Then, probe sets and image libraries are further divided on the training and test sets respectively. The complete gait contour sequence is processed into three sets of input data with frame rates from low to high and resolutions from high to low according to different frame sampling strategies and resolution reduction operations. Specifically, the input data includes low frame rate high resolution input, medium frame rate medium resolution input, and high frame rate low resolution input.

[0145] The model building module is used to construct a deep learning-based multi-frame branch gait recognition model. The overall network architecture of the multi-frame branch gait recognition model is a three-branch structure. The first, second, and third branches handle low frame rate high-resolution input, medium frame rate medium-resolution input, and high frame rate low-resolution input, respectively. Each branch contains a position encoding module, a specific convolution module, a lateral feature set fusion connection, a lateral feature set fusion connection, a block convolution module, and a lateral feature set fusion connection. The specific convolution modules of the first, second, and third branches use 2D convolution, BP3D convolution, and 3D convolution, respectively. The features after the last lateral feature set fusion connection are then processed by a self-attention module, GeM pooling, a horizontal pyramid, a block fully connected layer, and BNNeck.

[0146] The model training module is used to train a multi-frame branch gait recognition model using the training set;

[0147] The gait recognition module is used to input the test set into the multi-frame branch gait recognition model trained on the test set, obtain the block gait features of the probe set and the image library on the test set, calculate the Euclidean distance between the block gait features of the probe set and the image library and sort them, and return the identity ID of the first ranked one as the recognition result.

[0148] Since the apparatus described in Embodiment 2 of this invention is the same apparatus used to implement the multi-branch pedestrian gait recognition method based on multi-frame rate and position coding in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this apparatus based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All apparatuses used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0149] Example 3

[0150] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0151] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the multi-branch pedestrian gait recognition method based on multi-frame rate and position coding in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0152] Example 4

[0153] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in Embodiment 1.

[0154] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the multi-branch pedestrian gait recognition method based on multi-frame rate and position coding in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0155] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A multi-branch pedestrian gait recognition method based on multi-frame rate and position coding, characterized in that, include: S1: Obtain the original dataset, crop and resize the images in the original dataset, and divide the dataset into a training set and a test set with mutually exclusive identity labels. Then, divide the training set and the test set into a probe set and an image library, respectively. The complete gait contour sequence is processed into three sets of input data with frame rates from low to high and resolutions from high to low according to different frame sampling strategies and down-resolution operations. Specifically, the input data includes low frame rate high resolution input, medium frame rate medium resolution input, and high frame rate low resolution input. S2: Construct a deep learning-based multi-frame branch gait recognition model. The overall network architecture of the multi-frame branch gait recognition model is a three-branch structure. The first, second, and third branches handle low frame rate high-resolution input, medium frame rate medium-resolution input, and high frame rate low-resolution input, respectively. Each branch contains a position encoding module, a specific convolution module, a lateral feature set fusion connection, a lateral feature set fusion connection, a block convolution module, and a lateral feature set fusion connection. The specific convolution modules of the first, second, and third branches adopt two-dimensional convolution, BP3D convolution, and three-dimensional convolution, respectively. The features after the last lateral feature set fusion connection are then processed by a self-attention module, GeM pooling, a horizontal pyramid, a block fully connected layer, and BNNeck. BP3D convolution refers to decomposing the classic three-dimensional convolution into spatial and temporal convolutions and then combining them. S3: Train the multi-frame branch gait recognition model using the training set; S4: Input the test set into the multi-frame branch gait recognition model trained on the test set, obtain the block gait features of the probe set and the image library on the test set, calculate the Euclidean distance between the block gait features of the probe set and the image library and sort them, and return the identity ID of the first ranked one as the recognition result.

2. The multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in claim 1, characterized in that, Step S1 includes: S1.1: Crop and resize the images, specifically including: for each gait contour image in the gait contour sequence, identify the top, bottom, left, and right edges of the human body, crop them into rectangles containing only the human body, and resize the images to a uniform height. H 0 and width W 0; S1.2: Divide the training set and the test set into two disjoint sets; S1.3: Divide the dataset into probe set and image set. The dataset structure is as follows: each identity ID folder contains different clothing types, and each clothing type contains several gait sequences from all angles in different rounds of shooting; divide the rounds in each type into probe set and test set according to a certain ratio. S1.4: The complete gait contour sequence is processed according to different frame sampling strategies and resolution reduction operations, resulting in three sets of input data with frame rates ranging from low to high and resolutions from high to low. , , These are used as inputs to the three branches, where the indices... f , s , t These represent the first branch, the second branch, and the third branch, respectively.

3. The multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in claim 1, characterized in that, Step S3 includes: S3.1: Input three sets of data with frame rates from low to high and resolutions from high to low into the three branches of the three-branch network, respectively. After passing through the position encoding module, data with added height, width, and temporal coordinates are obtained. Input the three sets of data with added coordinates into their respective convolution modules, and use 2D convolution, BP3D, and 3D convolution respectively according to the frame rate from low to high to obtain their respective first-level gait features. S3.2: The three sets of primary gait features are fused together, and the features of the upper and lower branches are converged into the middle branch. Specifically, the primary gait features with low frame rate and high resolution are spatially pooled, and the primary gait features with high frame rate and low resolution are spatially upsampled and temporally pooled, and then added element by element to the temporal position corresponding to the primary gait features with medium frame rate and medium resolution. S3.3: The first-level gait features of the upper and lower branches and the fused features of the middle branch are input into their respective convolutional modules to obtain their respective second-level gait features; the three sets of second-level gait features are then fused in a feature-dispersed manner, specifically by adding the features of the middle branch to the upper and lower branches after performing different processing. S3.4: Perform horizontal block convolution on the secondary gait features of the middle branch and the fusion features of the upper and lower branches according to different settings to obtain the tertiary gait features of the three branches; S3.5: The three sets of three-level gait features are fused together. Specifically, the features of the upper and lower branches are converged into the middle branch to obtain the preliminary frame-level gait features; the preliminary frame-level gait features are processed by the self-attention module to obtain the enhanced frame-level gait features; and the enhanced frame-level gait features are then processed by GeM temporal pooling to converge the information in the time dimension to obtain the sequence-level gait features. S3.6: The sequence-level gait features are passed through a horizontal pyramid pooling layer, and then horizontally segmented, shaped, and stitched together to obtain pyramid multi-scale features; S3.7: The multi-scale features of the pyramid are mapped to each block through a block-based fully connected layer to obtain block gait features, which are used to calculate the triplet loss. The block gait features are then passed through a BNNeck layer to obtain classification probabilities, which are used together with the real identity labels to calculate the labeled smooth cross-entropy loss. During the training phase, the loss of the entire network is a weighted sum of the triplet loss and the labeled smooth cross-entropy loss, and is optimized using the AdamW optimizer.

4. The multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in claim 3, characterized in that, Step S3.1 includes: S3.1.1: The position encoding module determines the height of the input data. H 0 and width W 0 performs absolute position encoding, generating values ​​ranging from [-1, 1] with intervals of 2 / 0 based on the height and width values. W 0 and 2 / H The floating-point coordinates of 0 are appended to the end of the channel dimension after the dimension is increased. The dimension of the three sets of gait sequence data with the additional coordinates is increased by 2 compared to the previous channel dimension. S3.1.2: Concatenate the absolute position coordinates to the end of the input channel dimension to obtain three sets of gait sequence data with additional coordinates; S3.1.3: First branch for low frame rate high resolution gait contour sequences Two-dimensional convolution is performed and then activated by the Leaky ReLU function to obtain local details in the space, thus obtaining the first-level gait features of the first branch. ; S3.1.4: Second branch mid-frame rate medium-resolution gait contour sequence BP3D convolution is performed in two paths. First, a two-dimensional convolution is performed on the height and width planes, followed by a convolution on the temporal dimension. The other path performs a three-dimensional convolution on the input. The results of the two convolutions are element-wise summed and then passed through the Leaky ReLU activation function to obtain its spatial and temporal features, resulting in the second branch of first-level gait features. , is represented as: ; S3.1.5: Third branch for high frame rate, low resolution gait contour sequences Perform global 3D convolution followed by LeakyReLU activation to focus on global motion patterns in the temporal direction, thus obtaining the third branch of first-level gait features. .

5. The multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in claim 4, characterized in that, Step S3.2 includes: S3.2.1: First-level gait features of the first branch Spatial pooling is performed, using max pooling to reduce the width and height to their original values. r One-third, making it related to the second branch. The height and width are consistent, and then they are added together according to the time and position coordinates to the first-level gait feature of the second branch. At the corresponding time position; S3.2.2: First-level gait features of the third branch Perform an upsampling operation in space, increasing the width and height to their original values. r times, with the second branch With the height and width consistent, max pooling is then performed on several adjacent frames over time to reduce the frame count back to the original value. r One-third, added to the first-level gait features of the second branch according to the time and position coordinates. At the corresponding time position; S3.2.3: After changing the temporal frame number and spatial height and width, the features of the upper and lower branches are integrated with the first-level gait features of the second branch. Adding them together yields the characteristics after fusion. , is represented as: .

6. The multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in claim 3, characterized in that, Step S3.3 includes: S3.3.1: First-level gait features for the first branch Continue with the two-dimensional convolution and Leaky ReLU activation function processing in S3.1.3 to obtain the first branch of secondary gait features. ; S3.3.2: Features after centralized fusion of the second branch Continue with the BP3D convolution processing in S3.1.4 to obtain the second branch secondary gait features. , is represented as: ; S3.3.3: First-level gait features for the third branch Continue with the 3D convolution and Leaky ReLU activation function processing in S3.1.5 to obtain the third branch of secondary gait features. , is represented as: ; S3.3.4: For the second branch of secondary gait features This involves upsampling in space, increasing the width and height to their original values. r times, compared to the first branch With the height and width consistent, max pooling is then performed on several adjacent frames over time to reduce the frame count back to the original value. r One-third, summed according to time and position coordinates, is added to the secondary gait features of the first branch. At the corresponding time position; S3.3.5: For the second branch of secondary gait features Spatial pooling is performed, and max pooling is used to reduce the width and height to their original values. r One-third, making it related to the third branch The height and width are consistent, and then they are added together according to the time and position coordinates to the secondary gait features of the third branch. At the corresponding time position; S3.3.6: Secondary gait features of the first branch Compared with the second branch secondary gait features after upsampling and temporal pooling Adding them together yields the first branch fusion feature. ; S3.3.7: Secondary gait characteristics of the third branch Second-order gait features of the second branch after spatial max pooling Adding them together yields the third branch fusion feature. .

7. The multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in claim 3, characterized in that, Step S3.4 includes: S3.4.1: Fusion features for the first branch Divide its height into horizontal sections. b 2 The first branch of the three-level features is obtained by performing a 3D convolution on each small block and activating it with Leaky ReLU, and then concatenating the results of each block along the height direction. , b The base number of blocks; S3.4.2: For the second branch of secondary gait features Divide its height into horizontal sections. b The second branch of tertiary features is obtained by performing 3D convolution on each small block and activating it with Leaky ReLU, and then concatenating the results of each block along the height direction. ; S3.4.3: Fusion characteristics for the third branch Without any partitioning, a 3D convolution is performed on the entire feature set and activated by Leaky ReLU to obtain the third branch of tertiary features. .

8. A multi-branch pedestrian gait recognition device based on multi-frame rate and position coding, characterized in that, include: The data preprocessing module is used to obtain the original dataset, crop and resize the images in the original dataset, and divide the dataset into a training set and a test set with mutually exclusive identity labels. Then, probe sets and image libraries are further divided on the training set and the test set, respectively. The complete gait contour sequence is processed into three sets of input data with frame rates from low to high and resolutions from high to low according to different frame sampling strategies and down-resolution operations. Specifically, the input data includes low frame rate high resolution input, medium frame rate medium resolution input, and high frame rate low resolution input. The model building module is used to construct a deep learning-based multi-frame branch gait recognition model. The overall network architecture of the multi-frame branch gait recognition model is a three-branch structure. The first, second, and third branches handle low frame rate high-resolution input, medium frame rate medium-resolution input, and high frame rate low-resolution input, respectively. Each branch contains a position encoding module, a specific convolution module, a lateral feature set fusion connection, a lateral feature set fusion connection, a block convolution module, and a lateral feature set fusion connection. The specific convolution modules of the first, second, and third branches adopt two-dimensional convolution, BP3D convolution, and three-dimensional convolution, respectively. The features after the last lateral feature set fusion connection are then processed by a self-attention module, GeM pooling, a horizontal pyramid, a block fully connected layer, and BNNeck. Among them, BP3D convolution refers to decomposing the classic three-dimensional convolution into spatial and temporal convolutions and then combining them. The model training module is used to train a multi-frame branch gait recognition model using the training set; The gait recognition module is used to input the test set into the multi-frame branch gait recognition model trained on the test set, obtain the block gait features of the probe set and the image library on the test set, calculate the Euclidean distance between the block gait features of the probe set and the image library and sort them, and return the identity ID of the first ranked one as the recognition result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-branch pedestrian gait recognition method based on multi-frame rate and position coding as described in any one of claims 1 to 7.

Images