Gait recognition method

Abstract

The application is suitable for the technical field of identity authentication and recognition, and provides a gait recognition method, which comprises the following steps: acquiring a to-be-recognized gait image sequence of a target object, wherein the to-be-recognized gait image sequence comprises a plurality of to-be-recognized gait images; inputting the to-be-recognized gait image sequence into an improved GaitPart model for gait recognition to obtain a gait recognition result of the target object; the improved GaitPart model comprises a comprehensive feature extraction module for extracting features of the to-be-recognized gait image sequence, a global time feature aggregation module for aggregating global time features, a horizontal pooling layer, a local time feature aggregation module for aggregating local time features, and a superposition module for outputting the gait recognition result of the target object. The application can improve the accuracy of gait recognition.

Description

Technical Field

[0001] This application belongs to the field of identity verification and recognition technology, and in particular relates to a gait recognition method. Background Technology

[0002] With social development and scientific and technological advancements, various biometric identification technologies have matured and are widely applied in smart cities, smart transportation, and smart security. Common biometric identification technologies based on facial recognition, iris scanning, and fingerprints are limited to close-range identification processes, requiring active cooperation from the subject. Furthermore, because these features are innate, they are easily stolen and forged. In contrast, gait recognition offers significant advantages. Gait recognition aims to identify individuals through their walking posture, offering non-contact, long-range, and difficult-to-fake characteristics, enabling seamless identification during walking. However, in complex and varied real-world scenarios, factors such as carrying a bag, wearing a coat, and different camera angles can cause significant variations in a pedestrian's gait appearance, posing challenges to gait recognition.

[0003] Traditional gait recognition methods mostly extract gait features on the whole human body. However, during people's walking, different parts of the body will show different states, and the same part will also show different states at different times. Although the gait recognition model GaitPart has taken this problem into account, it focuses on extracting local features and ignores global features, resulting in low accuracy of gait recognition. Summary of the Invention

[0004] This application provides a gait recognition method that can solve the problem of low accuracy in gait recognition.

[0005] This application provides a gait recognition method, including:

[0006] Obtain the gait image sequence of the target object to be identified, which includes multiple gait images to be identified;

[0007] The gait image sequence to be identified is input into the improved GaitPart model for gait recognition, and the gait recognition result of the target object is obtained.

[0008] The improved GaitPart model includes a comprehensive feature extraction module for extracting features from the gait image sequence to be recognized, a global temporal feature aggregation module for aggregating global temporal features, a horizontal pooling layer, a local temporal feature aggregation module for aggregating local temporal features, and an overlay module for outputting the gait recognition result of the target object. The output of the comprehensive feature extraction module is connected to the input of the global temporal feature aggregation module and the input of the horizontal pooling layer, respectively. The output of the horizontal pooling layer is connected to the input of the local temporal feature aggregation module. The outputs of the global temporal feature aggregation module and the local temporal feature aggregation module are both connected to the input of the overlay module.

[0009] Optionally, the comprehensive feature extraction module includes: a first convolutional module, a first max pooling layer, a second convolutional module, a second max pooling layer, and a third convolutional module connected in sequence, wherein the input of the first convolutional module is the input of the comprehensive feature extraction module, and the output of the third convolutional module is the output of the comprehensive feature extraction module;

[0010] The first convolution module, the second convolution module, and the third convolution module each include a first comprehensive convolution submodule and a second comprehensive convolution submodule connected in sequence;

[0011] Both the first and second comprehensive convolution submodules include: vertical block convolution units, horizontal block convolution units, ordinary convolution units, and a first stacking unit. The outputs of the vertical block convolution units, the horizontal block convolution units, and the ordinary convolution units are all connected to the input of the first stacking unit. The output of the first stacking unit of the first comprehensive convolution submodule is connected to the inputs of the vertical block convolution units, the horizontal block convolution units, and the ordinary convolution units of the second comprehensive convolution submodule, respectively.

[0012] In the first convolution module, the input terminals of the vertical block convolution units, horizontal block convolution units, and ordinary convolution units of the first comprehensive convolution submodule are the input terminals of the first convolution module. In the first convolution module, the output terminal of the first stacking unit of the second comprehensive convolution submodule is the output terminal of the first convolution module. In the second convolution module, the input terminals of the vertical block convolution units, horizontal block convolution units, and ordinary convolution units of the first comprehensive convolution submodule are the input terminals of the second convolution module. In the second convolution module, the output terminal of the first stacking unit of the second comprehensive convolution submodule is the output terminal of the second convolution module. In the third convolution module, the input terminals of the vertical block convolution units, horizontal block convolution units, and ordinary convolution units of the first comprehensive convolution submodule are the input terminals of the third convolution module. In the third convolution module, the output terminal of the first stacking unit of the second comprehensive convolution submodule is the output terminal of the third convolution module.

[0013] Optionally, the vertical block convolutional unit includes: a vertical cutting function, multiple first convolutional layers and a first cascaded layer, wherein the output of each of the multiple first convolutional layers is connected to the input of the first cascaded layer, and the output of the first cascaded layer is connected to the input of the first stacking unit.

[0014] The vertical cut function is used to vertically divide the data input to the vertical block convolutional unit into multiple sub-data blocks, and input the multiple sub-data blocks one by one into multiple first convolutional layers.

[0015] Optionally, the horizontal block convolutional unit includes: a horizontal cutting function, multiple second convolutional layers and a second cascaded layer, wherein the output of each of the multiple second convolutional layers is connected to the input of the second cascaded layer, and the output of the second cascaded layer is connected to the input of the first stacking unit;

[0016] The horizontal slicing function is used to horizontally divide the data input to the horizontal block convolutional unit into multiple sub-data blocks, and input these sub-data blocks one-to-one into multiple second convolutional layers.

[0017] Optionally, the ordinary convolutional unit includes: a third convolutional layer, the input of which is the input of the ordinary convolutional unit, and the output of which is connected to the input of the first stacking unit.

[0018] Optionally, the global temporal feature aggregation module includes: a first aggregation unit, a second aggregation unit, and a second overlay unit. The input ends of the first aggregation unit and the second aggregation unit are both connected to the output end of the comprehensive feature extraction module. The output ends of the first aggregation unit and the second aggregation unit are both connected to the input end of the second overlay unit. The output end of the second overlay unit is connected to the input end of the overlay module.

[0019] Optionally, both the first aggregation unit and the second aggregation unit include: a first three-dimensional convolutional layer, a ReLU activation function, a second three-dimensional convolutional layer, and a Sigmoid activation function connected in sequence; a three-dimensional max pooling layer, a three-dimensional average pooling layer, a stacking layer, and a multiplication layer;

[0020] The inputs of the first three-dimensional convolutional layer, the three-dimensional max pooling layer, and the three-dimensional average pooling layer are all connected to the output of the comprehensive feature extraction module. The outputs of the three-dimensional max pooling layer and the three-dimensional average pooling layer are all connected to the input of the stacked layer. The output of the stacked layer and the output of the Sigmoid activation function are all connected to the input of the product layer.

[0021] The output terminals of the product layer of the first aggregation unit and the product layer of the second aggregation unit are both connected to the input terminal of the second superposition unit.

[0022] The above-mentioned solution in this application has the following beneficial effects:

[0023] In the embodiments of this application, when performing gait recognition based on the improved GaitPart model, the comprehensive feature extraction module of the improved GaitPart model extracts local and global features of the gait image sequence to be recognized. The global temporal feature aggregation module aggregates adjacent gait frame features into global motion features, and the local temporal feature aggregation module aggregates adjacent gait frame features into local temporal features. The overlay module sums the global motion features and local temporal features and outputs the gait recognition result. Since the comprehensive feature extraction module extracts both local and global features of the gait image sequence to be recognized, and the final gait recognition result is a fusion of global motion features and local temporal features, the improved GaitPart model extracts richer gait features compared to the traditional GaitPart model, thus improving the accuracy of gait recognition.

[0024] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a gait recognition method provided in an embodiment of this application;

[0027] Figure 2 A schematic diagram of the structure of an improved GaitPart model provided in an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of the structure of the integrated convolution submodule in the integrated feature extraction module provided in an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the structure of a global time feature aggregation module provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the structure of a local time feature aggregation module provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of the structure of an MTB provided in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram illustrating the operation of the triplet loss function provided in an embodiment of this application. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0034] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0035] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0037] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0038] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0039] To address the low accuracy of current gait recognition methods, this application provides a gait recognition method using an improved GaitPart model. During gait recognition, the improved GaitPart model's comprehensive feature extraction module extracts local and global features from the gait image sequence to be recognized. A global temporal feature aggregation module aggregates adjacent gait frame features into global motion features, and a local temporal feature aggregation module aggregates adjacent gait frame features into local temporal features. An overlay module sums the global motion features and local temporal features and outputs the gait recognition result. Because the comprehensive feature extraction module extracts both local and global features from the gait image sequence to be recognized, and the final gait recognition result is a fusion of global motion features and local temporal features, the improved GaitPart model extracts richer gait features compared to the traditional GaitPart model, thus improving the accuracy of gait recognition.

[0040] The gait recognition method provided in this application will be described below with reference to specific embodiments.

[0041] like Figure 1 As shown, the gait recognition method provided in this application includes the following steps:

[0042] Step 11: Obtain the gait image sequence of the target object to be identified, which includes multiple gait images to be identified.

[0043] The target object can be a person whose gait needs to be recognized, and the sequence of gait images to be recognized for the target object includes multiple gait images of the target object. Generally, these multiple gait images to be recognized can be multiple consecutive images from a video.

[0044] For ease of description, the above sequence of gait images to be identified can be denoted as a gait sequence x1, x2, ..., x t x1 represents the first gait image to be identified, x2 represents the second gait image to be identified, and x... t Let t represent the t-th gait image to be identified, where t represents the number of gait images to be identified.

[0045] Step 12: Input the gait image sequence to be identified into the improved GaitPart model for gait recognition to obtain the gait recognition result of the target object.

[0046] In some embodiments of this application, to reduce the complexity of data processing, the gait image sequence to be identified can optionally be preprocessed before being input into the improved GaitPart model for gait recognition. Since only the human gait contour is a valid region in the image, the invalid region around the human contour can be removed by preprocessing, while retaining the main gait features.

[0047] Furthermore, observations show that the height of the human gait silhouette is longer than its width; that is, a rectangular silhouette is more consistent with the human body silhouette in real life than a square silhouette. Therefore, the image needs to be cropped twice before being input into the gait recognition model (i.e., the improved GaitPart model). After cropping, the final image size can be 64×48.

[0048] In practice, the human body contour is separated by finding the maximum height (top index), minimum height (bottom index), minimum width (leftmost index of the contour), and maximum width (rightmost index of the contour) of the person in the image.

[0049] In some embodiments of this application, such as Figure 2 As shown, the improved GaitPart model includes: a comprehensive feature extraction module (CFE) for extracting features from the gait image sequence to be identified, a global temporal feature aggregation module (GTFA) for aggregating global temporal features, a horizontal pooling layer, a local temporal feature aggregation module (LTFA) for aggregating local temporal features, and an overlay module for outputting the gait recognition result of the target object.

[0050] The output of the comprehensive feature extraction module is connected to the input of the global temporal feature aggregation module and the input of the horizontal pooling layer, respectively. The output of the horizontal pooling layer is connected to the input of the local temporal feature aggregation module. The outputs of both the global and local temporal feature aggregation modules are connected to the input of the overlay module.

[0051] In some embodiments of this application, the above-mentioned comprehensive feature extraction module is mainly used to extract local and global features of the gait image sequence to be identified; the above-mentioned global temporal feature aggregation module is mainly used to process the local and global features output by the comprehensive feature extraction module, aggregate adjacent gait frame features into global motion features, and obtain a global temporal feature matrix.

[0052] The aforementioned horizontal pooling layer is primarily used to perform horizontal pooling (HP) operations on the local and global features output by the comprehensive feature extraction module. It horizontally divides the input feature matrix (i.e., the local and global feature matrices) into n parts, performs max pooling and average pooling on each part, and then sums the results of both pooling operations. This segmentation operation prepares the module for subsequent local temporal feature fusion, while the pooling operation reduces the complexity of the feature matrix and preserves its main features.

[0053] The aforementioned local temporal feature aggregation module is mainly used to process the data output by the horizontal pooling layer, aggregating adjacent gait frame features into local temporal features to obtain a local temporal feature matrix; the aforementioned overlay module is mainly used to sum the global temporal feature matrix and the local temporal feature matrix to obtain the final comprehensive feature matrix (i.e., the gait recognition result of the target object).

[0054] In some application scenarios, after obtaining the gait recognition result of the target object, the gait recognition result can be compared with the gait feature matrix registered in the gait database (the gait database stores the identity information corresponding to the registered gait feature matrix) to determine the identity of the target object.

[0055] It is worth mentioning that, since the integrated feature extraction module can extract local and global features from the gait image sequence to be identified, and the final gait recognition result is an output that integrates global motion features and local temporal features, the improved GaitPart model extracts richer gait features than the traditional GaitPart model, thus improving the accuracy of gait recognition.

[0056] In some embodiments of this application, before performing step 12, the method further includes the steps of constructing the improved GaitPart model and training the constructed model.

[0057] The structure of the improved GaitPart model described above will be illustrated below with reference to specific embodiments.

[0058] In some embodiments of this application, such as Figure 2 As shown, the above-mentioned comprehensive feature extraction module includes: a first convolutional module (Block1), a first max pooling layer (Maxpool), a second convolutional module (Block2), a second max pooling layer (Maxpool), and a third convolutional module (Block3) connected in sequence. The input of the first convolutional module is the input of the comprehensive feature extraction module, and the output of the third convolutional module is the output of the comprehensive feature extraction module.

[0059] The first, second, and third convolution modules have the same structure, each including a first integrated convolution submodule and a second integrated convolution submodule connected in sequence. The first integrated convolution submodule and the second integrated convolution submodule are connected in series.

[0060] The first and second integrated convolution submodules have the same structure, both including: vertical block convolution units, horizontal block convolution units, ordinary convolution units, and a first stacking unit. The outputs of the vertical block convolution units, the horizontal block convolution units, and the ordinary convolution units are all connected to the input of the first stacking unit. The output of the first stacking unit of the first integrated convolution submodule is connected to the inputs of the vertical block convolution units, the horizontal block convolution units, and the ordinary convolution units of the second integrated convolution submodule, respectively.

[0061] In the first convolution module, the input terminals of the vertical block convolution units, horizontal block convolution units, and ordinary convolution units of the first comprehensive convolution submodule are the input terminals of the first convolution module. In the first convolution module, the output terminal of the first stacking unit of the second comprehensive convolution submodule is the output terminal of the first convolution module. In the second convolution module, the input terminals of the vertical block convolution units, horizontal block convolution units, and ordinary convolution units of the first comprehensive convolution submodule are the input terminals of the second convolution module. In the second convolution module, the output terminal of the first stacking unit of the second comprehensive convolution submodule is the output terminal of the second convolution module. In the third convolution module, the input terminals of the vertical block convolution units, horizontal block convolution units, and ordinary convolution units of the first comprehensive convolution submodule are the input terminals of the third convolution module. In the third convolution module, the output terminal of the first stacking unit of the second comprehensive convolution submodule is the output terminal of the third convolution module.

[0062] The first and second max pooling layers are mainly used to remove redundant information from the feature matrix and reduce its complexity.

[0063] In some embodiments of this application, such as Figure 3 As shown, the above-mentioned vertical block convolutional unit includes: a vertical cutting function, multiple first convolutional layers and a first cascaded layer. The output of each of the multiple first convolutional layers is connected to the input of the first cascaded layer, and the output of the first cascaded layer is connected to the input of the first stacking unit.

[0064] The aforementioned vertical segmentation function is used to vertically divide the data input to the vertically segmented convolutional unit into multiple sub-data blocks, and then input these sub-data blocks one-to-one into multiple first convolutional layers. For example, as shown... Figure 3 As shown, the number of the first convolutional layers can be four (e.g., ...). Figure 3The first four convolutional layers (Conv) from left to right use a vertical cut function to vertically divide the input data (represented as a matrix) into d1 (d1 = 4) blocks. The first convolutional layer performs convolution on the corresponding blocks and outputs the convolution result to the first cascaded layer (e.g., ...). Figure 3 The first C from left to right in the middle), the first concatenated layer is mainly used to concatenate the outputs of multiple first convolutional layers and output them to the first stacking unit (such as...). Figure 3 In ).

[0065] like Figure 3 As shown, the above-mentioned horizontal block convolutional unit includes: a horizontal cutting function, multiple second convolutional layers and a second cascaded layer. The output of each of the multiple second convolutional layers is connected to the input of the second cascaded layer, and the output of the second cascaded layer is connected to the input of the first stacking unit.

[0066] The aforementioned horizontal segmentation function is used to horizontally divide the data input to the horizontally segmented convolutional unit into multiple sub-data blocks, and then input these sub-data blocks one-to-one into multiple second convolutional layers. For example, as shown... Figure 3 As shown, the number of the second convolutional layers mentioned above can be four (e.g., ...). Figure 3 The 5th to 8th convolutional layers (Conv) from left to right in the middle layer have a horizontal cutting function that divides the input horizontally block-based convolutional unit data (represented as a matrix) into d2 (d2 = 4) blocks. The second convolutional layer performs convolution on the corresponding blocks and outputs the convolution result to the second cascaded layer (e.g., ...). Figure 3 The second C from left to right in the middle), the second concatenation layer is mainly used to concatenate the outputs of multiple second convolutional layers and output them to the first stacking unit (such as...). Figure 3 In ).

[0067] like Figure 3 As shown, the above-mentioned ordinary convolutional unit includes: a third convolutional layer, the input of the third convolutional layer being the input of the ordinary convolutional unit, and the output of the third convolutional layer being connected to the input of the first stacking unit.

[0068] The aforementioned third convolutional layer (such as Figure 3 The 9th convolutional layer (Conv) from left to right is mainly used to convolve the data (which can be represented as a matrix) input to ordinary convolutional units and output it to the first stacking unit (e.g., ...). Figure 3 In ).

[0069] The first stacking unit mentioned above is mainly used to sum the data output by the vertical block convolution unit, the horizontal block convolution unit, and the ordinary convolution unit, and output the summation result.

[0070] It should be noted that the first, second, and third convolutional layers mentioned above are all ordinary convolutional layers (i.e., traditional convolutional layers). It is worth mentioning that vertical and horizontal convolution after segmentation ensures the extraction of local features, while direct convolution of the original image ensures the extraction of global features. Therefore, the above comprehensive feature extraction module extracts richer gait features compared to the feature extraction module (Frame-level Part Feature Extractor) of the traditional GaitPart model.

[0071] It is understandable that although the first, second, and third convolutional modules have the same structure, their specific parameters (such as the size of the convolutional layers) are different. Similarly, the specific parameters (such as the size of the convolutional layers) in the first and second integrated convolutional submodules are different.

[0072] In some embodiments of this application, such as Figure 4 As shown, the aforementioned global time feature aggregation module includes: a first aggregation unit (GTB1), a second aggregation unit (GTB2), and a second overlay unit (e.g., ...). Figure 4 The second one from top to bottom The input terminals of the first aggregation unit and the second aggregation unit are both connected to the output terminal of the comprehensive feature extraction module. The output terminals of the first aggregation unit and the second aggregation unit are both connected to the input terminal of the second overlay unit. The output terminal of the second overlay unit is connected to the input terminal of the overlay module.

[0073] like Figure 4 As shown, the first aggregation unit and the second aggregation unit have the same structure, both including two branches, branch 1 and branch 2, specifically including: a first three-dimensional convolutional layer connected in sequence ( Figure 4 Conv3d1), ReLU activation function, second 3D convolutional layer ( Figure 4 Conv3d2 and Sigmoid activation functions; three-dimensional max pooling layer ( Figure 4 Maxpool3d1 and three-dimensional average pooling layers (in the context of Maxpool3d1) Figure 4 Avgpool3d1 in the middle), overlays (such as Avgpool3d1), and stacking layers (such Figure 4 The first one from top to bottom ) and product layers (such as Figure 4 The first one from top to bottom ).

[0074] The inputs of the first three-dimensional convolutional layer, the three-dimensional max pooling layer, and the three-dimensional average pooling layer are all connected to the output of the comprehensive feature extraction module. The outputs of the three-dimensional max pooling layer and the three-dimensional average pooling layer are all connected to the input of the stacked layer. The output of the stacked layer and the output of the Sigmoid activation function are all connected to the input of the product layer.

[0075] The output terminals of the product layer of the first aggregation unit and the product layer of the second aggregation unit are both connected to the input terminal of the second superposition unit.

[0076] For example, the size of Conv3d1 can be 3×5×5, the size of Conv3d2 can be 3×3×3, the size of Maxpool3d1 can be 3×3×3, and the size of Avgpool3d1 can be 3×3×3.

[0077] Conv3d is a 3D convolutional layer, Maxpool3d is a 3D max pooling layer, and Avgpool3d is a 3D average pooling layer. Assume the set of feature matrices input to the global temporal feature aggregation module is G. f ={f i |i=1,2,3,...,t}, there are t frames in total, f i This represents the feature matrix obtained by the comprehensive feature extraction module for the i-th frame in a gait sequence. Since GTB1 and GTB2 have the same structure but different parameters, GTB1 will be used as an example here. Its operation flow is as follows:

[0078] GTB1 contains two branches. Branch 1 connects GTB1 through two serially linked 3D convolutional layers, Conv3d1 and Conv3d2. f Adjacent feature matrices are aggregated while extracting deep features. Branch 2 aggregates G through parallel 3D max pooling layer Maxpool3d1 and 3D average pooling layer Avgpool3d1. f Adjacent feature matrices are aggregated while preserving the main features.

[0079] The operation performed in branch 1 is as follows:

[0080]

[0081] Feature matrix set G f The input is fed into the first 3D convolutional layer, Conv3d1. The result is processed by a ReLU activation function, which retains positive values ​​and eliminates negative values, thus reducing computation and preserving effective features. This result is then fed into the second 3D convolutional layer, Conv3d2, where it is processed to obtain the feature matrix set.

[0082] The operation performed in branch 2 is as follows:

[0083]

[0084] Feature matrix set G f Perform three-dimensional max pooling and three-dimensional average pooling respectively, and sum the results to obtain the set of feature matrices.

[0085] Finally, output from branch 1. After processing with the Sigmoid activation function, the values ​​are mapped to the interval [0,1], and then combined with the output of branch 2. Performing a dot product operation yields the aggregated feature matrix set f. Since the original gait contour image is a binary image, the dot product operation is essentially equivalent to an AND operation, which can retain the same features extracted from the two branches and eliminate different features. Its expression is as follows:

[0086]

[0087] Summing the outputs of GTB1 and GTB2 yields the set of feature matrices F = F i For each i = 1, 2, 3, ..., t, the feature matrix with the largest value in F is selected as the final output global temporal feature matrix u. Here, it is assumed that the larger the value of the feature matrix, the richer the gait information it contains. Its expression is as follows:

[0088] u = max(F1, F2, ..., F t )

[0089] This module introduces 3D convolutional layers and 3D pooling layers because each step of 3D convolution and pooling operations can process multiple adjacent images simultaneously. This aligns with the module's goal of aggregating adjacent gait frames. Taking a 3×5×5 3D convolutional layer as an example, its depth is 3, meaning that each step can aggregate the feature matrices of three adjacent gait frames into a global micro-motion feature matrix. Through 3D convolution and pooling, a set of gait feature matrices for a person can be aggregated into a global temporal feature matrix set, corresponding to the local temporal feature matrix set in the traditional GaitPart model. Here, global temporal features are incorporated into the original local temporal features, making the final feature matrix contain more comprehensive temporal characteristics.

[0090] In some embodiments of this application, the aforementioned local temporal feature aggregation module is the Temporal Feature Aggregator module (i.e., temporal feature aggregator) in the traditional GaitPart model, whose purpose is to aggregate adjacent gait frame feature matrices into a local temporal feature matrix. It contains n parallel micro-motion capture modules (MCMs), each MCM responsible for aggregating the feature vectors of the same part of the pedestrian at different times from the feature matrix set into a micro-motion feature vector. The MCM consists of a micro-motion template builder (MTB) and temporal pooling (TP), and its specific structure is as follows: Figure 5 As shown.

[0091] MTB1 and MTB2 have the same structure but different parameters. The outputs of MTB1 and MTB2 are summed, and then subjected to temporal pooling (TP) to obtain the final micro-motion feature vector. The specific structure of MTB is as follows: Figure 6 As shown.

[0092] ConvNet1d is a one-dimensional convolutional layer containing two serially connected one-dimensional convolutional kernels, with operation 'S' being the sigmoid activation function. TempFunc is the temporal template.

[0093] Assume the set of input feature matrices is G. f ={f i,j Given a string of frames |i=1,2,3,...,t;j=1,2,3,...n}, there are t frames in total. Each frame is divided into n parts by a horizontal pooling operation. i,j This represents the feature vector obtained from the j-th part of the i-th frame in a gait sequence after comprehensive feature extraction and horizontal pooling. The processing flow of this module is as follows:

[0094] 1. Micro-motion capture module (MCM)

[0095] w j =MCM j (f i,j )

[0096] This involves aggregating the j-th portion of each frame in a person's gait sequence into a feature vector w. j The j feature vectors are cascaded through a fully connected layer to obtain the final local time feature matrix.

[0097] 2. Micro Motion Template Builder (MTB)

[0098] Select G f The set G is obtained by taking a row in the dataset, i.e., all the feature vectors of the same part at different times. p ={f i|i=1,2,3,...,t}, where the TempFunc module performs a max pooling operation Maxpool2d and an average pooling operation Avgpool2d with a kernel size of k, and sums the results to obtain the feature vector set G. m The kernel size represents the aggregation of k consecutive feature vectors into a single feature vector, and its expression is as follows:

[0099] G m =Maxpool2d(G p )+Avgpool2d(G p )

[0100] In addition, to extract more discriminative micro-motion features, MTB introduces a channel attention mechanism, which involves adding a one-dimensional convolutional layer ConvNet1d and assembling G... p The eigenvectors in the set are input to obtain the set G. logits , will G logits After activation by the Sigmoid function and G m Performing dot product operation yields the processed feature vector set. Its expression is as follows:

[0101]

[0102]

[0103] Temporal pooling (TP) is a process that pools features from a set of feature vectors. We select the maximum value from the set as the final feature vector w, and its expression is as follows:

[0104]

[0105]

[0106] The feature vectors w obtained from each MCM process are concatenated together through a fully connected layer to obtain the final local temporal feature matrix.

[0107] In some embodiments of this application, to achieve gait recognition, the gait contour maps in the gait dataset CASIA-B can be used as the training set, and the improved GaitPart model can be trained using the triplet loss function.

[0108] Because the original gait contour images are quite large, to reduce the complexity of data processing, the gait contour images in the CASIA-B gait dataset need to be cropped to 64×48 pixels. Specifically, the human body contour is separated by finding the maximum height (height_max, top index), minimum height (height_min, bottom index), minimum width (wide_min, leftmost index of the contour), and maximum width (wide_max, rightmost index of the contour) of the person in the image. Since the original human body contour images in the CASIA-B gait dataset are binary images, all columns of the binary image are first summed to form a column c. The index of the first non-zero pixel value in c is the maximum height, and the index of the last non-zero pixel value in c is the minimum height. Then, all rows of the binary image are summed to form a row r. The index of the first non-zero pixel value in r is the minimum width, and the index of the last non-zero pixel value in r is the maximum width. Let the difference between the maximum and minimum height be size. The resulting silhouette image will have dimensions of size × size. Finally, the image will be normalized and cropped again, resulting in a gait silhouette image with dimensions of 64 × 48. size = height_max - height_min.

[0109] To enable the model to generalize and find the closest match to the target in a wide range of pedestrian gait data, thereby identifying the target's identity, a triplet loss function is used to train the model.

[0110] The triplet loss function consists of a triple {s_o, s_c, s_w}, where each parameter corresponds to a gait sequence. s_o is the original sample, s_c is a positive sample (belonging to the same category as the original sample, representing the same person but with different states or shooting angles), and s_w is a negative sample (not belonging to the same category as the original sample, representing a different person). All three types of samples are randomly selected from the training set.

[0111] The triplet loss function works by training the model to reduce the Euclidean distance between s_o and s_c, while increasing the Euclidean distance between s_o and s_w. The specific formula is as follows:

[0112] Loss=max(d(s_o,s_c)-d(s_o,s_w)+μ,0)

[0113] In the above formula, Loss is the loss value, max(,) is the maximum value function, d(,) represents the Euclidean distance between two samples, and μ represents the threshold margin, which aims to control the difference between the Euclidean distance d(s_o,s_c) between the original sample and the positive sample and the Euclidean distance d(s_o,s_w) between the original sample and the negative sample, thus ensuring the stability of model training. The working process of the triplet loss function is as follows: Figure 7 As shown. Among them, Figure 7 In this context, LEARNING means training.

[0114] The parameters of the trained GaitPart model (i.e., the improved GaitPart model in step 12 above) are illustrated below.

[0115] In the table below, Block represents the module name, Layer represents the network layer name within each module, In_C represents the number of input channels of the convolutional layer, Out_C represents the number of output channels of the convolutional layer, Kernel represents the size of the convolutional kernel (for example, the Kernel of C-conv1 (i.e., the first integrated convolutional submodule mentioned above) is 5, meaning its convolutional kernel size is 5×5), Pad represents the number of zero-padding layers, Stride represents the stride, d1 represents the parameters of the vertically segmented convolutional layer, and d2 represents the parameters of the horizontally segmented convolutional layer. C-conv2 represents the second integrated convolutional submodule.

[0116] 1. The specific parameters of the comprehensive feature extraction module are shown in Table 1.

[0117] Table 1

[0118]

[0119] 2. The specific parameters of the global time feature aggregation module are shown in Table 2.

[0120] Table 2

[0121]

[0122] 3. The specific parameters of the local time feature aggregation module are shown in Table 3.

[0123] Table 3

[0124]

[0125] In some embodiments of this application, the detailed processing procedure of the improved GaitPart model for gait data is as follows:

[0126] Training was performed on the CASIA-B gait dataset, which contains 124 individuals, each with 11 viewpoints (0 degrees, 18 degrees, 36 degrees, ..., 180 degrees), collected under three walking conditions (normal, wearing a coat, carrying a package). Gait data from the first 73 individuals was used for training, and data from the last 51 individuals was used for testing. The learning rate was 0.0001, and each batch training specification was (8, 16, 30), meaning 8 individuals, 16 randomly selected gait sequences per individual, and 30 frames extracted from each sequence. The first two parameters in the training specification can be adjusted according to GPU performance; here, they were set to 8 and 16. The detailed process is as follows:

[0127] 0. Input

[0128] The original input image is 3840×1×64×48 (3840=8×16×30, which is the training specification for each batch).

[0129] 1a.C-conv1(d1=d2=1)

[0130] Using a 5×5 convolution kernel (sliding stride of 1, padding of 2), with 1 input channel and 32 output channels, a ReLU operation is performed after convolution, and the output matrix has a size of 3840×32×64×48.

[0131] 1b.C-conv2(d1=d2=1)

[0132] Using a 3×3 convolution kernel (sliding stride of 1, padding of 1), with 32 input channels and 32 output channels, ReLU operation is performed after convolution, and the output matrix has a size of 3840×32×64×48.

[0133] 1c.C-conv3(d1=d2=4)

[0134] Using a 3×3 convolution kernel (sliding stride of 1, padding of 1), with 32 input channels and 64 output channels, a ReLU operation is performed after convolution, and the output matrix has dimensions of 3840×64×32×24.

[0135] 1d.C-conv4(d1=d2=4)

[0136] Using a 3×3 convolution kernel (sliding stride of 1, padding of 1), with 64 input channels and 64 output channels, a ReLU operation is performed after convolution, and the output matrix has dimensions of 3840×64×32×24.

[0137] 1e.C-conv5(d1=6,d2=8)

[0138] Using a 3×3 convolution kernel (sliding stride of 1, padding of 1), with 64 input channels and 128 output channels, a ReLU operation is performed after convolution, and the output matrix has a size of 3840×128×16×12.

[0139] 1f.C-conv6(d1=6,d2=8)

[0140] Using a 3×3 convolution kernel (sliding stride of 1, padding of 1), with 128 input channels and 128 output channels, ReLU operation is performed after convolution, and the output matrix has a size of 3840×128×16×12.

[0141] 2. HP (Horizontal Pooling)

[0142] The matrix obtained by performing horizontal pooling on the output of 1f has a size of 128×30×128×16.

[0143] 3. Global Temporal Feature Aggregation Module (GTFA)

[0144] 3a.GTB1

[0145] Branch 1: A 3D convolutional layer Conv3d1 with a kernel size of 3×5×5 and padding of (1, 2, 2) is convolved and then subjected to ReLU operation. A 3D convolutional layer Conv3d2 with a kernel size of 3×3×3 and padding of (1, 1, 1) is convolved and then subjected to Sigmoid operation. The resulting output matrix 1 has a size of 128×128×30×16×12.

[0146] Branch 2: The output matrix 2 obtained by summing the results of the three-dimensional max pooling layer Maxpool3d1 and the three-dimensional average pooling layer Avgpool3d1 with pooling kernel size of 3×3×3 and padding of (1,1,1) has a size of 128×128×30×16×12.

[0147] The product of output matrix 1 and output matrix 2 yields a final output matrix with dimensions of 128×128×30×16×12.

[0148] 3b.GTB2

[0149] Branch 1: A 3D convolutional layer Conv3d3 with a kernel size of 5×5×5 and padding of (2, 2, 2) is convolved and then ReLU is performed. A 3D convolutional layer Conv3d4 with a kernel size of 5×5×5 and padding of (2, 2, 2) is convolved and then Sigmoid is performed. The resulting output matrix 1 has a size of 128×128×30×16×12.

[0150] Branch 2: The output matrix 2 obtained by summing the results of two three-dimensional max pooling layers, Maxpool3d2 and Avgpool3d2, with a pooling kernel size of 5×5×5 and a padding of (2, 2, 2), has a size of 128×128×30×16×12.

[0151] The product of output matrix 1 and output matrix 2 yields a final output matrix with dimensions of 128×128×30×16×12.

[0152] The matrix obtained by adding the results of GTB1 and GTB2 is 128×128×30×16×12. The matrix obtained by cascading through a fully connected layer is 128×128×30×16. Finally, the final output matrix is ​​128×128×16 by taking the largest matrix.

[0153] 4. Local Temporal Feature Aggregation Module (LTFA)

[0154] 4a.MTB1

[0155] A one-dimensional convolutional layer Conv1d1 with a kernel size of 3, padding of 1, 2048 input channels, and 128 output channels, and a one-dimensional convolutional layer Conv1d2 with a kernel size of 1, padding of 0, 128 input channels, and 2048 output channels are concatenated in series. After convolution, a sigmoid operation is performed, resulting in an output matrix 1 with dimensions of 128×2048×30. Then, a max pooling layer Maxpool1d1 with a kernel size of 3 and padding of 1, and an average pooling layer Avgpool1d1 are applied, with the results summed to obtain an output matrix 2 with dimensions of 128×2048×30. The product of output matrix 1 and output matrix 2 yields the final output matrix with dimensions of 128×2048×30.

[0156] 4b.MTB2

[0157] A one-dimensional convolutional layer Conv1d3 with a kernel size of 3, padding of 1, 2048 input channels, and 128 output channels, and a one-dimensional convolutional layer Conv1d4 with a kernel size of 3, padding of 1, 128 input channels, and 2048 output channels are sequentially connected. After convolution, a sigmoid operation is performed, resulting in output matrix 1 with dimensions of 128×2048×30. Then, a max pooling layer Maxpool1d2 with a kernel size of 5 and padding of 2, and an average pooling layer Avgpool1d2 are applied, and the sum of the results yields output matrix 2 with dimensions of 128×2048×30. Multiplying output matrix 1 and output matrix 2 yields the final output matrix with dimensions of 128×2048×30.

[0158] The matrix obtained by adding the results of MTB1 and MTB2 is 128×2048×30. Then, the final output matrix is ​​128×128×16 after time pooling.

[0159] Summing the results of GTFA and LTFA, the final feature matrix has a size of 128×128×16.

[0160] In summary, the gait recognition method based on pedestrian comprehensive features optimizes the original GaitPart model, which not only enriches the gait features extracted by the model, but also integrates temporal features more comprehensively into the feature matrix, thereby improving the gait recognition accuracy of the model.

[0161] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A gait recognition method, characterized in that, include: Obtain a sequence of gait images of a target object to be identified, wherein the sequence of gait images to be identified includes multiple gait images to be identified; The gait image sequence to be identified is input into the improved GaitPart model for gait recognition to obtain the gait recognition result of the target object; The improved GaitPart model includes a comprehensive feature extraction module for extracting local and global features from the gait image sequence to be recognized, a global temporal feature aggregation module for processing the local and global features output by the comprehensive feature extraction module, a horizontal pooling layer, a local temporal feature aggregation module for aggregating local temporal features, and an overlay module for outputting the gait recognition result of the target object. The output of the comprehensive feature extraction module is connected to the input of the global temporal feature aggregation module and the input of the horizontal pooling layer, respectively. The output of the horizontal pooling layer is connected to the input of the local temporal feature aggregation module. The outputs of the global temporal feature aggregation module and the local temporal feature aggregation module are both connected to the input of the overlay module. The comprehensive feature extraction module includes: a first convolutional module, a first max pooling layer, a second convolutional module, a second max pooling layer, and a third convolutional module connected in sequence. The input of the first convolutional module is the input of the comprehensive feature extraction module, and the output of the third convolutional module is the output of the comprehensive feature extraction module. The first convolution module, the second convolution module, and the third convolution module each include a first comprehensive convolution submodule and a second comprehensive convolution submodule connected in sequence; Both the first and second integrated convolutional submodules include: a vertical block convolutional unit, a horizontal block convolutional unit, a regular convolutional unit, and a first stacking unit. The outputs of the vertical block convolutional unit, the horizontal block convolutional unit, and the regular convolutional unit are all connected to the input of the first stacking unit. The output of the first stacking unit of the first integrated convolutional submodule is connected to the inputs of the vertical block convolutional unit, the horizontal block convolutional unit, and the regular convolutional unit of the second integrated convolutional submodule, respectively. In the first convolution module, the input terminals of the vertical block convolution unit, horizontal block convolution unit, and ordinary convolution unit of the first comprehensive convolution submodule are the input terminals of the first convolution module. In the first convolution module, the output terminal of the first stacking unit of the second comprehensive convolution submodule is the output terminal of the first convolution module. Similarly, in the third convolution module, the input terminals of the vertical block convolution unit, horizontal block convolution unit, and ordinary convolution unit of the first comprehensive convolution submodule are the input terminals of the third convolution module. The output terminal of the first stacking unit of the second comprehensive convolution submodule in the second convolution module is the output terminal of the third convolution module.

2. The method according to claim 1, characterized in that, The vertical block convolutional unit includes: a vertical cutting function, multiple first convolutional layers and a first cascaded layer, wherein the output of each of the multiple first convolutional layers is connected to the input of the first cascaded layer, and the output of the first cascaded layer is connected to the input of the first stacking unit. The vertical cutting function is used to vertically divide the data input to the vertical block convolutional unit into multiple sub-data blocks, and input the multiple sub-data blocks one by one into the multiple first convolutional layers.

3. The method according to claim 1, characterized in that, The horizontal block convolutional unit includes: a horizontal cutting function, multiple second convolutional layers and a second cascaded layer, wherein the output of each of the multiple second convolutional layers is connected to the input of the second cascaded layer, and the output of the second cascaded layer is connected to the input of the first stacking unit. The horizontal slicing function is used to horizontally divide the data input to the horizontal block convolutional unit into multiple sub-data blocks, and input the multiple sub-data blocks one by one into the multiple second convolutional layers.

4. The method according to claim 1, characterized in that, The ordinary convolutional unit includes: a third convolutional layer, the input of which is the input of the ordinary convolutional unit, and the output of which is connected to the input of the first stacking unit.

5. The method according to claim 1, characterized in that, The global temporal feature aggregation module includes: a first aggregation unit, a second aggregation unit, and a second overlay unit. The input terminals of the first aggregation unit and the second aggregation unit are both connected to the output terminal of the comprehensive feature extraction module. The output terminals of the first aggregation unit and the second aggregation unit are both connected to the input terminal of the second overlay unit. The output terminal of the second overlay unit is connected to the input terminal of the overlay module.

6. The method according to claim 5, characterized in that, Both the first aggregation unit and the second aggregation unit include: a first three-dimensional convolutional layer, a ReLU activation function, a second three-dimensional convolutional layer, and a Sigmoid activation function connected in sequence; a three-dimensional max pooling layer, a three-dimensional average pooling layer, a stacking layer, and a multiplication layer; The input terminals of the first three-dimensional convolutional layer, the three-dimensional max pooling layer, and the three-dimensional average pooling layer are all connected to the output terminal of the comprehensive feature extraction module. The output terminals of the three-dimensional max pooling layer and the three-dimensional average pooling layer are all connected to the input terminal of the stacked layer. The output terminal of the stacked layer and the output terminal of the Sigmoid activation function are all connected to the input terminal of the product layer. The output terminals of the product layer of the first aggregation unit and the product layer of the second aggregation unit are both connected to the input terminal of the second superposition unit.

Images