Transform and graph convolution-based gait spatial-temporal feature extraction method and system

Abstract

The invention provides a gait spatial-temporal feature extraction method and system based on Transform and graph convolution, and the method comprises the steps: obtaining a human skeleton sequence, carrying out the data enhancement processing of the obtained human skeleton sequence, and obtaining a human skeleton sequence YDA after the data enhancement; performing data pre-processing on the YDA to obtain a pre-processed gait skeleton sequence YProcess; based on a Transform feature extraction network, feature extraction is carried out on the YProcess, and an output feature YT is obtained; performing feature extraction on the YProcess based on a feature extraction network of graph convolution to obtain an output feature YG; the output features YT and YG are subjected to batch normalization neck processing, and a feature YTBNNecks and a feature YGBNNecks are obtained; the output features YT and YG are spliced, and a spliced feature YConcat is obtained; then, the features YTBNNecks and the features YGBNNecks are spliced, and the spliced feature YBNNecks is obtained, and then the feature YTBNNecks and the feature YGBNNecks are obtained; carrying out feature mapping operation on the YConcat to obtain a skeleton spatio-temporal feature YFCs; a joint loss function composed of a triple loss function and a cross entropy loss function is used to supervise the spatial-temporal feature YFCs and the feature YBNNecks of the skeleton, and the discrimination capability of the skeleton features is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of gait feature extraction, and more specifically, to a method and system for extracting spatiotemporal features of gait based on Transformer and graph convolution. Background Art

[0002] The graph convolution-based gait feature extraction network models the human skeleton as graph-structured data, with joints as graph nodes and skeletal connections as edges. This effectively captures the topological relationships of human motion through graph convolution operations. The Transformer-based gait feature extraction network uses the Transformer network to effectively model long-range dependencies between joints through a self-attention mechanism.

[0003] In controlled environments, the viewing angle is typically fixed and pre-set. However, in open environments, pedestrians' movements are highly random, resulting in diverse and realistic viewing angles. This results in significant differences in the skeleton of the same person from different viewing angles, exacerbating the difficulty of extracting skeletal features and severely impacting recognition performance. Current mainstream gait recognition methods are insufficient in extracting detailed skeletal features when faced with diverse viewing angles, and lack the ability to simultaneously extract long-term, global, and local spatial features. Summary of the Invention

[0004] In response to the technical problems raised above, a method and system for extracting spatiotemporal features of gait based on Transformer and graph convolution is provided. The method of the present invention can improve the performance degradation problem caused by diverse viewpoints in open environments.

[0005] The technical means adopted in the present invention are as follows:

[0006] A gait spatiotemporal feature extraction method based on Transformer and graph convolution, including:

[0007] S1. Obtain a human skeleton sequence and perform data enhancement on the obtained human skeleton sequence to obtain a human skeleton sequence Y after data enhancement. DA ;

[0008] S2, human skeleton sequence Y after data enhancement DA Perform data preprocessing to obtain the preprocessed gait skeleton sequence Y Process ;

[0009] S3, Transformer-based feature extraction network, the preprocessed gait skeleton sequence Y Process Perform feature extraction and obtain output feature Y T ;

[0010] S4, a feature extraction network based on graph convolution, is used to extract the preprocessed gait skeleton sequence Y Process Perform feature extraction and obtain output feature Y G ;

[0011] S5. Output feature Y T And the output feature Y G Perform batch normalization processing respectively to obtain feature Y T_BNNecks and feature Y G_BNNecks ;

[0012] S6. Output feature Y T And the output feature Y G Perform splicing to obtain the spliced ​​feature Y Concat ; Then feature Y T_BNNecks and feature Y G_BNNecks Perform splicing to obtain the spliced ​​feature Y BNNecks ;

[0013] S7, the spliced ​​feature Y Concat Perform feature mapping operation to obtain the skeleton spatiotemporal feature Y FCs ;

[0014] S8. Use the joint loss function composed of the triple loss function and the cross entropy loss function to calculate the skeleton spatiotemporal feature Y FCs and feature Y BNNecks Provide supervision to improve the discriminative ability of skeleton features.

[0015] Furthermore, step S1 specifically includes:

[0016] S11. Given the input skeleton sequence Y, Where N represents the batch size, T represents the length of the contour sequence, V represents the number of joint points of each skeleton, and 3 represents the number of input skeleton channels;

[0017] S12. Set a threshold value ranging from 0 to 1. When the generated random number (ranging from 0 to 1) is greater than the threshold, the order of the input gait skeleton sequence is flipped. The gait skeleton sequence after data enhancement is Y DA ,

[0018] Furthermore, step S2 specifically includes:

[0019] S21, let the human skeleton after data enhancement be J, Slice the channels of the human skeleton, retain the joint coordinate channels, remove the confidence channels, and obtain the human skeleton J1.

[0020]

[0021] S22, set the joint feature to the joint point coordinates J1[:,:,i], and use the nose joint point as the reference point to determine the relative joint coordinates J1[:,:,i]-J1[:,:,i nose ], where i nose Indicates the nose joint point;

[0022] S23, obtain velocity features by first-order frame difference J1[:,t+1,:]-J1[:,t,:] and second-order frame difference J1[:,t+2,:]-J1[:,t,:];

[0023] S24, set the bone feature to J1[:,:,i]-J1[:,:,i adj ], where i adj Represents the adjacent joint of the i-th joint point;

[0024] S25, combining the joint features, velocity features, and bone features to form the human skeleton J2 after data preprocessing. The gait skeleton sequence after data preprocessing is Y Process ,

[0025] Furthermore, step S3 specifically includes:

[0026] S31, using GaitTR network as Transformer-based feature extraction network;

[0027] S32, the pre-processed gait skeleton sequence Y Process Input to GaitTR network, output feature Y T , C represents the output channel dimension.

[0028] Furthermore, step S4 specifically includes:

[0029] S41. Use GaitGraph network as a feature extraction network based on graph convolution;

[0030] S42. In order to enable the GaitGraph network to better extract skeleton detail features, the bottleneck residual block in the GaitGraph network is replaced with a basic residual block;

[0031] S43, the pre-processed gait skeleton sequence Y Process Input to GaitGraph network, output feature Y G , C represents the output channel dimension.

[0032] Furthermore, step S5 specifically includes:

[0033] S51, the output feature Y of the Transformer-based feature extraction network T After batch normalization, feature Y is obtained T_BNNecks , Q represents the number of pedestrians identified by gait in the training set;

[0034] S52, extract the output feature Y of the feature extraction network based on graph convolution G After batch normalization, feature Y is obtained G_BNNecks ,

[0035] Furthermore, step S6 specifically includes:

[0036] S61, through the splicing operation, the output feature Y of the Transformer-based feature extraction network T And the output feature Y of the graph convolution-based feature extraction network G Fusion is performed to obtain the spliced ​​feature Y Concat ,in, The calculation process is as follows:

[0037] Y Concat =Concat(Y T ,Y G )

[0038] Among them, Concat(·) represents the feature concatenation operation;

[0039] S62, through the splicing operation, the output feature Y of the Transformer-based feature extraction network T Feature Y after batch normalization T_BNNecks And the output feature Y of the graph convolution-based feature extraction network G Feature Y after batch normalization G_BNNecks Fusion is performed to obtain the spliced ​​feature Y BNNecks , The calculation process is as follows:

[0040] Y BNNecks =Concat(Y T_BNNecks ,Y G_BNNecks ).

[0041] Furthermore, step S7 specifically includes:

[0042] S71, using ReLU activation function, the concatenated feature Y Concat Perform nonlinear transformation operations;

[0043] S72, then use the parallel fully connected layer to perform feature mapping to obtain the skeleton spatiotemporal feature YFCs , The calculation process is as follows:

[0044] Y FCs =FCs(ReLU(Y concat ))

[0045] Among them, FCs(·) represents the operation of using parallel fully connected layers to perform feature mapping, and ReLU(·) represents the operation of using activation functions to perform nonlinear transformation on features.

[0046] Furthermore, step S8 specifically includes:

[0047] S81, let the skeleton spatiotemporal feature Y in the j∈[1,M]th valid triplet FCs The i-th eigenvector in is v i,j , The skeleton spatiotemporal feature Y is analyzed using the triplet loss function FCs For supervision, the triplet loss function is as follows:

[0048]

[0049] Among them, M represents the number of valid triplets in the batch, and P represents the skeleton spatiotemporal feature Y FCs The number of eigenvectors in , [·] + Indicates a positive function, margin indicates the minimum distance difference between the preset positive and negative sample pairs, and the two-norm is used to measure the feature distance. represents the anchor sample, represents the positive sample, and The identity tag is the same as represents negative samples, and The identity labels are different;

[0050] S82, let Y in the r∈[1,N]th sample BNNecks The i∈[1,P]th eigenvector of i,r , N represents the batch size, given the true identity label y i,r , let the eigenvector q i,r Identity y i,r The probability of ∈[1,Q] is Q represents the number of identities in the training set, and the cross entropy loss function is used to calculate the feature Y BNNecks For supervision, the cross entropy loss function is as follows:

[0051]

[0052] S83. The triple loss function and the cross entropy loss function are combined into a joint loss function as follows:

[0053] L=λL triplet +(1-λ)L cross_entropy

[0054] Among them, L triplet represents the triple loss function, L cross_entropy Represents the cross entropy loss function, λ is a hyperparameter, and represents the weight value of the triplet loss function.

[0055] The present invention also provides a gait spatiotemporal feature extraction system based on Transformer and graph convolution, which is implemented based on the above-mentioned gait spatiotemporal feature extraction method based on Transformer and graph convolution, comprising: a data enhancement module, a data preprocessing module, a Transformer-based feature extraction module, a graph convolution-based feature module, a batch normalization module, a feature splicing module, a feature mapping module, and a loss function supervision module, wherein:

[0056] The data enhancement module is used to obtain a human skeleton sequence and perform data enhancement processing on the obtained human skeleton sequence to obtain a human skeleton sequence Y after data enhancement. DA ;

[0057] The data preprocessing module is used to process the human skeleton sequence Y after data enhancement. DA Perform data preprocessing to obtain the preprocessed gait skeleton sequence Y Process ;

[0058] The Transformer-based feature extraction module is used for the Transformer-based feature extraction network to extract the gait skeleton sequence Y after preprocessing. Process Perform feature extraction and obtain output feature Y T ;

[0059] The graph convolution-based feature module is used for the feature extraction network based on graph convolution to extract the gait skeleton sequence Y after preprocessing. Process Perform feature extraction and obtain output feature Y G ;

[0060] The batch normalization module is used to transform the output feature Y T And the output feature Y G Perform batch normalization processing respectively to obtain feature Y T_BNNecks and feature Y G_BNNecks ;

[0061] The feature splicing module is used to output feature Y T And the output feature Y G Perform splicing to obtain the spliced ​​feature Y Concat ; Then feature YT_BNNecks and feature Y G_BNNecks Perform splicing to obtain the spliced ​​feature Y BNNecks ;

[0062] The feature mapping module is used to map the concatenated features Y Concat Perform feature mapping operation to obtain the skeleton spatiotemporal feature Y FCs ;

[0063] The loss function supervision module uses a joint loss function consisting of a triplet loss function and a cross entropy loss function to calculate the skeleton spatiotemporal feature Y FCs and feature Y BNNecks Provide supervision to improve the discriminative ability of skeleton features.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] 1. The present invention provides a gait spatiotemporal feature extraction method based on Transformer and graph convolution, which splices the output features of the Transformer-based gait feature extraction network and the output features of the graph convolution-based gait feature extraction network, and simultaneously extracts the global spatial domain features, local spatial domain features and long-term domain features of the skeleton, thereby enhancing the skeleton feature extraction capability.

[0066] 2. The present invention provides a gait spatiotemporal feature extraction method based on Transformer and graph convolution, which improves the expressive ability of skeleton features by performing feature mapping on the spliced ​​features.

[0067] 3. The present invention provides a gait spatiotemporal feature extraction method based on Transformer and graph convolution. For the gait feature extraction network based on graph convolution, the residual graph convolution uses basic residual blocks to improve the ability to extract skeleton detail features.

[0068] 4. The present invention provides a gait spatiotemporal feature extraction method based on Transformer and graph convolution, which uses triple loss function and cross entropy loss function for joint supervision to improve the discriminative ability of skeleton features.

[0069] 5. The present invention provides a gait spatiotemporal feature extraction method based on Transformer and graph convolution, which can improve the performance degradation problem caused by diverse perspectives in open environments. Its Rank-1 performance on the GREW dataset surpasses the existing typical skeleton feature-based methods GaitGraph, GaitGraph2, GaitTR and GPGait, fully demonstrating the superior performance of this method among the skeleton feature-based methods.

[0070] Based on the above reasons, the present invention can be widely promoted in fields such as gait feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] Figure 1 Flow chart of the method of the present invention.

[0073] Figure 2 This is the GaitTR network structure of the present invention.

[0074] Figure 3 This is the residual graph convolution structure based on the basic residual block of the present invention.

[0075] Figure 4 This is the GaitGraph network structure based on the basic residual block of the present invention.

[0076] Figure 5 This is the feature mapping module structure of the present invention. DETAILED DESCRIPTION

[0077] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0078] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatuses.

[0079] like Figure 1 As shown, the present invention provides a gait spatiotemporal feature extraction method based on Transformer and graph convolution, including:

[0080] S1. Obtain a human skeleton sequence and perform data enhancement on the obtained human skeleton sequence to obtain a human skeleton sequence Y after data enhancement. DA ;

[0081] S2, human skeleton sequence Y after data enhancement DA Perform data preprocessing to obtain the preprocessed gait skeleton sequence Y Process ;

[0082] S3, Transformer-based feature extraction network, the preprocessed gait skeleton sequence Y Process Perform feature extraction and obtain output feature Y T ;

[0083] S4, a feature extraction network based on graph convolution, is used to extract the preprocessed gait skeleton sequence Y Process Perform feature extraction and obtain output feature Y G ;

[0084] S5. Output feature Y T And the output feature Y G Perform batch normalization processing respectively to obtain feature Y T_BNNecks and feature Y G_BNNecks ;

[0085] S6. Output feature Y T And the output feature Y G Perform splicing to obtain the spliced ​​feature Y Concat ; Then feature Y T_BNNecks and feature Y G_BNNecks Perform splicing to obtain the spliced ​​feature Y BNNecks ;

[0086] S7, the spliced ​​feature Y Concat Perform feature mapping operation to obtain the skeleton spatiotemporal feature Y FCs ;

[0087] S8. Use the joint loss function composed of the triple loss function and the cross entropy loss function to calculate the skeleton spatiotemporal feature Y FCs and feature Y BNNecks Provide supervision to improve the discriminative ability of skeleton features.

[0088] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0089] S11. Given the input skeleton sequence Y, Where N represents the batch size, T represents the length of the contour sequence, V represents the number of joint points of each skeleton, and 3 represents the number of input skeleton channels;

[0090] S12. Set a threshold value ranging from 0 to 1, which is set to 0.5 here. When the generated random number (ranging from 0 to 1) is greater than the threshold, the order of the input gait skeleton sequence is flipped. The gait skeleton sequence after data enhancement is Y DA ,

[0091] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0092] S21, let the human skeleton after data enhancement be J, Slice the channels of the human skeleton, retain the joint coordinate channels, remove the confidence channels, and obtain the human skeleton J1.

[0093]

[0094] S22, set the joint feature to the joint point coordinates J1[:,:,i], and use the nose joint point as the reference point to determine the relative joint coordinates J1[:,:,i]-J1[:,:,i nose ], where i nose Indicates the nose joint point;

[0095] S23, obtain velocity features by first-order frame difference J1[:,t+1,:]-J1[:,t,:] and second-order frame difference J1[:,t+2,:]-J1[:,t,:];

[0096] S24, set the bone feature to J1[:,:,i]-J1[:,:,i adj ], where i adj Represents the adjacent joint of the i-th joint point;

[0097] S25, combining the joint features, velocity features, and bone features to form the human skeleton J2 after data preprocessing. The gait skeleton sequence after data preprocessing is Y Process ,

[0098] In specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:

[0099] S31, use GaitTR network as the feature extraction network based on Transformer; GaitTR network structure is as follows Figure 2 As shown, the stacking number of module 3 is 5.

[0100] S32, the pre-processed gait skeleton sequence Y Process Input to GaitTR network, output feature Y T , C represents the output channel dimension.

[0101] In specific implementation, as a preferred embodiment of the present invention, step S4 specifically includes:

[0102] S41. Use GaitGraph network as a feature extraction network based on graph convolution;

[0103] S42. In order to enable the GaitGraph network to better extract skeleton detail features, the bottleneck residual block in the GaitGraph network is replaced with a basic residual block; the residual graph convolution structure based on the basic residual block is as follows Figure 3 As shown, the GaitGraph network structure based on the basic residual block is as follows Figure 4 As shown, Figure 4 The basic blocks in the middle represent residual graph convolution based on basic residual blocks. The number of basic blocks stacked in module 1 is 3, and the number of basic blocks stacked in module 2 is 6.

[0104] S43, the pre-processed gait skeleton sequence Y Process Input to GaitGraph network, output feature Y G , C represents the output channel dimension.

[0105] In specific implementation, as a preferred embodiment of the present invention, step S5 specifically includes:

[0106] S51, the output feature Y of the Transformer-based feature extraction network T After batch normalization, feature Y is obtained T_BNNecks , Q represents the number of pedestrians identified by gait in the training set;

[0107] S52, extract the output feature Y of the feature extraction network based on graph convolution G After batch normalization, feature Y is obtained G_BNNecks ,

[0108] In specific implementation, as a preferred embodiment of the present invention, step S6 specifically includes:

[0109] S61, through the splicing operation, the output feature Y of the Transformer-based feature extraction network T And the output feature Y of the graph convolution-based feature extraction network G Fusion is performed to obtain the spliced ​​feature Y Concat ,in, The calculation process is as follows:

[0110] YConcat =Concat(Y T ,Y G )

[0111] Among them, Concat(·) represents the feature concatenation operation;

[0112] S62, through the splicing operation, the output feature Y of the Transformer-based feature extraction network T Feature Y after batch normalization T_BNNecks And the output feature Y of the graph convolution-based feature extraction network G Feature Y after batch normalization G_BNNecks Fusion is performed to obtain the spliced ​​feature Y BNNecks , The calculation process is as follows:

[0113] Y BNNecks =Concat(Y T_BNNecks ,Y G_BNNecks ).

[0114] In specific implementation, as a preferred embodiment of the present invention, step S7 specifically includes:

[0115] S71, using ReLU activation function, the concatenated feature Y Concat Perform nonlinear transformation operations;

[0116] S72, then use the parallel fully connected layer to perform feature mapping to obtain the skeleton spatiotemporal feature Y FCs , The calculation process is as follows:

[0117] Y FCs =FCs(ReLU(Y concat ))

[0118] Among them, FCs(·) represents the operation of using parallel fully connected layers to perform feature mapping, and ReLU(·) represents the operation of using activation functions to perform nonlinear transformation on features. The feature mapping module structure is as follows Figure 5 shown.

[0119] In specific implementation, as a preferred embodiment of the present invention, step S8 specifically includes:

[0120] S81, let the skeleton spatiotemporal feature Y in the j∈[1,M]th valid triplet FCs The i-th eigenvector in is v i,j , The skeleton spatiotemporal feature Y is analyzed using the triplet loss function FCs For supervision, the triplet loss function is as follows:

[0121]

[0122] Among them, M represents the number of valid triplets in the batch, and P represents the skeleton spatiotemporal feature Y FCs The number of eigenvectors in , [·] + Indicates a positive function, margin indicates the minimum distance difference between the preset positive and negative sample pairs, and the two-norm is used to measure the feature distance. represents the anchor sample, represents the positive sample, and The identity tag is the same as represents negative samples, and The identity labels are different;

[0123] S82, let Y in the r∈[1,N]th sample BNNecks The i∈[1,P]th eigenvector of i,r , N represents the batch size. Given the true identity label y i,r , let the eigenvector q i,r Identity y i,r The probability of ∈[1,Q] is Q represents the number of identities in the training set, and the cross entropy loss function is used to calculate the feature Y BNNecks For supervision, the cross entropy loss function is as follows:

[0124]

[0125] S83. The triple loss function and the cross entropy loss function are combined into a joint loss function as follows:

[0126] L=λL triplet +(1-λ)L cross_entropy

[0127] Among them, L triplet represents the triple loss function, L cross_entropy Represents the cross entropy loss function, λ is a hyperparameter, and represents the weight value of the triplet loss function.

[0128] Corresponding to the gait spatiotemporal feature extraction method based on Transformer and graph convolution in this application, this application also provides a gait spatiotemporal feature extraction system based on Transformer and graph convolution, including: a data enhancement module, a data preprocessing module, a Transformer-based feature extraction module, a graph convolution-based feature module, a batch normalization module, a feature splicing module, a feature mapping module and a loss function supervision module, wherein:

[0129] The data enhancement module is used to obtain a human skeleton sequence and perform data enhancement processing on the obtained human skeleton sequence to obtain a human skeleton sequence Y after data enhancement. DA ;

[0130] The data preprocessing module is used to process the human skeleton sequence Y after data enhancement. DA Perform data preprocessing to obtain the preprocessed gait skeleton sequence Y Process ;

[0131] The Transformer-based feature extraction module is used for the Transformer-based feature extraction network to extract the gait skeleton sequence Y after preprocessing. Process Perform feature extraction and obtain output feature Y T ;

[0132] The graph convolution-based feature module is used for the feature extraction network based on graph convolution to extract the gait skeleton sequence Y after preprocessing. Process Perform feature extraction and obtain output feature Y G ;

[0133] The batch normalization module is used to transform the output feature Y T And the output feature Y G Perform batch normalization processing respectively to obtain feature Y T_BNNecks and feature Y G_BNNecks ;

[0134] The feature splicing module is used to output feature Y T And the output feature Y G Perform splicing to obtain the spliced ​​feature Y Concat ; Then feature Y T_BNNecks and feature Y G_BNNecks Perform splicing to obtain the spliced ​​feature Y BNNecks ;

[0135] The feature mapping module is used to map the concatenated features Y Concat Perform feature mapping operation to obtain the skeleton spatiotemporal feature Y FCs ;

[0136] The loss function supervision module uses a joint loss function consisting of a triplet loss function and a cross entropy loss function to calculate the skeleton spatiotemporal feature Y FCs and feature Y BNNecks Provide supervision to improve the discriminative ability of skeleton features.

[0137] As for the embodiment of the present invention, since it corresponds to the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the above embodiment, which will not be described in detail here.

[0138] Example

[0139] The performance comparison of the proposed method and existing typical skeleton-based methods on the GREW dataset is shown in Table 1. It should be noted that since the original papers of the GaitTR, GaitGraph, and GaitGraph2 methods did not provide their performance data on the GREW dataset, the data listed in Table 1 for comparison are directly quoted from the relevant results in the original paper of the GPGait method.

[0140] Table 1 Performance comparison of this method and existing typical methods based on skeleton features on the GREW dataset

[0141]

[0142] As shown in Table 1, the Rank-1 performance of the proposed method on the GREW dataset surpasses the existing typical skeleton feature-based methods GaitGraph, GaitGraph2, GaitTR and GPGait, fully demonstrating the superior performance of the proposed method among the skeleton feature-based methods.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gait spatiotemporal feature extraction method based on Transformer and graph convolution, characterized by: include: S1. Obtain a human skeleton sequence and perform data enhancement on the obtained human skeleton sequence to obtain a human skeleton sequence Y after data enhancement. DA ; S2, human skeleton sequence Y after data enhancement DA Perform data preprocessing to obtain the preprocessed gait skeleton sequence Y Process ; S3, Transformer-based feature extraction network, the preprocessed gait skeleton sequence Y Process Perform feature extraction and obtain output feature Y T ; S4, a feature extraction network based on graph convolution, is used to extract the preprocessed gait skeleton sequence Y Process Perform feature extraction and obtain output feature Y G ; S5. Output feature Y T And the output feature Y G Perform batch normalization processing respectively to obtain feature Y T_BNNecks and feature Y G_BNNecks ; S6. Output feature Y T And the output feature Y G Perform splicing to obtain the spliced ​​feature Y Concat ; Then feature Y T_BNNecks and feature Y G_BNNecks Perform splicing to obtain the spliced ​​feature Y BNNecks ; S7, the spliced ​​feature Y Concat Perform feature mapping operation to obtain the skeleton spatiotemporal feature Y FCs ; S8. Use the joint loss function composed of the triple loss function and the cross entropy loss function to calculate the skeleton spatiotemporal feature Y FCs and feature Y BNNecks Provide supervision to improve the discriminative ability of skeleton features.

2. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S1 specifically includes: S11. Given the input skeleton sequence Y, Where N represents the batch size, T represents the length of the contour sequence, V represents the number of joint points of each skeleton, and 3 represents the number of input skeleton channels; S12. Set a threshold value ranging from 0 to 1. When the generated random number is greater than the threshold, the order of the input gait skeleton sequence is flipped. The gait skeleton sequence after data enhancement is Y DA , 3. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S2 specifically includes: S21, let the human skeleton after data enhancement be J, Slice the channels of the human skeleton, retain the joint coordinate channels, remove the confidence channels, and obtain the human skeleton J1. S22, set the joint feature to the joint point coordinates J1[:,:,i], and use the nose joint point as the reference point to determine the relative joint coordinates J1[:,:,i]-J1[:,:,i nose ], where i nose Indicates the nose joint point; S23, obtain velocity features by first-order frame difference J1[:,t+1,:]-J1[:,t,:] and second-order frame difference J1[:,t+2,:]-J1[:,t,:]; S24, set the bone feature to J1[:,:,i]-J1[:,:,i adj ], where i adj Represents the adjacent joint of the i-th joint point; S25, combining the joint features, velocity features, and bone features to form the human skeleton J2 after data preprocessing. The gait skeleton sequence after data preprocessing is Y Process , 4. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S3 specifically includes: S31, using GaitTR network as Transformer-based feature extraction network; S32, the pre-processed gait skeleton sequence Y Process Input to GaitTR network, output feature Y T , C represents the output channel dimension.

5. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S4 specifically includes: S41. Use GaitGraph network as a feature extraction network based on graph convolution; S42, replacing the bottleneck residual block in the GaitGraph network with a basic residual block; S43, the pre-processed gait skeleton sequence Y Process Input to GaitGraph network, output feature Y G , C represents the output channel dimension.

6. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S5 specifically includes: S51, the output feature Y of the Transformer-based feature extraction network T After batch normalization, feature Y is obtained T_BNNecks , Q represents the number of pedestrians identified by gait in the training set; S52, extract the output feature Y of the feature extraction network based on graph convolution G After batch normalization, feature Y is obtained G_BNNecks , 7. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S6 specifically includes: S61, through the splicing operation, the output feature Y of the Transformer-based feature extraction network T And the output feature Y of the graph convolution-based feature extraction network G Fusion is performed to obtain the spliced ​​feature Y Concat ,in, The calculation process is as follows: AND Concat =Concat(And T ,AND G ) Among them, Concat(·) represents the feature concatenation operation; S62, through the splicing operation, the output feature Y of the Transformer-based feature extraction network T Feature Y after batch normalization T_BNNecks And the output feature Y of the graph convolution-based feature extraction network G Feature Y after batch normalization G_BNNecks Fusion is performed to obtain the spliced ​​feature Y BNNecks , The calculation process is as follows: AND BNNecks =Concat(And T_BNNecks ,AND G_BNNecks )。 8. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S7 specifically includes: S71, using ReLU activation function, the concatenated feature Y Concat Perform nonlinear transformation operations; S72, then use the parallel fully connected layer to perform feature mapping to obtain the skeleton spatiotemporal feature Y FCs , The calculation process is as follows: AND FCs =FCs(ReLU(Y concat )) Among them, FCs(·) represents the operation of using parallel fully connected layers to perform feature mapping, and ReLU(·) represents the operation of using activation functions to perform nonlinear transformation on features.

9. The method for extracting spatiotemporal features of gait based on Transformer and graph convolution according to claim 1, characterized in that: Step S8 specifically includes: S81, let the skeleton spatiotemporal feature Y in the j∈[1,M]th valid triplet FCs The i-th eigenvector in is v i,j , The skeleton spatiotemporal feature Y is analyzed using the triplet loss function FCs For supervision, the triplet loss function is as follows: Among them, M represents the number of valid triplets in the batch, and P represents the skeleton spatiotemporal feature Y FCs The number of eigenvectors in , [·] + Indicates a positive function, margin indicates the minimum distance difference between the preset positive and negative sample pairs, and the two-norm is used to measure the feature distance. represents the anchor sample, represents the positive sample, and The identity tag is the same as represents negative samples, and The identity labels are different; S82, let Y in the r∈[1,N]th sample BNNecks The i∈[1,P]th eigenvector of i,r , N represents the batch size; given the true identity label y i,r , let the eigenvector q i,r Identity i,r The probability of ∈[1,Q] is Q represents the number of identities in the training set, and the cross entropy loss function is used to calculate the feature Y BNNecks For supervision, the cross entropy loss function is as follows: S83. The triple loss function and the cross entropy loss function are combined into a joint loss function as follows: L=λL triplet +(1-λ)L cross_entropy Among them, L triplet represents the triple loss function, L cross_entropy Represents the cross entropy loss function, λ is a hyperparameter, and represents the weight value of the triplet loss function.

10. A gait spatiotemporal feature extraction system based on Transformer and graph convolution implemented based on the gait spatiotemporal feature extraction method based on Transformer and graph convolution according to any one of claims 1 to 9, characterized in that: include: Data enhancement module, data preprocessing module, Transformer-based feature extraction module, graph convolution-based feature module, batch normalization module, feature splicing module, feature mapping module and loss function supervision module, among which: The data enhancement module is used to obtain a human skeleton sequence and perform data enhancement processing on the obtained human skeleton sequence to obtain a human skeleton sequence Y after data enhancement. DA ; The data preprocessing module is used to process the human skeleton sequence Y after data enhancement. DA Perform data preprocessing to obtain the preprocessed gait skeleton sequence Y Process ; The Transformer-based feature extraction module is used for the Transformer-based feature extraction network to extract the gait skeleton sequence Y after preprocessing. Process Perform feature extraction and obtain output feature Y T ; The graph convolution-based feature module is used for the feature extraction network based on graph convolution to extract the gait skeleton sequence Y after preprocessing. Process Perform feature extraction and obtain output feature Y G ; The batch normalization module is used to transform the output feature Y T And the output feature Y G Perform batch normalization processing respectively to obtain feature Y T_BNNecks and feature Y G_BNNecks ; The feature splicing module is used to output feature Y T And the output feature Y G Perform splicing to obtain the spliced ​​feature Y Concat ; Then feature Y T_BNNecks and feature Y G_BNNecks Perform splicing to obtain the spliced ​​feature Y BNNecks ; The feature mapping module is used to map the concatenated features Y Concat Perform feature mapping operation to obtain the skeleton spatiotemporal feature Y FCs ; The loss function supervision module uses a joint loss function consisting of a triplet loss function and a cross entropy loss function to calculate the skeleton spatiotemporal feature Y FCs and feature Y BNNecks Provide supervision to improve the discriminative ability of skeleton features.

Images