A progressive learning gait recognition method based on memory enhancement

Abstract

The present invention discloses a progressive learning gait recognition method based on memory enhancement, which allows the model to learn better feature expression by progressively learning the difficult and easy samples in the gait data from easy to difficult. The specific steps of the method are as follows: obtaining gait data; defining the optimization target; modeling the difficulty of the current sample; determining the current sample weight according to the sample difficulty and the training progress; using the memory to assist in constraining the feature space; iterative training. The present invention is suitable for complex scenes of gait recognition and has good effect and robustness in the face of various walking conditions.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a progressive learning gait recognition method based on memory enhancement. Background Art

[0002] Gait recognition is defined as the following problem: identifying the specific walking pattern of pedestrians from a video sequence containing pedestrians walking and identifying the pedestrians. Gait recognition is also widely used in many fields, such as smart cities, security systems, video retrieval, etc. In recent years, the main work has been to achieve better gait recognition performance by modeling the human body, but has ignored the two essential difficult sample problems in gait recognition.

[0003] Due to the effectiveness of statistical modeling, learning-based methods are gradually being applied to gait recognition tasks. Existing deep learning-based methods output pedestrian identities by inputting video sequences. Direct end-to-end deep learning can solve basic feature expressions, but the difficult intra-class and inter-class samples in gait recognition bring difficulty to the learning itself and become one of the bottlenecks in the development of gait recognition performance. Summary of the invention

[0004] To solve the above problems, the purpose of the present invention is to solve the problem of difficult samples in gait recognition and provide a progressive learning gait recognition method based on memory enhancement. In the present invention, first, a method for modeling sample difficulty is provided, and then the weight of the sample is determined in combination with the training progress and sample difficulty, thereby completing a progressive training process from easy to difficult; in addition, this method also uses memory to supervise the entire training process, thereby reducing the difficulty in the learning process.

[0005] To achieve the above object, the technical solution of the present invention is:

[0006] A progressive learning gait recognition method based on memory enhancement comprises the following steps:

[0007] S1. Obtain a training data set for training a gait recognition model;

[0008] S2. Model the sample difficulty in the training data set, and determine the sample weight in the training data set according to the training progress and sample difficulty;

[0009] S3, based on the triplet loss constraint, combining the memory to perform auxiliary constraints on the feature space, using the training data set to iteratively train the gait recognition model, and performing weighted correction on the distance between the positive sample and the negative sample relative to the anchor point using the sample weight when calculating the triplet loss;

[0010] S4. Use the trained gait recognition model to perform gait recognition on the target gait data.

[0011] Preferably, in S1, the obtained training data set I train It contains gait video sequences collected by different people from different perspectives and under different walking conditions.

[0012] Preferably, in S2, the difficulty of samples in the training data set is modeled based on the distance between each sample and an anchor point. For positive samples, the farther from the anchor point in the feature space, the greater the difficulty; for negative samples, the closer to the anchor point in the feature space, the greater the difficulty.

[0013] Preferably, in step S2, the weight of each sample x in the training data set is combined with the current model training progress and sample difficulty to give the sample a specific weight, and the calculation function is as follows:

[0014]

[0015] Where: σ represents the normalization function, d is the distance calculation function, d(F(x * ),F(x)) represents the relationship between sample x and anchor point x * The distance between t Indicates the model progress which is positively correlated with the current iteration number of model training. Indicates x * is the sample set of anchor points.

[0016] Preferably, the specific method of S3 is as follows:

[0017] S31, set the content of memory M to:

[0018]

[0019] Among them, s is the video sequence S corresponding to the person ID indexed as i in the training data set i F(s) is the feature extracted by the feature extractor F for the video sequence segment s, N s For the video sequence S i The total number of segments in ;

[0020] S32, setting the update strategy of memory M to:

[0021]

[0022] Among them, M[i] t and M[i] t-1 Respectively represent the contents of memory M before and after the update, y B Represents the label set consisting of the person IDs that appeared in the current training batch, y brepresents the person ID label of sample b in the current training batch, B represents the sample set contained in the current training batch, and N B represents the size of the current training batch, F(b) represents the features of sample b after being extracted by feature extractor F, α and β represent the momentum hyperparameters of the two momentum updates respectively;

[0023] S33, setting the supervision signal of the memory M to the overall training process to constrain the inter-class dispersion and intra-class compactness of the feature space, the calculation formula of the memory supervision signal loss is:

[0024]

[0025] Where τ is the temperature coefficient used to balance the characteristic distribution, Represents the average characteristics of the current batch;

[0026] S34. The distances between the positive samples and the negative samples in the triplet relative to the anchor point are multiplied by the sample weights for weighted correction, and the weighted corrected distance value is used instead of the original distance value to calculate the triplet loss, and the sum of the triplet loss and the memory supervision signal loss is used as the total loss. The gait recognition model is iteratively trained using the training data set until the total loss converges to obtain the trained gait recognition model.

[0027] Compared with the existing gait recognition methods, the memory-enhanced progressive learning gait recognition method of the present invention has the following beneficial effects:

[0028] First of all, the memory-enhanced progressive learning gait recognition method of the present invention solves the very essential difficult sample problem in gait recognition, that is, the difficult samples existing in the existing gait data set bring many difficulties to the gait recognition learning process. The present invention solves the difficult sample problem by making the learning objectives of easy and difficult samples more consistent through a progressive learning strategy from easy to difficult.

[0029] Secondly, the present invention proposes a memory-based learning enhancement strategy, which uses auxiliary signals to supervise the learning process and optimize the feature space, thereby reducing the learning difficulty of difficult samples, enhancing the progressive learning process, and better solving the difficult sample problem.

[0030] Finally, the present invention has a significant improvement on mainstream data sets, especially in the most complex clothing-changing scene, with at least 6.4% performance improvement over previous methods. At the same time, the learning strategy and memory of this method can also be well transferred to the previously proposed feature extractor, with considerable performance improvement.

[0031] The memory-enhanced progressive learning gait recognition method of the present invention solves the problem of difficult samples in data in the field of gait recognition, can accurately identify pedestrians from data with high inter-class variance and high intra-class variance, can effectively improve the accuracy and efficiency of gait recognition, and has good application value. For example, in the field of smart urban security, the memory-enhanced progressive learning method of the present invention can perform better than previous methods in complex clothing-changing and backpacking scenarios, thereby better identifying the identity of pedestrians in the security area and providing support for subsequent security tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the present invention;

[0033] Figure 2 It is a progressive learning schematic diagram of the present invention;

[0034] Figure 3 It is a schematic diagram of the network structure of the present invention;

[0035] Figure 4 It is a schematic diagram of memory update of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0038] refer to Figure 1 In a preferred embodiment of the present invention, a progressive learning gait recognition method based on memory enhancement is provided, comprising the following steps:

[0039] S1. Obtain a training data set for training a gait recognition model.

[0040] In this embodiment, the obtained training data set I train It contains gait video sequences collected from different persons (with unique person IDs) at different viewing angles and under different walking conditions. Training Dataset I train This can be achieved using existing datasets such as CASIA-B and OU-MVLP.

[0041] S2. Model the sample difficulty in the training data set and determine the sample weights in the training data set based on the training progress and sample difficulty.

[0042] In this embodiment, the so-called difficulty of the samples in the modeling training data set refers to the difficulty of each sample in the training data set for the gait recognition learning task. The triplet consists of an anchor, a positive sample, and a negative sample. Figure 2 As shown, the present invention intends to reduce the learning difficulty of difficult samples through a progressive learning strategy from easy to difficult, and better solve the problem of difficult samples. The present invention models the sample difficulty in the training data set based on the distance between each sample and the anchor point, and the definition of sample difficulty for positive samples and negative samples is different: for positive samples, the farther from the anchor point in the feature space, the greater the difficulty; for negative samples, the closer to the anchor point in the feature space, the greater the difficulty.

[0043] In addition, the present invention also combines the current model training progress with the sample difficulty to obtain the weight of each sample x in the training data set. By controlling the weight of the sample, the learning process from easy to difficult can be achieved. The weight W of each sample x x*,x,t All need to be calculated in combination with the current model training progress and sample difficulty. The calculation function is as follows:

[0044]

[0045] Where: σ represents the normalization function, d is the distance calculation function, d(F(x * ),F(x)) represents the relationship between sample x and anchor point x * The distance between t Indicates the model progress which is positively correlated with the current iteration number of model training. Indicates x * is the sample set of anchor points.

[0046] In the present invention, δ t The form of can be various, and its core is positively correlated with the current iteration number of model training. In this embodiment, the form that can be adopted is Where T is the maximum number of iterations for model training.

[0047] Get the weight W of each sample x above x*,x,tAfter that, the weight can be used in the subsequent training process to correct the distance between the positive and negative samples and the anchor point in the conventional triplet loss. At this point, the training progress can be adjusted to adaptively achieve the distinction between difficult and easy samples, so that the model pays more attention to difficult samples in the later stage of the learning stage, thereby better solving the problem of difficult samples.

[0048] S3. Based on the triplet loss constraint, the memory is combined to perform auxiliary constraints on the feature space, and the gait recognition model is iteratively trained using the training data set. When calculating the triplet loss, the distance between the positive sample and the negative sample relative to the anchor point is weightedly corrected using the sample weight.

[0049] In this embodiment, if Figure 3 As shown, the specific method of step S3 is as follows:

[0050] S31. In order to optimize the feature space and reduce the difficulty of learning difficult samples during the learning process, the auxiliary signal supervision training process of the memory is introduced to make the intra-class feature space more compact and the inter-class feature space more dispersed, providing a better optimization direction for progressive learning. Specifically, in order to facilitate cross-condition and cross-angle retrieval, the content M[i] of the memory M is modeled as the average feature under each person ID:

[0051]

[0052] Among them, 2 is the video sequence S corresponding to the person ID indexed as i in the training data set i F(s) is the feature extracted by the feature extractor F for the video sequence segment s, N s For the video sequence S i The total number of segments in ;

[0053] S32, in order to ensure the validity and robustness of the features in memory, e.g. Figure 4 As shown, the present invention performs a structural alignment update strategy for momentum update on the features in the memory, so that these features are continuously corrected during the training process and have better expression capabilities. The specific update strategy for setting the memory M is as follows:

[0054]

[0055] Among them, M[i] t and M[i] t＝1 Respectively represent the contents of memory M before and after the update, y B Represents the label set consisting of the person IDs that appeared in the current training batch, y b represents the person ID label of sample b in the current training batch, B represents the sample set contained in the current training batch, and NB represents the size of the current training batch, F(b) represents the features of sample b after being extracted by feature extractor F (e.g., CNN), α and β represent the momentum hyperparameters of the two momentum updates respectively;

[0056] S33, set the supervision signal of the memory M to the overall training process, which is used to constrain the inter-class dispersion and intra-class compactness of the feature space to optimize the feature space. The memory supervision signal loss L GSAM The calculation formula is:

[0057]

[0058] Where τ is the temperature coefficient used to balance the characteristic distribution, Represents the average characteristics of the current batch;

[0059] S34, multiplying the distance between the positive sample and the negative sample in the triplet relative to the anchor point by the sample weight to perform weighted correction, and using the weighted corrected distance value instead of the original distance value to calculate the triplet loss L DRPL , and finally use the triple loss L DRPL and the memory supervision signal loss L GSAM The sum is taken as the total loss, and the gait recognition model is iteratively trained using the training data set until the total loss converges to obtain a trained gait recognition model.

[0060] In this step, the so-called weighted correction means that when calculating the triplet loss, the distance between the positive sample and the anchor point and the distance between the negative sample and the anchor point need to be multiplied by the corresponding sample weight to achieve weighted correction, and then the weighted distance value after weighted correction is used instead of the original distance value to calculate the triplet loss L DRPL The form of triplet loss belongs to the prior art and will not be described in detail. In practical applications, the weighted correction process can be implemented in matrix form, that is, the original distance matrix D is re-weighted by the weighting matrix A to obtain the weighted distance matrix D'.

[0061] In the training process of S3, since it is combined with the training progress according to the learning order of the difficulty of the samples from easy to difficult, and the auxiliary signal of the memory is used for supervision in the whole training process, it is a dynamic reweighted progressive learning method. It should be noted that the core of the present invention is to improve the progressive learning method of the gait recognition model, but the gait recognition model targeted can be any gait recognition model in the prior art, and there is no limitation on this.

[0062] S4. Use the trained gait recognition model to perform gait recognition on the target gait data.

[0063] It should be noted that during training, after the method determines the feature extractor, a dynamic reweighted progressive learning method and memory can be added to it, thereby realizing a progressive learning gait recognition method based on memory enhancement. However, during testing or actual application, only the features extracted by the original feature extractor are needed for gait recognition. When performing actual gait recognition, it is only necessary to input the video sequence of pedestrians into the trained gait recognition model, which can output features. By comparison, the person who is most similar to the input probe to be queried is found as the recognition result, and then the identity of the pedestrian is output.

[0064] The method provided by the present invention can effectively achieve better learning effects in mainstream feature extractors. Subsequent examples fully verify the effectiveness of the method and its components with experimental results on mainstream datasets CASIA-B and OU-MVLP.

[0065] The above method is applied to a specific embodiment below so that those skilled in the art can better understand the effect of the present invention.

[0066] Example

[0067] The implementation method of this embodiment is as described above, and the specific steps will not be elaborated in detail. The following only shows its effect based on case data. The present invention is implemented on two data sets with true value annotations according to certain settings, namely:

[0068] CASIA-B dataset: contains 124 person IDs, 11 perspectives, and 13,640 gait sequences. This dataset will be mainly used as a test dataset.

[0069] OU-MVLP dataset: contains 10,307 person IDs and 14 perspectives, and is the largest academic dataset.

[0070] The gait recognition model trained in this example is a Multiple-Temporal-Scale 3D Convolutional Neural Network, and the trained gait recognition model is denoted as GaitMPL.

[0071] This example uses the GaitMPL method trained by this method to compare with other mainstream methods, and trains and tests on the CASIA-B and OU-MVLP datasets to verify the effectiveness of this method. The recognition accuracy of the final test results is shown in Tables 1 and 2. The data in the table shows that the virtual dataset proposed by the present invention is used as a pre-training, and the performance of the Rank-1 index of this method on the CASIA-B dataset and the OU-MVLP dataset is significantly improved, especially in the most complex clothing-changing scene. At the same time, there is also a considerable improvement under average conditions, which is more in line with the needs of cross-condition retrieval under real conditions, and provides a good foundation for subsequent applications.

[0072] Table 1 Test results of this example on the CASIA-B dataset

[0073]

[0074] Table 2 Test results of this example on the OU-MVLP dataset

[0075]

[0076] It should be noted that in Tables 1 and 2 above, NM represents a normal scene, BG represents a backpack scene, and CL represents a dressing scene. CNN-LB, GaitSet, GaitPart, GLN, GaitNet, MT3D, and DynamicGait are all gait recognition models in the prior art. For details, please refer to the following prior art documents (where the symbol Indicates that the corresponding data is the reproduction result of this example):

[0077] CNN-LB: Wu, Z., Huang, Y., Wang, L., Wang, X., & Tan, T. (2016). A comprehensive study on cross-view gait based human identification with deep cnns. IEEE transactions on pattern analysis and machine intelligence, 39(2), 209-226.

[0078] GaitSet:Chao,H.,He,Y.,Zhang,J.,&Feng,J.(2019,July).Gaitset:Regardinggait as a set for cross-view gait recognition.In Proceedings of the AAAIconference on artificial intelligence(Vol.33,No.01,pp.8126-8133).

[0079] GaitPart:Fan,C.,Peng,Y.,Cao,C.,Liu,X.,Hou,S.,Chi,J.,...&He,Z.(2020).Gaitpart:Temporal part-based model for gait recognition.In Proceedings ofthe IEEE / CVF conference on computer vision and pattern recognition(pp.14225-14233).

[0080] GLN:Hou,S.,Cao,C.,Liu,X.,&Huang,Y.(2020,August).Gait lateral network:Learning discriminative and compact representations for gait recognition.InEuropean Conference on Computer Vision(pp.382-398).Springer,Cham.

[0081] GaitNet:Song,C.,Huang,Y.,Huang,Y.,Jia,N.,&Wang,L.(2019).Gaitnet:Anend-to-end network for gait based human identification.Pattern recognition,96,106988.

[0082] MT3D:Lin,B.,Zhang,S.,&Bao,F.(2020,October).Gait Recognition withMultiple-Temporal-Scale 3D Convolutional Neural Network.In Proceedings of the28th ACM International Conference on Multimedia(pp.3054-3062).

[0083] DynamicGait: Wu, H., Tian, ​​J., Fu, Y., Li, B., & Li, X. (2020). Condition-AwareComparison Scheme for Gait Recognition. IEEE Transactions on Image Processing, 30, 2734-2744.

[0084] Through the above technical solutions, the present invention implements a progressive learning gait recognition method based on memory enhancement, providing a method for simply and efficiently processing difficult samples in gait recognition. The present invention can achieve the best performance on mainstream data sets, and its components can be flexibly inserted into existing methods, and can perform well under various cross-view and cross-scenario conditions.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A progressive learning gait recognition method based on memory enhancement, characterized in that: The following steps are involved: S1. Obtain a training data set for training a gait recognition model; S2. Model the sample difficulty in the training data set, and determine the sample weight in the training data set according to the training progress and sample difficulty; S3, based on the triplet loss constraint, combining the memory to perform auxiliary constraints on the feature space, using the training data set to iteratively train the gait recognition model, and performing weighted correction on the distance between the positive sample and the negative sample relative to the anchor point using the sample weight when calculating the triplet loss; S4, using the trained gait recognition model to perform gait recognition on the target gait data; The specific method of S3 is as follows: S31, set the content of memory M to: Among them, s is the video sequence S corresponding to the person ID indexed as i in the training data set i F(s) is the feature of the video sequence segment s after being extracted by the feature extractor F, N s For the video sequence S i The total number of segments in ; S32, setting the update strategy of memory M to: Among them, M[i] t and M[i] t-1 Respectively represent the contents of memory M before and after the update, y B Represents the label set consisting of the person IDs that appeared in the current training batch, y b represents the person ID label of sample b in the current training batch, B represents the sample set contained in the current training batch, and N B represents the size of the current training batch, F(b) represents the features of sample b after being extracted by feature extractor F, α and β represent the momentum hyperparameters of the two momentum updates respectively; S33, setting the supervision signal of the memory M to the overall training process to constrain the inter-class dispersion and intra-class compactness of the feature space, the calculation formula of the memory supervision signal loss is: Where τ is the temperature coefficient used to balance the characteristic distribution, Represents the average characteristics of the current batch; S34. The distances between the positive samples and the negative samples in the triplet relative to the anchor point are multiplied by the sample weights for weighted correction, and the weighted corrected distance value is used instead of the original distance value to calculate the triplet loss, and the sum of the triplet loss and the memory supervision signal loss is used as the total loss. The gait recognition model is iteratively trained using the training data set until the total loss converges to obtain the trained gait recognition model.

2. The memory-enhanced progressive learning gait recognition method according to claim 1, characterized in that: In S1, the obtained training data set I train It contains gait video sequences collected by different people from different perspectives and under different walking conditions.

3. The memory-enhanced progressive learning gait recognition method according to claim 2, characterized in that: In S2, the difficulty of samples in the training data set is modeled based on the distance between each sample and an anchor point. For positive samples, the farther away from the anchor point in the feature space, the greater the difficulty; for negative samples, the closer to the anchor point in the feature space, the greater the difficulty.

4. The memory-enhanced progressive learning gait recognition method according to claim 3, characterized in that: In step S2, the weight of each sample x in the training data set is calculated in combination with the current model training progress and sample difficulty, and the calculation function is as follows: Where: σ represents the normalization function, d is the distance calculation function, d(F(x * ),F(x)) represents the relationship between sample x and anchor point x * The distance between t Indicates the model progress which is positively correlated with the current number of iterations of model training. Indicates x * is the sample set of anchor points.

Images