A joint learning gait representation method based on limited supervision information

Abstract

The application discloses a kind of joint learning gait representation method based on limited supervision information, belong to deep learning and pattern recognition technical field;Propose a kind of simultaneously using semi-supervised and self-supervised learning, the new method of joint learning gait space-time feature representation, contain two main branches:1) in semi-supervised branch, by introducing sequence level morphological disturbance in the input gait sequence, increase the diversity of data, and learn robust feature representation using Mean Teacher architecture;2) in self-supervised branch, two auxiliary tasks related to gait recognition task are defined, including pipe mask reconstruction and segment order prediction, to enhance the ability of network to learn relevant features.The method of the application applies semi-supervised and self-supervised learning to the gait recognition task, can learn specific task features from limited labeled gait sequences, and can mine powerful general features from unlabeled data, and the recognition performance is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning and pattern recognition, and particularly relates to a joint learning gait representation method based on limited supervision information. BACKGROUND

[0002] Gait is a unique biometric feature, and the basic task of gait recognition is to learn the invariable features from the time-varying changes of human walking, which does not require high-resolution images and can be performed at a long distance without the awareness of the person being identified. Therefore, gait recognition plays a key role in security monitoring, crime prevention, and maintenance of social public safety. However, in real-world scenarios, they are often affected by covariate factors such as occlusion by obstacles, pedestrians wearing coats and backpacks, and changes in the recognition angle, which makes them have large intra-object differences, which often has a great impact on the performance of gait recognition.

[0003] A large number of deep learning-based gait recognition methods provide an effective solution to alleviate the above problems, one of which is a model-based method, which extracts the posture information and body skeleton structure of human walking to complete the recognition task, but it is difficult to detect the key points of the body from the low-pixel gait images collected at a long distance; the other is an appearance-based method, also known as a model-free or holistic method, such as compressing all gait contour sequences into a single gait template, but a large amount of time information and fine-grained spatial information will be lost in the compression process. Almost all methods rely on labeled gait sequence data, and labeling a large number of gait sequences is a tedious task, which not only consumes a lot of time and effort of experts, but also introduces uncertainty in the gait data between annotators during the labeling process. In order to reduce the dependence on labeled gait data, it is urgent to develop a gait recognition algorithm that only uses a small amount of labeled data or no labeled data.

[0004] Semi-supervised learning is a class of algorithms that attempts to learn representations from a large amount of unlabeled and a small amount of labeled samples, which usually assumes that these data are sampled from the same or similar distribution, where pseudo-labeling and consistency regularization loss are two important semi-supervised learning methods. The pseudo-labeling method uses neural networks to train on labeled data, first extracts the features of unlabeled data, and then uses clustering methods to induce approximate classes and assign pseudo-labels to them. A key problem of this method is that the number of classes of unlabeled data is usually unknown, so it is difficult to set appropriate hyperparameters for clustering algorithms, and their performance is often not as good as consistency regularization loss methods. Consistency regularization loss is used to measure the difference between the model's predictions when the unlabeled data is perturbed, common methods include ΠModel, Temporal Ensemble, Mean Teacher. At present, a large amount of work in gait recognition tasks is completed under supervised data, but as the accumulation of gait data, the problem of time-consuming and laborious data labeling begins to appear, so it is more important to use semi-supervised learning to complete the recognition task.

[0005] Self-supervised learning is a new paradigm of unsupervised learning, which can learn the inherent structure of unlabeled data by defining various auxiliary tasks, and capture more rich high-level semantic information from it. For image data, common auxiliary tasks include predicting the relative position of image patches, reconstructing missing pixel regions, coloring grayscale images, etc. Compared with image data, the special attribute of video data is the information in the time dimension, and some works often use frame order prediction, order verification, and application tracking to capture the appearance changes of moving objects as auxiliary tasks to explore the feature representation of unlabeled video data. These auxiliary tasks do not directly use semantic labels, but they provide effective supervision signals. In the gait recognition task, appearance-based methods are very sensitive to covariate factors such as view changes, clothing changes, occlusions, and carrying conditions, so how to use self-supervised learning methods to design an auxiliary task that can effectively cope with these covariate factors becomes an important challenge.

[0006] Therefore, it is urgent to design and develop a gait recognition algorithm based on existing deep learning methods, which only uses a small amount or no labeled data, reduces the dependence on a large amount of labeled data, and improves the recognition accuracy. SUMMARY

[0007] The application aims at solving the problems in the prior art, providing a joint learning gait representation method based on limited supervision information, simultaneously learning gait space-time feature representation by using semi-supervised and self-supervised learning, and establishing an end-to-end gait recognition framework which can pay attention to global visual information of gait and can also pay attention to details of local regions; introducing morphological disturbance in the semi-supervised branch model and constructing an auxiliary task in the self-supervised branch model can obtain more abundant deep essential features from a large amount of unlabeled gait data and improve recognition accuracy.

[0008] To achieve the above-mentioned purpose, the application adopts the following technical scheme: a joint learning gait representation method based on limited supervision information, comprising the following steps:

[0009] S1, constructing the relationship between input data and output data in the network:

[0010] Given a gait contour sequence with a length of k In the formula, Where h and w represent the height and width of the frame respectively, The feature matrix is represented as as input data;

[0011] Suppose that N candidate subjects are defined as y i ,i∈1,2,…,N, where y i is the i-th subject, and the output data is represented as an n-dimensional vector: In the formula, That is The possibility of belonging to y i ; further assuming that y is the identity ID of the input data , y is expressed as:

[0012]

[0013] S2, optimizing the semi-supervised branch model:

[0014] Based on the loss optimization model student model f(x, θ, η) and teacher model f(x, θ', η') of semi-supervised, a sequence-level morphological disturbance is added to the model, and the gait contour sequence obtained from the above step S1 is randomly selected μ frame gait contour map, where μ is a hyperparameter, and for the selected μ frame gait map, a morphological disturbance scheme is adopted, first, the contour edge is detected to detect the pixel points on the contour edge, then the pixel points on the contour edge are randomly discarded according to a certain probability, or new pixel values are added in the neighborhood of the pixel points, the value of the new pixel value should be the average value of the neighborhood pixels, and the optimized semi-supervised branch model is obtained.

[0015] S3, constructing auxiliary tasks in the self-supervised branch model that are highly relevant to the gait recognition task:

[0016] 1) Pipeline mask reconstruction: simulating the case where the identified subject is blocked by obstacles in a real scene, missing key information, covering the same area of the gait frame along the time dimension, and then reconstructing, so that the model learns high-level semantic information;

[0017] The specific steps of pipeline mask reconstruction include: 1) Gait clip: using a time sampling strategy with a step size of τ to extract a clip consisting of n frames from the gait contour sequence , denoted as c, and sample the gait clip data; 2) Pipeline mask: using the pipeline mask strategy, setting the value of the step size τ on the CASIA-B and OUMVLP datasets, respectively, and dividing the clip data c into multiple non-overlapping cubes according to the set value of the step size τ; 3) Token embedding: each cube is encoded using the token embedding method, and each token is mapped to the channel dimension; 4) Reconstructing the gait clip: using a high masking rate strategy to discard part of the cubes, using the unmasked cubes as the input of the base module for training, and finally reconstructing the discarded cubes using the masking features and learnable mask tokens obtained by network training;

[0018] 2) Clip order prediction: the original gait sequence is shuffled, and the clip consisting of gait frames is used to predict the order of the gait sequence,

[0019] The specific steps of order prediction include: 1) First, uniformly sample N ordered clips from the complete gait sequence to form a tuple C = <c1, c2,..., cn>, where n is the number of frames, and then randomly shuffle the order of the clips in the tuple C, and the tuple C is input into the neural network to learn high-level semantic features, and the probability distribution of different orders is output, and the true order is used as the target for training and optimization; n

[0020] S4, jointly establishing the semi-supervised branch model and the self-supervised branch model on the gait recognition network GaitGL, and optimizing training to obtain a joint loss function:

[0021] The base component for feature extraction in the gait recognition network GaitGL adopts the R(2+1)D structure, including M i two-dimensional convolution kernels with a size of N i-1 ×1×d×d and N i one-dimensional convolution kernels with a size of M i ×t×d×d;

[0022] The formula of the joint loss function is: L​total = L tri + λ cont L cont + λ recons L recons + λ order L order , where L totall is the total loss, λ cont is the weighting term of the semi-supervised loss, λ recons and λ order are the weighting terms of the self-supervised loss, L tri is the supervised recognition loss in the semi-supervised branch, L cont is the consistency regularization loss of the Mean Teacher framework in the semi-supervised branch, L recons is the loss of the pipe mask reconstruction task in the self-supervised branch, and L order is the loss of the segment order prediction task in the self-supervised branch.

[0023] In the step S2, θ and θ' in the student model f(x, θ, η) and the teacher model f(x, θ', η') represent the network weights of the model, and η and η' represent the disturbance noise introduced into the model.

[0024] The network weight θ is optimized by the supervised loss of the labeled sample, the network weight θ' is updated by the exponential moving average EMA of the student model sequence of different training iterations, and the calculation formula of the network weight θ' is θ' t = αθ' t-1 + (1-α)θ t , where t represents the number of training iterations, and α is a smoothing coefficient set to 0.999.

[0025] In the step S3, the N ordered segments are uniformly sampled from the gait sequence in a fixed interval of m frames without overlapping; the size of a segment is 1×c×h×w, where c is the number of frame channels, and h and w represent the height and width of the frame, respectively.

[0026] In the step S3, the pipe mask strategy is mathematically represented as: II[p x,y,· ∈Ω]~Bernoulli(ρ mask ), where p x,y,. represents the index of the token, Ω is the set of tokens that are masked, they conform to the Bernoulli distribution, ρ mask represents the mask rate, and different times t share the same value.

[0027] In the step S4, in the semi-supervised branch, the supervised recognition loss L triis optimized using a triplet loss, a consistency regularization loss L cont is a mean square error loss, MSE loss; in the self-supervised branch, the loss L of the pipe mask reconstruction task recons is a mean square error loss, MSE loss, the loss L of the segment order prediction task order is a cross-entropy loss.

[0028] The beneficial effects of the present application are:

[0029] 1) The method of the present application applies semi-supervised and self-supervised learning to the gait recognition task, and on this basis establishes an end-to-end gait recognition framework, which can focus on the global visual information of the gait and can pay attention to the details of the local area; can learn specific task features from limited labeled gait sequences, and can mine powerful general features from unlabeled data, and the performance is verified through the CASIAB and OU-MVLP two gait data sets, and the experimental results show that in the case of very limited label data, the recognition performance is significantly improved compared with the most advanced supervised method.

[0030] 2) In the semi-supervised branch of the method of the present application, sequence-level morphological disturbance is introduced in the input gait sequence, which increases the diversity of samples while prompting the model to reduce the sensitivity to the appearance changes of the recognition subject; and the MeanTeacher architecture is used to improve the robustness and generalization of the training model.

[0031] 3) In the self-supervised branch of the method of the present application, two auxiliary tasks highly related to the gait recognition task are designed: 1) Pipe mask reconstruction: along the time dimension, the pixels in the same area of the gait frame are covered and then reconstructed, which can simulate the case where the recognition subject is blocked by an obstacle in the real scene and the key information is missing. 2) Segment order prediction: the gait segment composed of multiple frames is randomly shuffled, and then the original correct order is predicted to learn the time dimension information of the gait sequence. Through these two auxiliary tasks, the model can adaptively learn key features from both spatial and temporal dimensions. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a principle flowchart of the method of the present application;

[0033] Figure 2 is a mask strategy comparison diagram for comparing the method of the present application with the conventional method;

[0034] Figure 3 is a task flowchart of the pipe mask reconstruction task in the method of the present application;

[0035] Figure 4 is a task flowchart of the segment order prediction task in the method of the present application;

[0036] Figure 5 Architecture diagram of the feature extraction basic component in the method of the present application;

[0037] Figure 6 Comparison result diagram of the method of the present application and other methods on the CASIA-B dataset;

[0038] Figure 7 Comparison result diagram of the method of the present application and other methods on the OUMVLP dataset;

[0039] Figure 8 Result diagram in the ablation experiment of the present application. DETAILED DESCRIPTION

[0040] The present application will be further explained in conjunction with the accompanying drawings and specific embodiments.

[0041] Embodiment: As shown in the accompanying drawings, the present application provides a joint learning gait representation method based on limited supervised information, comprising the following steps: Figures 1-8

[0042] S1, constructing the relationship between the input data and the output data in the network:

[0043] Given a gait profile sequence of length k wherein, wherein h and w represent the height and width of the frame respectively, represents a feature matrix, and as input data;

[0044] Suppose N candidate subjects are defined as y i ,i∈1,2,...,N, wherein y i is the i-th subject, and the output data is represented as an n-dimensional vector: wherein, that is, the possibility of belonging to y i ; further assuming that y is the identity ID of the input data , y is expressed as:

[0045]

[0046] S2, optimizing the semi-supervised branch model:

[0047] ​Based on semi-supervised loss optimization model student model f(x, θ, η) and teacher model f(x, θ', η'), θ, θ' in the model represent the network weights of the model, and η, η' represent the disturbance noise introduced into the model; wherein the network weight θ is optimized by the supervised loss of the labeled sample, the network weight θ' is updated by the exponential moving average EMA of the student model sequence of different training iterations, and the calculation formula of the network weight θ' is: θ t ′=αθ t -1 +(1-α)θ t , wherein t represents the number of training iterations, and a is a smoothing coefficient set to 0.999.

[0048] In the model, a sequence-level morphological disturbance is added to the gait contour sequence obtained from the above step S1 μ-frame gait contour maps are randomly selected from the gait contour sequence, wherein μ is a hyperparameter, and for the selected μ-frame gait contour maps, a morphological disturbance scheme is adopted, first, edge detection is performed on the contour maps to detect the pixel points on the contour edges, then the pixel points on the contour edges are randomly discarded according to a certain probability, or new pixel values are added to the neighborhood of the pixel points, the value of the new pixel value should be the average value of the neighborhood pixels, and an optimized semi-supervised branch model is obtained;

[0049] S3, constructing an auxiliary task highly related to the gait recognition task in the self-supervised branch model:

[0050] 1) Pipe mask reconstruction: simulating the case that the subject to be recognized in the real scene is blocked by an obstacle and some key information is missing, covering the pixels in the same area of the gait frame along the time dimension and then reconstructing, so that the model learns high-level semantic information.

[0051] In the mask reconstruction task of the gait contour sequence, there are usually two problems, namely the redundancy and correlation of information in the time dimension. First, as time goes on, the semantic change between the neighborhood frames of the densely captured gait contour sequence is small, indicating the redundancy of information; if we maintain the original sampling frame rate for training, not only the reconstruction efficiency will be reduced, but also the model will pay more attention to static features, so that the reconstruction task can be well completed at a lower mask rate. In addition, the semantic correlation of consecutive frames in time may exist information leakage in the reconstruction process. As Figure 2 shown, using ordinary frame masks or random masks, the network can directly use the semantic information obtained in the same area of the adjacent frames that are not covered to reconstruct the covered pixels, without truly understanding the higher-level semantic information.

[0052] The specific steps of the pipe mask reconstruction are as follows Figure 3 ​As shown, it includes: 1) Gait segments: a time sampling strategy with a step size of τ is used to extract gait contour sequences. 1) Extract a segment consisting of n frames, denoted as c, and sample the gait segment data; 2) Pipeline masking: Using a pipeline masking strategy, the stride length τ is set to 2 and 4 on the CASIA-B and OUMVLP datasets, respectively. Based on the stride length τ, the segment data c is divided into 2×8×11 non-overlapping cubes; 3) Token embedding: Each cube is encoded using a token embedding method. Therefore, the embedding layer of the multidimensional data obtains... 3D tokens are generated and each token is mapped to a channel dimension. This design can reduce the spatial and temporal dimensions of the input and help alleviate the spatiotemporal redundancy problem in the video; 4) Reconstruct gait segments: A high occlusion rate (70%) strategy is used to discard some cubes. The unoccluded cubes are used as input to the basic module for training. Finally, the discarded cubes are reconstructed together with the occlusion feature map obtained by the network training and the learnable mask tokens.

[0053] The pipeline masking strategy is mathematically represented as: II[p x,y,· ∈Ω]~Bernoulli(ρ mask ), where p x,y,· ρ represents the index of the token, Ω is the set of masked tokens that follow a Bernoulli distribution, and ρ is the index of the token. mask This represents the mask rate, and different times t share the same value.

[0054] Under this mechanism, the neighborhood of the occluded cube is always masked in the time dimension. Therefore, for some cubes, it is impossible to find a corresponding copy in other frames. This encourages the pipeline mask reconstruction task to reason about high-level semantics in order to recover these completely lost cubes, thereby mitigating the problem of information leakage in the mask reconstruction task.

[0055] 2) Segment order prediction: such as Figure 4 The diagram shows the task details. The original gait sequence is shuffled, and the order of the gait sequence is predicted using segments composed of gait frames.

[0056] The specific steps of sequential prediction include: first, uniformly sampling N ordered segments from the complete gait sequence to form a tuple C = <c1,c2,...,c n > where n is the number of frames. After randomly shuffling the order of the segments in tuple C, tuple C is fed into the neural network as input to learn high-level semantic features and output the probability distribution of different orders. The true order will be used as the target for training and optimization.

[0057] The size of a segment is 1×c×h×w, where c is the number of frame channels, and h and w represent the height and width of the frame, respectively. In the experiment, the number of segments was limited to 3. The N ordered segments were uniformly sampled from the gait sequence at fixed intervals of m frames in a non-overlapping manner, which can avoid the model from completing the task of order prediction by comparing low-level features such as texture and color.

[0058] Given that the total length of the training samples is 30 frames, 7 frames are chosen as the length of a segment, and the interval m is set to 4. Assuming a tuple contains three segments, after shuffling, a new tuple C is obtained.<c2,c3,c1> The cross-entropy loss function is used to measure the accuracy of the prediction, as shown below.

[0059] Where y i and p i Let be the probability that the sample belongs to class i in both the correct and predicted order, and C be the number of all possible ordered classes.

[0060] S4. The semi-supervised branch model and the self-supervised branch model are jointly built on the gait recognition network GaitGL, and the joint loss function is obtained through optimization training.

[0061] like Figure 5 As shown, the basic components for feature extraction in the GaitGL gait recognition network adopt an R(2+1)D structure, including M i Each size is N i-1 A two-dimensional convolution kernel of ×1×d×d and N i Each size is M i A one-dimensional convolutional kernel of size ×t×d×d; there are two methods to connect local and global feature maps, one of which is to add them bit by bit, and the other is to concatenate them horizontally.

[0062] The aforementioned modeling of the spatial and temporal dimensions allows for doubling the number of ReLU nonlinear activation layers in the network with the same number of parameters. This enables the model to represent more complex features and significantly improves accuracy. Simultaneously, the decomposition operation reduces the optimization difficulty, making the model more likely to converge.

[0063] To achieve joint learning of gait spatiotemporal representation using semi-supervised and self-supervised methods, a unified framework is used to optimize the proposed joint method in an end-to-end manner. The semi-supervised branch is trained using both labeled and unlabeled data, while the self-supervised branch uses all data but discards their labels.

[0064] In the semi-supervised branch, the supervised recognition loss L tri Optimization is performed using triplet loss, with consistency regularization loss L. contMSE loss; in the self-supervised branch, the loss L of the pipeline mask reconstruction task recons MSE loss, the loss L of the segment order prediction task order is the cross-entropy loss.

[0065] The joint loss function is defined by the weighted sum of the above loss functions, and the formula definition of the joint loss function is: L total = L tri + λ cont L cont + λ recons L recons + λ order L order , wherein L total is the total loss, λ cont is the weighted item of the semi-supervised loss, λ recons and λ order are the weighted items of the self-supervised loss, L tri is the supervised recognition loss in the semi-supervised branch, L cont is the consistency regularization loss of the Mean Teacher framework in the semi-supervised branch, L recons is the loss of the pipeline mask reconstruction task in the self-supervised branch, and L order is the loss of the segment order prediction task in the self-supervised branch.

[0066] Experimental verification: The performance of the method of the present application is evaluated on two commonly used gait datasets CASIA-B and OUMVLP.

[0067] CASIA-B is a cross-view gait dataset composed of 124 subjects, a total of 113640 video data. Three different walking states are considered for each subject, a total of 10 gait type sequences. That is, 6 normal walking (NM) sequences, 2 coat walking (CL) sequences and 2 bag walking (BG) sequences. Each sequence is collected from 0° to 180°, with an increment of 18°, from 11 different angles.

[0068] OUMVLP is one of the largest cross-view public gait recognition databases in the world, which contains 10307 subjects. Each subject has 14 angles, which are evenly distributed between [0°, 90°] and [180°, 270°] with an interval of 15°. Each angle has 2 gait sequences (#00 and #01), a total of 28 gait sequences.

[0069] The present application is divided into three different scales of training modes according to the size of the training data in the CASIA-B dataset, namely small sample training (ST), medium sample training (MT) and large sample training (LT). Under the ST setting, the first 24 subjects are used for training, and the remaining 100 subjects are used for testing. Under the MT setting, the first 62 subjects are used for training, and the remaining 62 subjects are used for testing. Under the LT setting, the first 74 subjects are used for training, and the remaining 50 subjects are used for testing. In the test phase, the first 4 sequences of each subject under the NM condition, i.e. NM#1-4, are regarded as the registration set, and the remaining 6 sequences are divided into 3 probe set subsets, i.e. NM subset NM#5-6, BG subset BG#1-2 and CL subset CL#1-2. In the OUMVLP dataset, the first 5154 subjects are used for training, and the remaining 5154 subjects are used for testing. In the test phase, the sequence with index #01 is regarded as the registration set, and the sequence with index #00 is used as the probe set.

[0070] 1. Comparison experiment with full supervision method:

[0071] The present application method is compared with the existing full supervision gait recognition method including GaitSet, GaitPart, MGAN, GaitGL and GaitGL-R(2+1)D on the CASIA-B dataset, and compared with GEINet, GaitSet, GaitPart, GLN, GaitGL, GaitGL-R(2+1)D method on the performance of OUMVLP dataset. The experiment is carried out under two different settings: (1) full supervision method. All labeled data is used for training, i.e. 100% label; (2) SSGait semi-supervised self-supervised joint training method. In the semi-supervised branch, all the labeled data is divided into two parts, x% of the training data retains the label, (100-x)% of the training data label is discarded, and only used as unsupervised data. While in the self-supervised branch, all the data are used without their labels.

[0072] The experimental results are shown in Table 1. When using 100% labeled data, the SSGait method can obtain the best recognition accuracy at almost all viewing angles, and the recognition performance is better than all fully supervised methods. In the LT setting, the recognition accuracy of SSGait under the NM, BG and CL conditions is 97.74%, 94.91% and 83.87%, respectively. Compared with the recognition rates of GaitGL of 97.04%, 94.27% and 82.63%, the recognition rates are improved by 0.7%, 0.64% and 1.24%, respectively. Similarly, in the other MT and ST scales, we also see similar recognition results. The recognition accuracy of SSGait is improved by 0.68%, 0.44%, 1.56% and 0.49%, 0.48%, 1.32% than GaitGL under the three conditions, respectively. Notably, the performance improvement under the CL condition is the highest, which is higher than the performance improvement under the NM and BG conditions, and better performance is obtained. This shows that the SSGait method is more adaptable to changes in the recognition subject's backpack state and external environment than other fully supervised methods when using the same labeled data.

[0073] The performance results of the comparison between the GaitGL-R(2+1)D method and the GaitGL method show that the R(2+1)D convolution structure in the application uses a spatial-temporal split convolution manner, which does not bring a large improvement in recognition performance (even under the MT and ST scales, the recognition performance under the BG condition is decreased by 0.24% and 0.2%, respectively), but this does not mean that the R(2+1)D convolution structure is not suitable for gait recognition tasks. Because it is found in the experiment that the R(2+1)D convolution structure, compared with the traditional 3D convolution structure, only needs fewer iterations to converge to achieve the same recognition performance as GaitGL. Therefore, the spatial-temporal split convolution manner helps to improve the gait recognition efficiency.

[0074] Table 1 is the Rank-1 accuracy (%) value of CASIA-B under all viewing angles, different settings and conditions (excluding the same viewing angle case)

[0075]

[0076] As shown in Table 2 and Figure 6As shown, SSGait method is significantly better than the fully supervised method of STOA with only a small number of label data (20%-40%). Meanwhile, it can also be found from Table 2 that the recognition performance of SSGait method is comparable to that of the fully supervised GaitGL method trained with 100% labels when only 80% of the labels are used. When there are more labels (i.e. 90% and 100%), the recognition performance of SSGait is further improved. In addition, when only 60% of the labels are used, the gap in recognition accuracy between the SSGait method and the GaitGL method using 100% label data is less than 3%.

[0077] Table 2 is a comparison of Rank-1 accuracy (%) of GaitGL, GaitGL-R(2+1D) and SSGait on CASIA-B under LT setting, different label proportions, NM#5-6, BG#1-2 and CL#1-2 conditions

[0078]

[0079] Similarly, Table 3 and Figure 7 The performance of the SSGait method on the OUMVLP dataset is shown, and it can be seen that the method of the present application can achieve the best recognition effect in most cases.

[0080] Table 3 is the Rank-1 accuracy (%) of OUMVLP under 14 probe angles, excluding the same view

[0081]

[0082] 2. Comparison experiment with semi-supervised baseline:

[0083] The recognition performance of various semi-supervised methods, including ΠModel, TemporalEnsemble and Mean Teacher, in the gait recognition task is reported under the LT setting of the CASIA-B dataset. In order to make a fair comparison, all semi-supervised methods use GaitGL-R(2+1)D as the feature extraction base module, and then are trained using 20% and 60% of the labels respectively, and are evaluated using exponential moving average weights. The experimental results are shown in Table 4, and the experimental results show that the recognition performance of the method of the present application is significantly better than other semi-supervised methods. Specifically, when only 20% of the labels are used for training, the average recognition accuracy of the SSGait method under the three conditions of NM, BG and CL is improved by 3.17%, 2.59% and 2.27% respectively compared with the three comparative semi-supervised methods. When the number of labels increases to 60%, the recognition performance is further improved.

[0084] Table 4 shows the comparison results between the LT setting and the CASIA-B semi-supervised method under different label ratios, NM#5-6, BG#1-2, and CL#1-2 conditions.

[0085]

[0086] 3. Ablation experiment:

[0087] Through ablation studies, we systematically quantified the contribution of each component of the SSGait method to the gait recognition task. All ablation experiments were conducted on the CASIA-B dataset with a low-level (LT) setting, using only 20% of the labeled data. We decomposed the complete SSGait method into different components, including the GaitGL-R(2+1)D basic module component for feature extraction (denoted as B), the gait morphology perturbation component (denoted as P), the pipeline mask reconstruction component (denoted as R), and the segment order prediction component (denoted as C).

[0088] Experimental results are as follows Figure 8 As shown, when the four components B+P+R+C are combined (i.e., the SSGait method), the best performance is achieved on the three probe subsets CL, NM, and BG. This indicates that the four components are complementary and each contributes to improving the overall performance. From the results of B+P and B+C+R, we can observe that the semi-supervised model with a single perturbation performs better than the self-supervised model that only adds auxiliary tasks to the basic module. Simultaneously, the recognition performance of joint training by directly adding a single auxiliary task to the basic module is compared. As observed in Figure 8, both self-supervised auxiliary tasks contribute to improving the recognition accuracy of the gait recognition task. A noteworthy finding is that the result of directly applying pipe mask reconstruction (R) to the basic module is better than the result of applying segment order prediction (C). Applying both auxiliary tasks to the basic module simultaneously further improves the recognition performance. These experimental results demonstrate the effectiveness of the self-supervised auxiliary tasks.

[0089] This invention proposes a novel method that simultaneously utilizes semi-supervised and self-supervised learning to jointly learn spatiotemporal gait feature representations, and establishes an end-to-end gait recognition framework based on this. Specifically, in the proposed SSGait method, we introduce a sequence-level morphological perturbation scheme and utilize a semi-supervised architecture, Mean Teacher, to learn robust feature representations. Simultaneously, we design two task-related self-supervised auxiliary tasks, pipeline mask reconstruction and segment order prediction, to enhance the network's ability to learn relevant features. Experimental validation on two of the most popular cross-view gait datasets demonstrates that, with limited labeled data, the proposed SSGait significantly outperforms the fully supervised STOA method in gait recognition performance.

[0090] The above description is only to illustrate the technical solutions of the present application but not to limit the present application. Other modifications or equivalent replacements of the technical solutions of the present application made by those skilled in the art should be covered in the scope of claims of the present application as long as they are not deviated from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for joint learning gait representation based on limited supervision information, characterized in that: The method comprises the following steps: S1, constructing the relationship between input data and output data in the network: Given a sequence of gait profiles of length k wherein, where h and w represent the height and width of the frame, respectively, denotes a feature matrix, and as input data; N candidate subject definitions are defined as y i , i = 1, 2, …, N, wherein y i is the i-th subject, and the output data is represented as an n-dimensional vector: wherein, i.e. the possibility of belonging to y i ; further, let y be the identity ID of the input data , y is expressed as: S2, optimizing the semi-supervised branch model: Based on the semi-supervised loss optimization model student model f(x, θ, η) and teacher model f(x, θ', η'), a sequence level morphological disturbance is added to the model, and the gait profile sequence obtained in step S1 above A μ-frame gait profile graph is randomly selected from the above-mentioned gait profile graph, where μ is a hyperparameter. For the selected μ-frame gait profile graph, a morphological disturbance scheme is adopted. Firstly, edge detection is performed on the profile graph to detect the pixel points on the profile edge. Subsequently, the pixel points on the profile edge are randomly discarded according to a probability, or new pixel values are added in the neighborhood of the pixel points. The new pixel values should be the average value of the neighboring pixels. Finally, an optimized semi-supervised branch model is obtained. S3, constructing an auxiliary task highly related to the gait recognition task in the self-supervised branch model: 1) Pipe mask reconstruction: simulate the situation that the identified subject is blocked by obstacles and lacks key information in the real scene, cover the pixels in the same area of the gait frame along the time dimension, and then reconstruct, so that the model learns high-level semantic information; The specific steps of pipeline mask reconstruction include: 1) gait segment: a segment composed of n frames is extracted from a gait profile sequence as c, and the gait segment data is sampled with a time sampling strategy with a step of τ 2) pipeline mask: using the pipeline mask strategy, the value of the step τ is set on the two datasets of CASIA-B and OUMVLP respectively, and the segment c is divided into a plurality of non-overlapping cubes according to the set value of the step τ; 3) token embedding: each cube is encoded by the token embedding method, and each token is mapped to the channel dimension; 4) reconstruction of the gait segment: using the strategy of high masking rate, part of the cubes is discarded, and the unmasked cubes are used as the input of the basic module for training, and finally the masking feature map obtained through network training and the learnable mask tokens are used to reconstruct the discarded cubes; 2) Segment order prediction: the original gait sequence is disordered, and the segment composed of gait frames is used to predict the order of the gait sequence, The specific steps of sequential prediction include: 1) First, uniformly sample N ordered segments from the complete gait sequence to form a tuple C = <c1,c2,...,c n > where n is the number of frames. After randomly shuffling the order of the segments in tuple C, tuple C is fed into the neural network as input to learn high-level semantic features and output the probability distribution of different orders. The true order will be used as the target for training and optimization. S4, the semi-supervised branch model and the self-supervised branch model are combined on the gait recognition network GaitGL, and optimized training is performed to obtain a joint loss function: The basic component for feature extraction in the gait recognition network GaitGL adopts an R(2+1)D structure, including M i two-dimensional convolution kernels with a size of N i-1 ×1×d×d and N i one-dimensional convolution kernels with a size of M i ×t×d×d; The formula of the joint loss function is: L total = L tri + λ cont L cont + λ recons L recons + λ order L order , wherein L total is the total loss, λ cont is the weighting item of the semi-supervised loss, λ recons and λ order are the weighting items of the self-supervised loss, L tri is the supervised recognition loss in the semi-supervised branch, L cont is the consistency regularization loss of the Mean Teacher framework in the semi-supervised branch, L recons is the loss of the pipeline mask reconstruction task in the self-supervised branch, and L order is the loss of the segment order prediction task in the self-supervised branch.

2. The joint learning gait representation method based on limited supervision information according to claim 1, characterized in that: In the step S2, θ and θ' in the student model f(x, θ, η) and the teacher model f(x, θ', η') represent the network weights of the model, and η and η' represent the disturbance noise introduced into the model; where the network weights θ are optimized by the supervised loss of labeled samples, the network weights θ' are updated by the exponential moving average EMA of the student model sequence at different training iterations, and the calculation formula of the network weights θ' is: θ' t = aθ' t-1 + (1-a)θ t , where t represents the number of training iterations, and a is a smoothing coefficient set to 0.

999.

3. The method of claim 1, wherein: In the step S3, in the order prediction step, N ordered segments are obtained by uniformly sampling in a non-overlapping manner with a fixed interval of m frames from the gait sequence; the size of a segment is 1×c×h×w, wherein c is the number of frame channels, and h and w represent the height and width of the frame, respectively.

4. The joint learning gait representation method based on limited supervision information according to claim 1, characterized in that: In the step S3, wherein the pipe mask strategy is mathematically represented as: II[p x,y,· ∈Ω] ~ Bernoulli(ρ mask ), where p x,y,· denotes the index of the token, Ω is the set of tokens that are masked, which follows a Bernoulli distribution, ρ mask denotes the mask rate, which shares the same value across different times t.

5. The method of claim 1, wherein: In the step S4, in the semi-supervised branch, the supervised identification loss L tri Optimized using triplet loss, consistency regularization loss L cont MSE loss; in the self-supervised branch, the loss L of the pipeline mask reconstruction task recons MSE loss, the loss L of the segment order prediction task order Cross-entropy loss.

Images