Gait sequence repair and monitoring audience identity recognition method based on prior knowledge

Abstract

The application discloses a gait sequence repair and monitoring audience identity recognition method based on prior knowledge, comprising the following steps: (1) selecting a fixed frame number of continuous sequences from a gait dataset, artificially synthesizing a known occlusion mode for the selected sequences, and obtaining an occluded gait image; (2) detecting the local occlusion area of the occluded gait image in the gait sequence through a YOLO neural network, and using the local occlusion area as prior knowledge to assign a higher repair weight to the occlusion repair area; using 3D-CNN and Transformer to perform end-to-end global repair on the occluded sequence, obtaining the repair result of the occlusion area from the global repair sequence through the prior knowledge, fusing the repair result and the original occluded gait, and obtaining the repaired gait image; (3) dividing into two cases of known occlusion and unknown occlusion, training a repair network model and test data; (4) using a Gaitset network to extract gait features of the repaired gait image, and using the gait features for identity recognition of a monitoring audience, and finally outputting a recognition result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image recognition, and relates to a gait sequence repair and monitoring audience identity recognition method based on prior knowledge. BACKGROUND

[0002] Biometric-based identity authentication is becoming increasingly important in various applications such as access control, visual monitoring, and forensics. Gait recognition is one of the research hotspots in the field of biometric recognition, because it has unique advantages that cannot be matched by biometric features such as face, iris, and fingerprint. For example, it can capture gait features of an object at a distance without the need for the object's cooperation, and it also has good recognition ability in relatively low-resolution image sequences. Currently, gait recognition has been used in the fields of public security criminal investigation and health monitoring, and has achieved good results.

[0003] Currently, the gait contours extracted from the public gait dataset are missing or incomplete, but most gait recognition research methods are implicitly based on complete gait contour sequences. However, in actual applications, the pedestrian contours captured by monitoring may be occluded contour images. The existence of these limiting factors makes the contour extraction process more complex, and even makes it difficult to extract complete gait contour sequences. The processing of occluded gait can be divided into two categories: one is to perform identity recognition without reconstructing the gait contour, and the other is to repair the gait contour and then use it for identity recognition.

[0004] In the gait recognition method without reconstructing the gait contour, GEI or weighted gait representation method (GEI, GHEI, GEI+HOG) is used to reduce the influence of occlusion on average template and other gait representation methods. However, using GEI often loses the change feature information between consecutive images in the contour averaging process, greatly reducing the recognition rate (Ortells, J., Mollineda, R. A., Mederos, B., and Martín-Félez, R. 2017. Gait recognition from corrupted silhouettes: a robust statistical approach. Machine Vision and Applications, 28(1-2), 15-33.). Although this method has a certain effect on occluded sequences, the method of extracting robust features from gait contours needs to be used under the condition that the gait period is determined or complete, and however, when the gait sequence is occluded, the gait period cannot be accurately detected.

[0005] Among the gait recognition methods for gait contour restoration, most of the methods are based on the restoration of gait contour sequences. Traditional methods can use geometric methods - polynomial interpolation, autoregressive prediction based on random linear time series, methods such as using the convex set projection (POCS) proposed by Papoulis, etc. to restore the occluded sequence, and the traditional method can only have good restoration effect based on small occlusion (Lee, T. K. M., Belkhatir, M., and Sanei, S. 2009. Coping with full occlusion in fronto-normal gait by using missing data theory. 2009 7th International Conference on Information, Communications and Signal Processing (ICICS), 1-5.). There are methods that use VGG-16 model to detect occlusion for each frame in the sequence, use the continuity between frames to use the fully convolutional long short-term memory network (FC-LSTM) to reconstruct the occluded frame (Das, D., Agarwal, A., Chattopadhyay, P., and Wang, L. 2020. RGait-NET: An Effective Network for Recovering Missing Information from Occluded Gait Cycles. ArXiv:1912.06765 [Cs].), or by constructing a general key posture sequence database, matching the key posture sequence of the occluded sequence with the key posture database to obtain a most matched key posture sequence, then embedding the key posture as auxiliary information into the contour sequence, using the network composed of BiLSTM to reconstruct the occluded frame detected by the VGG network (Kumara, S. S., Chattopadhyaya, P., and Wang, L. 2021. BGaitR-Net: Occluded Gait Sequence reconstruction with temporally constrained model for gait recognition. arXiv.). But these two methods using LSTM to restore gait sequence cannot restore the sequence in the case of occlusion in the whole gait sequence. SUMMARY

[0006] In view of the deficiencies of the prior art, the purpose of the present application is to provide a gait sequence repair and monitoring audience identity recognition method based on prior knowledge, which can repair the occluded sequence in the presence of various occlusion modes or global occlusion, and the repaired sequence has good identity recognition effect.

[0007] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0008] The present application provides a gait sequence repair and monitoring audience identity recognition method based on prior knowledge, comprising:

[0009] Step (1), select a fixed number of continuous sequences from the gait data set, artificially synthesize known occlusion modes for the selected sequences, and obtain occluded gait images;

[0010] Step (2), detect the local occlusion area of the occluded gait image in the gait sequence by YOLO neural network, assign a higher repair weight to the occlusion repair area as prior knowledge, use 3D-CNN and Spatio-Temporal Transformer (STT) to perform end-to-end global repair on the occluded sequence, obtain the repair result of the occluded area from the global repair sequence through prior knowledge, and fuse the repair result and the original occluded gait to obtain the repaired gait image;

[0011] Step (3), divide into two cases of known occlusion and unknown occlusion, train the gait spatiotemporal sequence repair network GSTRNet and test data;

[0012] Step (4), use the Gaitset network to extract the gait features of the repaired gait image for the identity recognition of the monitoring audience, and finally output the recognition result.

[0013] Preferably, in step (1), 32 continuous frames are randomly selected from the OU_MVLP as a sequence, and then the sequence is artificially synthesized into a known occlusion mode.

[0014] In the method of the present application, the gait spatiotemporal sequence repair network (GSTRNet) is constructed by using YOLO, 3DCNN, Spatio-Temporal Transformer and Gaitset.

[0015] Preferably, in step (2), in the GSTRNet network, 3D-CNN and Spatio-Temporal Transformer (STT) are used to repair the sequence, and the spatial information is repaired while maintaining the spatiotemporal coherence between frames.

[0016] Preferably, in step (2), specifically:

[0017] Step S201, using YOLOv5 network to detect the missing area of each frame in the sequence as prior knowledge, by defining the joint reconstruction loss function of hole loss and non-hole loss fusion, focusing on repairing the local occlusion area, and effectively preserving the unoccluded area information;

[0018] Step S202, using the method of 3D-CNN and skip connection to construct spatio-temporal codec 3DCE, which can repair spatio-temporal information while reducing gradient disappearance;

[0019] Step S203, using Spatio-Temporal Transformer (STT) to repair the spatio-temporal information of deep features.

[0020] Further, the step S201 comprises:

[0021] 2.1.1) Let S1 be a set of gait training samples in D X×Z ; the input space X is the set of all occluded gait sequences; the target space (prediction space) Z represents the set of anchor points of the occluded area (rectangle) in all gait graphs in the corresponding occluded sequence; each gait training sample is composed of a group of sequences (x, z); the target sequence z=(z1, z2, …, z N ) has the same length as the input sequence x=(x1, x2, …, x N ), because there may be more than one occluded area in a graph, and each z i represents the set of all anchor points in x i ;

[0022] The purpose of the application is to train a YOLOv5 network using the gait sample set S1, and to detect the occluded area of each frame in any length of occluded gait sequence data, and output the anchor point set of the occluded area, i.e.:

[0023] anchors=maxp(z│x)

[0024] In the formula, anchors is the anchor point set; max is the maximum function; p(z│x) is the anchor point probability detected in each occluded graph that may exist occlusion;

[0025] 2.1.2) Create a vector sequence with all values of 0 and the same dimension as X Use the anchor point set anchors detected by YOLOv5 network to set the corresponding area value to 1, and finally generate the sequence mask M (the occluded area value is set to 1, and the unoccluded area value is 0);

[0026] 2.1.3) using a sequence mask M = (M1, M2,..., M N ) of length N as part of the input to the spatio-temporal codec network;

[0027] 2.1.4) using a sequence mask M = (M1, M2,..., M N ) of length N to obtain the global inpainting result from the spatio-temporal codec network to obtain the occluded and non-occluded regions to construct the reconstruction loss;

[0028]

[0029] where L hole is the loss of the occluded region; L valid is the loss of the non-occluded region; is the set of occluded regions for each gait image in the occluded sequence; is the global inpainting result; is the real unoccluded sequence.

[0030] Further, the step S202 comprises:

[0031] 2.2.1) Let S2 be a set of gait training samples in a fixed distribution D X×M×Y ; the input space X is the set of all gait sequences with occlusions; M represents the set of mask sequences in the corresponding occluded sequences (generated by the YOLO network); the target sequence Y represents the set of sequences with the same ID as X but without occlusions; each gait training sample is composed of a set of sequences (x, m, y); the target sequence y = (y1, y2,..., y N ) has the same length as the input sequence x = (x1, x2,..., x N ) and m = (m1, m2,..., m N );

[0032] The purpose of the present application is to train a spatio-temporal codec network 3DCE using a set of gait samples S2, which can reconstruct the gait sequence with occlusions, and output the complete gait sequence, i.e.:

[0033]

[0034] wherein, is the global inpainting result; is the occluded sequence, and N represents the number of gait images in a sequence;

[0035] 2.2.2) The encoder consists of a 3D-CNN convolutional layer with kernel 3x3x3 and stride 1x1x1 and two convolutional blocks, each of which is composed of a convolutional layer with kernel 3x3x3 and stride 1x1x1 followed by a convolutional layer with kernel 4x4x4 and stride 2x2x2, and the number of convolutional kernels also increases from 64 to 256, and batch normalization (BN) and RELU nonlinear activation are used in each convolutional layer; In order to make each element participate in the image down-sampling process and retain more detailed information, a convolution with a stride of 2 is used instead of a pooling layer; At the same time, in order to retain more temporal information, a convolution is first performed before each down-sampling;

[0036] The encoder obtains low-dimensional features f with spatio-temporal information from the occlusion sequence The low-dimensional features f obtained from the encoder are represented as:

[0037]

[0038] 2.2.3) The decoder first uses 3 residual blocks composed of dilated convolution, and then uses 3 convolutional layers and 2 deconvolutional layers arranged alternately, with the same kernel and stride as the encoder, except for the last layer, all convolutional layers are subjected to batch normalization (BN) and RELU nonlinear activation, and the last convolutional layer uses a Tanh activation function to map all data to [-1, 1] to maintain consistency with the input data range; The features in the encoder and decoder are mixed using a skip connection method to slow down the loss of details during the encoding process and minimize the impact of gradient disappearance;

[0039] The decoder takes the output of the Spatio-Temporal Transformer as input and outputs the global inpainting result of the occlusion sequence

[0040]

[0041] 2.2.4) Using a sequence mask M = (M1, M2,..., MN) of length N, obtain the local inpainting result from the global inpainting result of the spatio-temporal encoder-decoder network N and fuse it with the input X to generate the final inpainting sequence result

[0042]

[0043] ​​2.2.5) Final repair sequence results As part of the Gaitset network input.

[0044] Further, the step S203 comprises:

[0045] 2.3.1) The present application also introduces an additional Transformer to process deep features, the Spatio-Temporal Transformer consists of four Transformer blocks, and the Transformer block is divided into two sub-layers: MultiHead and MLP. MultiHead is a multi-scale self-attention module, head i (h i ) corresponds to different scales of attention, and designing different scales of attention can obtain spatio-temporal information from local and global to repair the occluded part;

[0046] The low-dimensional feature f e R t×h×w×c is divided into n (n is the number of heads in MultiHead) features f1…fn in the c dimension. n Each feature f i e R t×h×w×c / n is input into h i in MultiHead, each h i performs self-attention processing on different scales, and f i in h i is first mapped into q i , v i , and k i , then q i and k i are multiplied by the matrix to obtain the attention scale, and finally the attention is multiplied by v i as the output h i ;

[0047]

[0048] q i = v i = k i = f i

[0049] In the formula, Softmax is an activation function, and the output value is converted into a probability distribution with a range of [0, 1] and a sum of 1; matual is the matrix multiplication operation; q i is the query set in the i-th self-attention head; k i is the key set in the i-th self-attention head; v iis the set of values in the i-th self-attention head; q i .size is q i The number of elements in the set;

[0050] 2.3.2) The MLP sublayer uses two kernel 3x3, stride 1 2D convolution residual blocks to process the spatio-temporal features of multi-head attention, and fuses the features processed by different scale attention together, so as to repair the spatio-temporal information from multiple scales; all h i After feature splicing, the output of the Transformer block is obtained after processing by the MLP sublayer

[0051]

[0052] In the formula, LR is a LeakyRelu activation function; 3_3Conv is a 2D CNN with a kernel of 3; Concat represents splicing in the Channel dimension; and f is a low-dimensional feature obtained by the encoder.

[0053] In a preferred scheme, in the step (3), the gait spatio-temporal sequence repair network (GSTRNet) is trained by using the gait data of the OU_MVLP gait database to artificially simulate various occlusion modes, and is tested in the OU_MVLP.

[0054] In a preferred scheme, the step (3) is specifically:

[0055] S301, the OU_MVLP dataset is obtained by shooting 10,307 subjects from 14 perspectives (0°-90°, 180°-270°) at a frame rate of 25 fps, and is the largest gait dataset in the world with wide perspective changes; the subjects in the dataset have a large age difference, and the male and female distributions are equal; the database directly provides black and white contour images, the size is normalized using the center of interest object and the top, bottom and horizontal center of the object contour area, and first 64x64 black and white contour sequences are obtained; then the contour images of 44x64 pixels are finally generated by further cropping during training;

[0056] Only side view was selected as the dataset in this data set, while only subjects with at least two sequences were selected from 10,307 subjects, 9001 subjects; at the same time, the sequences were normalized to contour sequences containing 32 frames; if the number of frames of a sequence is less than 32, the last frame is repeated to keep the sequence consistent; the same experimental setup, 9001 subjects were randomly divided into three mutually exclusive sets of approximately equal size: 3000 training sets, 3001 validation sets and 3000 test sets; then, the validation set and the test set are divided into two subsets: gallery set and probe set; the validation set is used to select the iteration number n for the best recognition performance in the experiment, and the test set is used to evaluate the accuracy of the proposed method and other latest methods;

[0057] S302, occlusion mode: simulates several common types of occlusion in real life, namely relative dynamic occlusion, relative static occlusion and random occlusion, and the occlusion degree is divided into three cases of 30%, 40% and 50%;

[0058] Relative dynamic occlusion is divided into relative dynamic occlusion from left to right (RDLR) and relative dynamic occlusion from bottom to top (RDBT), RDLR is similar to being occluded by a tree or a telegraph pole in reality, by setting a vertical mask area (occlusion area is set to 0) conforming to the size of the occlusion range, first covering the rightmost side of the first frame of the contour sequence of a person walking from right to left, then gradually moving the position of the mask, and finally the mask is located at the leftmost side of the last frame to complete the simulation of relative dynamic occlusion from left to right;

[0059] Similarly, RDBT is simulated by setting a horizontal mask to slide from bottom to top; for relative static occlusion, masks are set at fixed positions of all frames in a gait sequence to simulate relative static occlusion in bottom (RSB), relative static occlusion in top (RST), relative static occlusion in left (RSL) and relative static occlusion in right (RSR); for random occlusion, it is divided into relative occlusion vertically (RandV) and relative occlusion horizontally (RandH), and the former is simulated by setting a mask in a random horizontal direction in each frame of a gait sequence, and the latter is simulated by setting a mask in a random vertical direction, totally 24 occlusion modes are simulated;

[0060] S303, in order to test the performance of the gait sequence repair method based on prior knowledge and the audience identity recognition method, a total of three groups of experiments are carried out, which are respectively that the occlusion mode is known, and the probe and gallery occlusion modes are consistent; the occlusion mode is known, but the probe and gallery occlusion types are inconsistent; the occlusion mode is unknown.

[0061] In a preferred scheme, in the step (4), the Gaitset network is used to extract the triplet feature loss and the reconstruction loss as a joint loss, so as to repair the gait sequence contour while keeping the consistency of the identity information.

[0062] Further, the step (4) comprises:

[0063] S401, the application optimizes the network by constructing a joint loss function of a gait space-time sequence repair network (GSTRNet), including a reconstruction loss function and a triplet feature loss function (Triplet Loss); the reconstruction loss uses a commonly used mse loss, since the repair target is to repair the occluded area, in order to avoid over-repair, therefore, the loss L hole and the loss L valid of the non-occluded area are allocated different weights:

[0064] L recon = γ h L hole + γ v L valid

[0065] wherein γ h is the weight value of the occluded area; γ v is the weight value of the non-occluded area; L recon is the reconstruction loss function;

[0066] S402, Gaitset network is a good method capable of directly extracting distinctive features from gait sequences, in order to ensure the consistency of features between the repaired sequence and the original sequence, Gaitset network is used to extract features:

[0067]

[0068] wherein s is the feature extracted from the gait sequence by the Gaitset network; Gaitset network is used to extract gait features from the repaired sequence;

[0069] The similarity of the monitored audience identity is judged by calculating the Euclidean distance between the gait features of the to-be-identified sample and the gait features of the recorded samples in the database, and the recognition of the audience identity is completed;

[0070] S403, in order to retain as much gait recognition information as possible, all samples are classified according to triplets in the training, and are divided into self samples, positive samples and negative samples;

[0071] Among them, the positive sample and the self sample are samples belonging to the same class, and the negative sample is a sample not belonging to the same class as the self sample; define the Gaitset feature loss function L trip based on triplets:

[0072]

[0073] wherein B is the number of subjects; V is the number of sequences taken by each subject; represents the similarity of the positive sample pair in the triplet, represents the similarity of the negative sample pair; margin is the anchor value of the triplet loss;

[0074] By minimizing the triplet feature loss function, the difference between the same class samples can be minimized, and the difference between the different class samples can be maximized, which can well ensure the consistency of the features when the GSTRNet network repairs the occluded sequence; at the same time, the total weight loss function of GSTRNet is defined as:

[0075] L total = ω r L recon + ω t L trip

[0076] In the formula, ω r represent the weight value of the reconstruction loss; ω t represent the weight value of the triple feature loss;

[0077] By controlling the size of ω r , ω t , the most suitable features for network training can be extracted, the consistency of the features can be ensured while repairing the gait sequence pictures, and the convergence speed of the network can be accelerated to a certain extent.

[0078] Compared with the prior art, the beneficial technical effects of the present application are:

[0079] (1) The present application proposes a gait sequence repair and monitoring audience identity recognition method based on prior knowledge, constructs a GSTRNet network through YOLOv5, 3DCNN and Transformer, Gaitset, and obtains a gait sequence repair and monitoring audience identity recognition method based on prior knowledge, so as to improve the application effect when facing complex application scenarios.

[0080] (2) The present application uses the YOLOv5 network to detect the missing area of each frame in the sequence as prior knowledge, defines a joint reconstruction loss function of hole loss and non-hole loss fusion, and focuses on repairing the local occluded area, thereby effectively preserving the information of the non-occluded area.

[0081] (3) The present application uses the current mainstream Spatio-Temporal Transformer to repair the spatio-temporal information of deep features.

[0082] (4) The present application combines the advantages of reconstruction loss and triple feature loss function, so that the dependence of the occluded gait repair and recognition method on the scene is reduced, and the method can face various occlusion modes or exist in the case of global occlusion, repair the occluded sequence, and the repaired sequence has good recognition effect. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 It is a GSTRNet network architecture for repairing and recognizing the occluded sequence of the present application.

[0084] Figure 2 It is a Transformer block architecture.

[0085] Figure 3 It is 24 kinds of occlusion modes simulated based on the OU_MVLP data set.

[0086] Figure 4Rank 1 / 5 recognition rate for known occlusion pattern, and gallery and probe occlusion pattern consistent.

[0087] Figure 5 Rank 1 / 5 recognition rate for known occlusion pattern, but gallery and probe occlusion pattern inconsistent.

[0088] Figure 6 Rank 1 / 5 recognition rate for unknown occlusion pattern.

[0089] Figure 7 Repair effect and recognition rate of each method for known occlusion pattern, and both gallery and probe being RDLR_30.

[0090] Figure 8 Repair effect and recognition rate of each method for known occlusion pattern, and both gallery and probe being RDLR_50.

[0091] Figure 9 Repair effect and recognition rate of each method for known occlusion pattern, and both gallery and probe being RDBT_30.

[0092] Figure 10 Repair effect and recognition rate of each method for known occlusion pattern, and both gallery and probe being RDBT_50.

[0093] Figure 11 Repair effect and recognition rate of each method for known occlusion pattern, gallery being RDLR_50, and probe being RDLR_30.

[0094] Figure 12 Repair effect and recognition rate of each method for known occlusion pattern, gallery being RDBT_50, and probe being RDLR_30. DETAILED DESCRIPTION

[0095] The embodiments of the present application will be further described in conjunction with the examples and the drawings, and the examples are only used to explain the present application, and are not used to limit the scope of the present application.

[0096] Example 1

[0097] The method described in the embodiment is implemented by using the Python programming language and the OU_MVLP gait dataset published by Osaka University. The OU_MVLP dataset is a multi-view gait dataset, which is the largest gait dataset with wide-angle changes in the world. It is photographed by 14 views (0°-90°, 180°-270°) of 10,307 subjects at a frame rate of 25 fps. The subjects in the dataset have a large age difference and an equal distribution of men and women. The database directly provides black and white contour images, which are normalized in size using the center of interest and the top, bottom, and horizontal center of the object contour area. First, a 64x64-pixel black and white contour sequence is obtained, and then the contour images are further cropped during training to finally generate 44x64-pixel contour images.

[0098] As shown in Figures 1-6 , the present application is a gait sequence repair and audience identity recognition method based on prior knowledge, comprising the following steps:

[0099] Step 1, randomly select 32 consecutive frames from the OU_MVLP dataset as a sequence, and then artificially synthesize a known occlusion pattern;

[0100] Step 2, detect the local occlusion area of the occluded gait image in the gait sequence by the YOLO neural network, assign a higher repair weight to the occlusion repair area as prior knowledge, use 3D-CNN and Transformer to perform end-to-end global repair on the occluded sequence, obtain the repair result of the occlusion area from the global repair sequence through prior knowledge, fuse the repair result and the original occluded gait to obtain the repaired gait image; use YOLO, 3DCNN, Spatio-Temporal Transformer and Gaitset to construct a gait spatio-temporal sequence repair network (GSTRNet);

[0101] S201, use the YOLOv5 network to detect the missing area of each frame in the sequence as prior knowledge, define a joint reconstruction loss function that combines hole loss and non-hole loss fusion, and focus on repairing the local occlusion area while effectively preserving the unoccluded area information;

[0102] 2.1.1) Let S1 be a gait training sample set with a fixed distribution D X×Z ; the input space X is a set of all occluded gait sequences; the target space (prediction space) Z represents a set of anchor points of the occlusion area (rectangle) in all gait images in the occluded sequence; each gait training sample consists of a sequence (x, z); the target sequence z=(z1, z2, …, z N ) has the same length as the input sequence x=(x1, x2, …, x N), because there may not be only one occlusion area in a picture, and each z i represents the x i set of all anchors in it;

[0103] The purpose of the present application is to train a YOLOv5 network using a gait sample set S1 and to detect the occlusion area of each frame in an occlusion gait sequence data of any length, output the anchor set of the occlusion area, that is:

[0104] anchors = maxp(z | x)

[0105] In the formula, anchors is the anchor set; max is the maximum function; p(z | x) is the anchor point probability that may exist occlusion in each occlusion picture detected;

[0106] 2.1.2) Create a vector sequence with all values of 0 and the same X dimension as X Using the anchor set anchors detected by the YOLOv5 network, set the corresponding region value to 1, and finally generate the sequence mask M (the occlusion region value is set to 1, and the non-occlusion region value is 0);

[0107] 2.1.3) Use the sequence mask M = (M1, M2, …, M N ) of length N as part of the input of the space-time encoder-decoder network;

[0108] 2.1.4) Use the sequence mask M = (M1, M2, …, M N ) of length N to obtain the global repair result from the space-time encoder-decoder network to obtain the occlusion area and the non-occlusion area to construct the reconstruction loss;

[0109]

[0110] In the formula, L hole is the loss of the occlusion area; L valid is the loss of the non-occlusion area; is the set of occlusion areas in each gait image in the occlusion sequence; is the global repair result; is the real unoccluded sequence;

[0111] S202, use the method of 3D-CNN and jump connection to build a space-time encoder-decoder 3DCE, which can repair space-time information while reducing gradient disappearance;

[0112] 2.2.1) Let S2 be a fixed distribution D X×M×YThe gait training sample set S2 in the application; the input space X is the set of all gait sequences with occlusion; M represents the set of mask sequences in the corresponding occlusion sequence (generated by the YOLO network); the target sequence Y represents the sequence set with the same ID as X but without occlusion; each gait training sample is composed of a sequence set (x, m, y); the target sequence y=(y1, y2,..., y N ) length is equal to the input sequence x=(x1, x2,..., x N ) and m=(m1, m2,..., m N );

[0113] The application aims to train a spatio-temporal codec network 3DCE using the gait sample set S2, which can reconstruct the gait sequence with occlusion and output the complete gait sequence, that is:

[0114]

[0115] In the formula, is the global repair result; is the occlusion sequence (N represents the number of gait images in a sequence);

[0116] 2.2.2) The encoder is composed of a 3D-CNN convolution layer with a kernel of 3x3x3 and a stride of 1x1x1 and two convolution blocks, each of which is composed of a convolution layer with a kernel of 3x3x3 and a stride of 1x1x1 followed by a convolution layer with a kernel of 4x4x4 and a stride of 2x2x2, and the number of convolution kernels also increases from 64 to 256, and batch normalization (BN) and RELU non-linear activation are used in each convolution layer; In order to let each element participate in the image down-sampling and retain more detailed information, the convolution with a stride of 2 is used instead of the pooling layer; At the same time, in order to retain more time information, a convolution is first performed before each down-sampling;

[0117] The encoder obtains low-dimensional features f with spatio-temporal information from the occlusion sequence Given the occlusion sequence, the low-dimensional features f obtained from the encoder are represented as:

[0118]

[0119] 2.2.3) Decoder first uses 3 residual blocks composed of dilated convolution, then adopts 3 layers of convolution and 2 layers of deconvolution arranged alternately, the kernel and stride are the same as the encoder, except for the last layer, all convolution layers are subjected to batch normalization (BN) and RELU nonlinear activation, and a Tanh activation function is used after the last convolution layer to map all data to [-1, 1] to keep consistent with the range of input data; the features in the encoder and decoder are mixed using the way of skip connection, which slows down the loss of details when downsampling in the encoding process, and minimizes the impact of gradient disappearance problem;

[0120] The decoder outputs a global inpainting result of the occluded sequence by taking the output of the Spatio-Temporal Transformer as input.

[0121]

[0122] 2.2.4) Obtain a local inpainting result from the global inpainting result of the Spatio-Temporal Transformer network using a sequence mask M = (M1, M2, …, MN) of length N, and fuse it with the input X to generate the final inpainting sequence result N

[0123]

[0124] In the formula, represents the multiplication between elements; is the set of occluded regions of each gait image in the occluded sequence;

[0125] 2.2.5) Take the final inpainting sequence result as part of the input of the Gaitset network;

[0126] S203, using Spatio-Temporal Transformer (STT) to repair spatial and temporal information of deep features;

[0127] 2.3.1) The present application additionally introduces a Transformer to process deep features, the Spatio-Temporal Transformer consists of four Transformer blocks, and the Transformer block is divided into two sub-layers of multi-head attention (MultiHead) and MLP. MultiHead is a multi-scale self-attention module, and head i (h i ​​​Attention of different scales is designed to obtain spatial and temporal information from local and global to repair the occluded part;

[0128] The low-dimensional feature f e R t×h×w×c In the c dimension, n(n is the number of heads in multi-head attention) features f1…fn are equally divided n Each feature f i e R t×h×w×c / n As the input corresponding to h i in MultiHead, each h i performs self-attention processing corresponding to different scales in h i f i is first mapped into q i ,v i ,k i , then q i and k i are multiplied by matrix to obtain attention scale, and finally the attention is multiplied by v i as the output h i ;

[0129]

[0130] q i =v i =k i =f i

[0131] In the formula, Softmax is an activation function, and the output value is converted into a probability distribution with a range of [0, 1] and a sum of 1; matual is a matrix multiplication operation; q i is the query set in the i-th self-attention head; k i is the key set in the i-th self-attention head; v i is the value set in the i-th self-attention head; q i .size is the number of elements in the q i set;

[0132] 2.3.2) The MLP sublayer uses two 2D convolution residual blocks with a kernel of 3x3 and a stride of 1 to process the spatial and temporal features of multi-head attention, and fuses the features processed by different scale attention together, so as to repair the spatial and temporal information from multiple scales; After all h i are spliced and processed by the MLP sublayer, the output of the Transformer block is obtained

[0133]

[0134] In the formula, LR is a LeakyRelu activation function; 3_3Conv is a 2DCNN with a kernel of 3; Concat represents splicing in the Channel dimension; and f is a low-dimensional feature obtained by an encoder.

[0135] Step 3, the gait data of the OU_MVLP gait database is used to artificially simulate various occlusion modes to train a gait spatiotemporal sequence repair network (GSTRNet) and test the GSTRNet in the OU_MVLP;

[0136] S301, the OU_MVLP dataset is obtained by photographing 10,307 subjects from 14 viewing angles (0°-90°, 180°-270°) at a frame rate of 25 fps, and is the largest gait dataset in the world with wide viewing angle changes; the subjects in the dataset have a large age difference and an equal male and female distribution; the database directly provides black and white contour images, the center of gravity of the object of interest and the top, bottom and horizontal center of the contour area of the object are used to normalize the size, 64x64 black and white contour sequences are obtained first; then, the contour images of 44x64 pixels are finally generated by further cropping during training;

[0137] In the present application, only side views are selected as the dataset, and at least two sequences are selected from 10,307 subjects, and there are 9001 subjects; at the same time, the sequences are normalized to contour sequences containing 32 frames; if the number of frames of a sequence is less than 32, the last frame is repeated to keep the sequence consistent; the experimental setup is the same, and the 9001 subjects are randomly divided into three mutually exclusive sets of approximately equal size: 3000 training sets, 3001 validation sets and 3000 test sets; then, the validation set and the test set are divided into two subsets: gallery set and probe set; the validation set is used to select the iteration number n for the best recognition performance in the experiment, and the test set is used to evaluate the accuracy of the proposed method and other latest methods;

[0138] S302, occlusion mode: several common occlusion types in real life are simulated, namely relatively dynamic occlusion, relatively static occlusion and random occlusion, and the occlusion degree is divided into three cases of 30%, 40% and 50%;

[0139] The relative dynamic occlusion is divided into relative dynamic occlusion from left to right (RDLR) and relative dynamic occlusion from bottom to top (RDBT). The RDLR is similar to being occluded by a tree or a telegraph pole in reality. A vertical mask region (the occlusion region is set to 0) is set to cover the rightmost side of the first frame in a contour sequence of a person walking from right to left, and then the position of the mask is gradually moved, and finally the mask is located at the leftmost side of the last frame to complete the simulation of the relative dynamic occlusion from left to right.

[0140] Similarly, the RDBT is simulated by setting a horizontal mask to slide from bottom to top. For the relative static occlusion, the mask is set at a fixed position in all frames in a gait sequence to simulate the relative static occlusion in bottom (RSB), the relative static occlusion in top (RST), the relative static occlusion in left (RSL) and the relative static occlusion in right (RSR). For the random occlusion, it is divided into relative occlusion vertically (RandV) and relative occlusion horizontally (RandH). The relative occlusion vertically is simulated by randomly setting a mask in the horizontal direction in each frame of the gait sequence, and the relative occlusion horizontally is simulated by randomly setting a mask in the vertical direction. A total of 24 occlusion modes are simulated.

[0141] S303, in order to test the performance of the gait sequence repair method based on prior knowledge and the audience identity recognition method, three groups of experiments are performed, which are that the occlusion mode is known, and the probe and gallery occlusion modes are consistent; the occlusion mode is known, but the probe and gallery occlusion types are inconsistent; and the occlusion mode is unknown.

[0142] Step 4, the gait features of the repaired gait image are extracted by using the Gaitset network, which are used for the identity recognition of the monitored audience, and finally the recognition result is output.

[0143] S401、The application optimizes the network by constructing a joint loss function of a gait spatiotemporal sequence repair network (GSTRNet), including a reconstruction loss function and a triplet feature loss function (Triplet Loss); the reconstruction loss uses a commonly used mse loss, since the repair target is to repair the occluded area, in order to avoid over-repair, therefore, different weights are assigned to the occluded area loss L hole and the non-occluded area loss L valid :

[0144] L recon =γ h L hole +γ v L valid

[0145] In the formula, γ h is the weight value of the occluded area; γ v is the weight value of the non-occluded area; L recon is the reconstruction loss function;

[0146] S402, Gaitset network is a good method capable of directly extracting distinctive features from gait sequences, in order to ensure the consistency of the features between the repair and the original sequence, Gaitset network is used to extract features:

[0147]

[0148] In the formula, s is the feature extracted from the gait sequence by the Gaitset network; Gaitset network is used to extract gait features from the repair sequence;

[0149] The similarity of the monitored audience identity is judged by calculating the Euclidean distance between the gait features of the to-be-identified sample and the gait features of the recorded samples in the database, so as to complete the identification of the audience identity;

[0150] S403, in order to retain as much gait recognition information as possible, all samples are classified according to triplets during training, and are divided into self samples, positive samples and negative samples;

[0151] Among them, the positive sample and the self sample are samples belonging to the same class, and the negative sample is a sample not belonging to the same class as the self sample; define a Gaitset feature loss function L trip based on triplets:

[0152]

[0153] In the formula, B is the number of subjects; V is the number of sequences taken by each subject; represents the similarity of the positive sample pair in the triplet, represents the similarity of negative sample pairs; margin is the anchor value of triplet loss;

[0154] By minimizing the triplet feature loss function, the difference between samples of the same class is minimized, and the difference between samples of different classes is maximized, which well guarantees the consistency of the features when the GSTRNet network repairs the occluded sequence. At the same time, the GSTRNet effectively combines the reconstruction loss function, the Triplet Loss loss function, and the total weight loss function is defined as:

[0155] L total = ω r L recon + ω r L trip

[0156] In the formula, ω r , ω t respectively represent the weight values of the reconstruction loss and the triplet feature loss;

[0157] By controlling the size of ω r , ω t , the most suitable features for network training can be extracted, which can repair the gait sequence pictures while guaranteeing the consistency of the features, and at the same time, to a certain extent, speed up the convergence speed of the network.

[0158] Application Example 1

[0159] The method described in the embodiment uses the publicly released Osaka University OULP (The OU-ISIR Gait Database Comprising the Large Population Dataset) gait dataset to judge its applicability. OULP contains a total of 4,016 subjects, each subject has two main subsets (gallery and probe subsets). According to the observation angle, each subset includes five angle sequences (55, 65, 75, 85), including all four angles. The subjects in the dataset have a large difference in age, and the male and female distribution is equal. The database directly provides black and white contour images, using the center of interest object and the top, bottom and horizontal center of the object contour area to normalize the size, first obtain 64x64 pixel black and white contour sequences, then further crop during training, and finally generate 44x64 pixel contour images.

[0160] Six groups of experiments were carried out on the OULP dataset to judge the applicability of the GSTRNet network, which were as follows: the occlusion mode was known, and the gallery and probe were RDLR_30; the occlusion mode was known, and the gallery and probe were RDLR_50; the occlusion mode was known, and the gallery and probe were RDBT_30; the occlusion mode was known, and the gallery and probe were RDBT_50; the occlusion mode was known, the gallery was RDLR_50, and the probe was RDLR_30; and the occlusion mode was known, the gallery was RDBT_50, and the probe was RDLR_30, as shown in Figures 7-12

[0161] The specific steps are as follows:

[0162] Step 1: First, 32 frames in the OULP dataset were randomly and continuously selected as a sequence, and then the sequence was artificially synthesized into a known occlusion mode.

[0163] Step 2: The occluded sequence was repaired using the VideoGAN network and the GSTRNet network respectively to obtain the repaired gait sequence (in the repair result in the figure, green represents that the real sequence exists but the repaired sequence does not, and red represents that the real sequence does not exist but the repaired sequence exists, that is, the result of over-repairing).

[0164] Step 3: The gait features were extracted from the occluded sequence, the VideoGan network and the GSTRNet network repaired sequence respectively using the Gaitset network.

[0165] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Improvements and changes obtained by those skilled in the art without departing from the technical concept of the present application should also be considered as the protection scope of the present application.​

Claims

1. A method for gait sequence repair and monitoring audience identity recognition based on prior knowledge, characterized in that Comprise: Step (1), select a fixed number of continuous sequences from the gait data set, artificially synthesize known occlusion patterns for the selected sequences, and obtain occluded gait images; Step (2), detect the local occlusion area of the occluded gait image in the gait sequence by the YOLO neural network, assign a higher repair weight to the occlusion repair area as prior knowledge, use 3D-CNN and Spatio-Temporal Transformer for end-to-end global repair of the occluded sequence, obtain the repair result of the occluded area from the global repair sequence through prior knowledge, and fuse the repair result and the original occluded gait to obtain the repaired gait image; Step (3), divide into two cases of known occlusion and unknown occlusion, train the gait spatio-temporal sequence repair network GSTRNet and test data; Step (4), use the Gaitset network to extract the gait features of the repaired gait image for audience identity recognition, and finally output the recognition result; GSTRNet is constructed by using YOLO, 3DCNN, Spatio-Temporal Transformer and Gaitset; In step (2), specifically: Step S201, use the YOLOv5 network to detect the missing area of each frame in the sequence as prior knowledge, define a joint reconstruction loss function that combines hole loss and non-hole loss, and focus on repairing the local occlusion area while effectively preserving the information of the unoccluded area; Step S202, use 3D-CNN and a method of skip connection to construct a spatio-temporal encoder-decoder 3DCE, which can repair spatio-temporal information while reducing gradient disappearance; Step S203, use Spatio-Temporal Transformer to repair the spatio-temporal information of deep features; The step S202 comprises: 2.2.1) Let S2 be a fixed distribution D X×M×Y The gait training sample set is defined as follows: the input space X is the set of all occluded gait sequences; M represents the set of mask sequences in the corresponding occluded sequence, generated by the YOLO network; the target sequence Y represents the set of sequences with the same ID as X but without occlusion; each gait training sample consists of a set of sequences (x, m, y); the target sequence y = (y1, y2, ..., y...). N The length is equal to the input sequence x = (x1, x2, ..., x...). N ) and m = (m1, m2, ..., m N ); Use the gait sample set S2 to train a spatio-temporal encoder-decoder network 3DCE, which can reconstruct the gait sequence with occlusion and output a complete gait sequence, namely: In the formula, is a global repair result; is an occlusion sequence, and N represents the number of gait images in a sequence. 2.2.2) The encoder is composed of a 3D-CNN convolution layer with a kernel of 3x3x3 and a stride of 1x1x1, and two convolution blocks, each of which is composed of a convolution layer with a kernel of 3x3x3 and a stride of 1x1x1 followed by a convolution layer with a kernel of 4x4x4 and a stride of 2x2x2. The number of convolution kernels also increases from 64 to 256. Batch normalization BN and RELU non-linear activation are used in each convolution layer. In order to make every element participate in image downsampling and preserve more detail information, stride-2 convolution is used instead of pooling layer. At the same time, in order to preserve more temporal information, a convolution is first performed before each downsampling. The encoder obtains low-dimensional features f with spatio-temporal information by taking the occlusion sequence Given the occlusion sequence, the low-dimensional features f obtained from the encoder are represented as: 2.2.3) Decoder first uses 3 residual blocks composed of dilated convolution, then uses 3 layers of convolution and 2 layers of deconvolution arranged alternately, the kernel and stride are the same as the encoder, except for the last layer, all convolution layers are subjected to batch normalization BN and RELU nonlinear activation, after the last convolution layer, a Tanh activation function is used to map all data to [-1, 1] to keep consistent with the range of input data; using the way of jump connection to mix the features in the encoder and decoder, slow down the loss of details when downsampling in the encoding process, and minimize the impact of gradient disappearance problem; The decoder passes the output of the Spatio-Temporal Transformer as input to the output occlusion sequence global inpainting result 2.2.4) Using a sequence mask M = (M1, M2,..., M N ), of length N, to obtain a local inpainting result from the global inpainting result of the spatio-temporal codec network and fuse it with the input X to generate the final inpainting sequence result ​ 2.2.5) Final repair sequence results are reported As part of the Gaitset network input.

2. The method of claim 1, wherein the method further comprises: In the step (2), in the GSTRNet, 3D-CNN and Spatio-Temporal Transformer are used to repair the sequence, repair the spatial information while maintaining the spatio-temporal coherence between frames.

3. The method of claim 1, wherein the method further comprises: The step S201 comprises: 2.1.1) Let S1 be a fixed distribution D X×Z The gait training sample set is defined as follows: the input space X is the set of all occluded gait sequences; the target space Z represents the set of anchor points of occluded regions in all gait graphs of the corresponding occluded sequence; each gait training sample consists of a set of sequences (x, z); the target sequence z = (z1, z2, ..., z...). N The length is equal to the input sequence x = (x1, x2, ..., x...). N Because an image may contain more than one occluded region, each z-axis... i Indicates that x i The set of all anchor points; A YOLOv5 network is trained using the gait sample set S1, and can detect the occlusion area of each frame of the occlusion gait sequence data, and output the anchor point set of the occlusion area, that is: anchors=max p(z|x) In the formula, anchors is the anchor point set; max is the maximum function; p(z|x) is the anchor point probability of each occlusion image that may exist occlusion; 2.1.2) Create a sequence of vectors with all values of 0 and consistent with the X dimension Using the anchor point set anchors detected by the YOLOv5 network, set the corresponding region value to 1, and finally generate a sequence mask M, set the occluded region value to 1, and the unoccluded region value to 0; 2.1.3) using a sequence mask M = (M1, M2,..., M N ) of length N as part of the input to the spatio-temporal codec network; 2.1.4) Use a sequence mask M = (M1, M2, ..., Mn) of length N. N ), from the global repair results of the spatiotemporal codec network Obtain the occluded and unoccluded regions to construct the reconstruction loss; In the formula, L hole is the loss of the occluded region; L valid is the loss of the non-occluded region; is the set of occluded regions of each gait image in the occluded sequence; is the global inpainting result; is the real unoccluded sequence.

4. The method of claim 1, wherein the method further comprises: The step S203 comprises: 2.3.1) Additional introduction of Transformer to process deep features, Spatio-Temporal Transformer consists of four Transformer blocks, and the Transformer block is divided into two sub-layers of MultiHead and MLP; MultiHead is a multi-scale self-attention module, head i (h i ) corresponds to different scales of attention, and designing different scales of attention can obtain spatio-temporal information from local and global to repair the occluded part; The low-dimensional feature f obtained by the encoder is f t×h×w×c In the c dimension, n features f1… fn are equally divided n , n is the number of heads in multi-head attention, and each feature f i ∈R t×h×w×c / n As the input corresponding to h i in MultiHead, each h i performs self-attention processing on different scales, and f i in h i is first mapped to q i , v i , and k i , then q i and k i are multiplied by matrix to obtain attention scale, and finally the attention is multiplied by v i as weight to obtain output h i ; q i = v i = k i = f i In the formula, Softmax is an activation function, and the output value is converted into a probability distribution with a range of [0, 1] and a sum of 1; matual is a matrix multiplication operation; q i is a query set in the i-th self-attention head; k i is a key set in the i-th self-attention head; v i is a value set in the i-th self-attention head; q i .size is the number of elements in the q i set. 2.3.2) The MLP sublayer uses two kernel 3x3, stride 1 2D convolution residual block to process the spatio-temporal features of multi-head attention, and fuse the features processed by different scale attention together, so as to repair the spatio-temporal information from multiple scales; all h i After feature splicing, the output of the Transformer block is obtained after processing by the MLP sublayer In the formula, LR is a LeakyRelu activation function; 3_3Conv is a 2D CNN with a kernel of 3; Concat represents concatenation in the Channel dimension; f is the low-dimensional feature obtained by the encoder.

5. The method of claim 1, wherein the method further comprises: In the step (3), the GSTRNet is trained by using various occlusion modes artificially simulated by using gait data of the OU_MVLP gait database, and is tested in the OU_MVLP.

6. The method of claim 5, wherein the method further comprises: The step (3) is specifically: S301, the OU_MVLP dataset is obtained from 10,307 subjects at 14 viewing angles of 0°-90° and 180°-270°, and is photographed at a frame rate of 25 fps, which is the largest gait dataset in the world with wide viewing angle changes; the subjects in the dataset have a large age difference, and the male and female distribution is equal; the database directly provides black and white contour images, the center of interest object and the top, bottom and horizontal center of the object contour area are used to normalize the size, first obtain a 64x64 black and white contour sequence; then further crop during training to finally generate a 44x64 pixel contour image; Only side view was selected as the dataset in this data set, and only subjects with at least two sequences were selected from 10,307 subjects, 9001 subjects; at the same time, the sequence was normalized to a contour sequence containing 32 frames; if the number of frames of a sequence is less than 32, the last frame is repeated to keep the sequence consistent; the 9001 subjects were randomly divided into three mutually exclusive sets of approximately equal size: 3000 training sets, 3001 validation sets and 3000 test sets under the same experimental conditions; then, the validation set and the test set were divided into two subsets: the gallery set and the probe set; the validation set is used to select the iteration number n for the best identification performance in the experiment, and the test set is used to evaluate the accuracy of the proposed method and other latest methods; S302, the occlusion mode: several common occlusion types in real life are simulated, namely relatively dynamic occlusion, relatively static occlusion and random occlusion, and the occlusion degree is divided into three cases of 30%, 40% and 50%; The relatively dynamic occlusion is divided into RDLR from left to right and RDBT from bottom to top, RDLR is similar to being occluded by a tree or a telegraph pole in reality, by setting a vertical mask with the size of the occlusion range, the occlusion area is set to 0, first covering the rightmost side of the first frame of the contour sequence of a person walking from right to left, then gradually moving the position of the mask, and finally the mask is located at the leftmost side of the last frame to complete the simulation of the relatively dynamic occlusion from left to right; RDBT is simulated by setting a horizontal mask to slide from bottom to top; for relatively static occlusion, a mask is set at a fixed position in all frames in a gait sequence, simulating the relatively static occlusion at the bottom RSB, top RST, left side RSL and right side RSR positions; for random occlusion, it is divided into horizontal random occlusion RandV and vertical random occlusion RandH, horizontal random occlusion is simulated by randomly setting a mask in the horizontal direction of each frame of the gait sequence, and vertical random occlusion is simulated by randomly setting a mask in the vertical direction, a total of 24 occlusion modes are simulated; S303, in order to test the performance of the gait sequence repair and monitoring audience identity recognition method based on prior knowledge, three groups of experiments were conducted, respectively, the occlusion mode is known, and the probe and gallery occlusion modes are consistent; the occlusion mode is known, but the probe and gallery occlusion types are inconsistent; the occlusion mode is unknown.

7. The method of claim 1, wherein the method further comprises: In step (4), the Gaitset network is used to extract the triplet feature loss and reconstruction loss as a joint loss, which maintains the consistency of the identity information while repairing the gait sequence contour, and the step (4) comprises: S401、Optimize the network by constructing the joint loss function of GSTRNet, including reconstruction loss function and triplet feature loss function Triplet Loss; the reconstruction loss uses the commonly used mse loss, since the repair target is to repair the occluded area, in order to avoid over-repair, therefore the loss L hole and the loss L valid of the non-occluded area are assigned different weights: L recon = γ h L hole + γ v L valid In the formula, γ h is a weight value of the occluded area; γ v is a weight value of the non-occluded area; L recon is a reconstruction loss function; S402, the Gaitset network is used to extract features: In the formula, s is a feature extracted from the gait sequence by the Gaitset network; extract gait features from the repaired sequence using the Gaitset network; The similarity of the monitoring audience identity is judged by calculating the Euclidean distance between the gait features of the to-be-identified sample and the gait features recorded in the database, and the audience identity is recognized; S403, in the training, all samples are classified according to the triplet, which is divided into self sample, positive sample and negative sample; Wherein, the positive sample and the self sample are samples belonging to the same class, and the negative sample is a sample not in the same class as the self sample; define a Gaitset feature loss function L based on triplets trip : where B is the number of subjects; V is the number of sequences taken from each subject; represents the similarity of positive sample pairs in triplets, represents the similarity of negative sample pairs; margin is the anchor value of the triplet loss. By minimizing the triplet feature loss function, the differences between samples of the same class can be minimized, and the differences between samples of different classes can be maximized, which well guarantees the consistency of the features when GSTRNet restores the occluded sequence. At the same time, the total weight loss function of GSTRNet is defined as the combination of reconstruction loss function and Triplet Loss loss function: L total = ω r L recon + ω t L trip In the formula, ω r a weight value representing the reconstruction loss; ω t a weight value representing the triple feature loss; By controlling the size of ω r , ω t , the most suitable features for network training can be extracted, which can ensure the consistency of features while repairing gait sequence images and accelerate the convergence speed of the network to some extent.

Images