Cross-view gait recognition method and system based on diffusion model under missing view

Abstract

The invention belongs to the technical field of gait recognition, and particularly relates to a cross-visual-angle gait recognition method and system based on a diffusion model under a missing visual angle, and the method comprises the steps: inputting a limited gait sequence under the condition of the missing visual angle into a cross-visual-angle gait recognition frame; and realizing gait sequence generation from an existing visual angle to a missing visual angle by using the gait sequence conversion path under any visual angle, training a gait recognition model by using all the expanded gait sequences, and finishing cross-visual-angle gait recognition under the condition of the missing visual angle by using the trained model. The cross-view gait sequence generation method has the advantages that the cross-view gait sequence generation model can generate the gait sequence of the same ID identity under the missing view based on the clue frame under any missing view by taking the existing gait information of a certain ID identity under any view as the constraint, and the missing view gait sequence generation method based on the cross-view conversion path is provided.

Description

Technical Field

[0001] The present application belongs to the technical field of gait recognition, and specifically relates to a cross-view gait recognition method and system based on a diffusion model under missing perspective. Background Art

[0002] The scarcity of gait recognition data is one of the main challenges facing the development of this field. Since the collection of gait data requires specific environments and equipment, such as high-definition cameras or professional somatosensory devices (such as Kinect), and involves multiple gait perspectives (such as different angles, light, clothing, etc.), it is costly and time-consuming to actually obtain a large-scale gait dataset. At the same time, gait data involves personal privacy, and issues such as pedestrian privacy protection and data security need to be considered, especially when collected in public places, which is prone to privacy protection disputes. High-quality gait recognition data is often difficult to obtain in large quantities, which limits the construction of large-scale datasets.

[0003] Various factors have led to the lack of diversity in existing gait datasets, making it difficult to cover people of different ages, genders, body shapes, walking habits, etc. Training based on these data affects the generalization ability of the recognition model. The generation of gait data can effectively improve the cross-perspective recognition ability of the gait recognition model under the condition of missing perspective. The current existing gait data generation methods can be mainly divided into two types. One is to generate a gait energy image (GEI). The GEI image is obtained by averaging all single-frame images in a gait sequence of a person. This type of image completely loses the time information in the gait sequence. Therefore, the current gait recognition model no longer uses GEI data training; another gait data generation method is to establish a mapping relationship between different perspectives of a person through a generative model. Given a gait sequence shot by a person at perspective 1, the gait sequence at perspective 2 is generated. This gait data generation method has great limitations, which are mainly reflected in two aspects: First, when it is necessary to generate gait data from a certain perspective, the model needs to input the gait data from another specific perspective of this ID (ID is used to represent the identity information of the person in the present invention). The specific perspective required depends on how the model divides all different perspectives during training. Once a certain perspective information is missing, the model cannot generate data from another perspective corresponding to the mapping relationship of the perspective. Second, for outdoor gait data, there is no explicit perspective label information in the data set, and gait data from any perspective may exist in all the data. At this time, the generation model that establishes a mapping relationship between different perspectives is completely inapplicable. Summary of the invention

[0004] The present invention focuses on how to improve the cross-view gait recognition performance of the model when any view is missing, especially when the extreme view is missing. In view of the shortcomings of the prior art, the present invention provides a cross-view gait recognition method and system based on a diffusion model. The technical solution is: A cross-view gait recognition method based on a diffusion model under missing view includes the following steps: S1. Obtain the original data set and perform data preprocessing. The processed data set has a uniform directory structure. The data set is used to train the gait feature extractor, feature space transformer and reinforcement learning model. S2. extracting gait identity embedding features in the data set using a gait feature extractor; The gait feature extractor includes a spatial encoder and a temporal encoder. For a single gait sequence of each ID in the data set, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are input into the temporal encoder to fuse the temporal features. After that, the feature dimension is transformed through the feature classification head to obtain the gait identity embedding feature. S3. If the original gait sequence is greater than or equal to the threshold, the feature space transformer is used to reduce the size of the single frame image and then input into the denoising model for training; if the gait identity embedded feature pixel is less than the threshold, then it is input into the denoising model for training; after the training is completed, the feature space transformer restores the image size and completes the reconstruction of the gait data; The denoising model uses the existing U-net structure as the basic framework, adds the gait identity embedding features extracted by the gait feature extractor, and the clue frame used to guide gait generation, integrates the gait identity embedding features and clue frames into the cross-attention interaction of the denoising process, and generates gait sequences from Gaussian pure noise sampling after multiple iterations; S4. The denoised generated gait sequence is input into the reinforcement learning module, and the discontinuous limb movements that deviate from the objective physical laws in the generated continuous gait frames under the missing perspective are corrected, and finally the gait sequence image of the ID under this missing perspective is output.

[0005] Preferably, the original data set includes indoor data and outdoor data; CASIA-B and OUMVLP are indoor data, which contain gait sequence data of multiple different and fixed perspectives, and the amount of data at different perspectives is roughly the same. CASIA-B has different dressing styles of NM, BG, and CL, and OUMVLP has only NM style; GREW and GAIT3D are outdoor data, in which the gait sequences of all IDs are random perspectives, there is no label information indicating the perspective in the data set, and the different perspective data of each ID are randomly distributed, and the dressing style is also completely random. In order to unify the data labels, the labels of all different fixed perspectives in the indoor data are set to be sorted (0, 1, 2, ..., j), the data perspective label in GREW is set to j+1, and the data perspective label in GAIT3D is set to j+2. The unified data set does not restrict the dressing style. The processed data set has a uniformly set directory structure and is divided into a training set and a test set.

[0006] Preferably, the gait feature extractor is based on several Transformer layers to design a spatial encoder and a temporal encoder respectively, and based on several CNN layers to design a feature classification head, and the gait sequence in the data set is extracted. After being mapped to the token sequence, it is input into the Transformer structure to extract features; The spatial encoder only models the interaction between different tokens in the same frame to obtain the representation of each frame. , all frame-level representations Connected together, after the interaction between tokens from different frames of the time encoder, we get ; Finally, the output of the encoder is resized using the feature classification head to obtain the gait feature encoding , expressed as: ; Among them, S1 is the length of the gait sequence, H is the height of the image, W is the width of the image, and C1 is the channel.

[0007] Preferably, the feature space transformer includes a two-part structure, the first half is an autoencoder, which outputs two learnable parameters; the two parameters are respectively used as the mean and variance, sampled from Gaussian noise, and passed through the feature decoder in the second half to gradually restore the original input gait sequence image, thereby realizing the reconstruction of gait data.

[0008] Preferably, S31. if the original gait sequence is greater than or equal to the threshold, the gait sequence Input the autoencoder, and after feature extraction, two feature codes are obtained, namely, mean code and variance coding , where the mean encoding represents the mean of the Gaussian distribution that the input gait sequence obeys after being encoded, and the variance encoding is the variance of the Gaussian distribution, which can be expressed as: ;

[0009] By using the mean and variance to randomly sample the Gaussian distribution, the obtained features are reduced by n times compared to the original sequence single frame size; ; S32. When generating gait sequences, the autoencoder is no longer used and is directly sampled from standard Gaussian noise. ; At the same time, the gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. The output of the denoising model is then partially restored to the original input gait sequence through the decoder. This process is expressed as: .

[0010] Preferably, the denoising model includes two processes: forward denoising and reverse denoising. Assume that the original denoised data is expressed as , according to time t=1,2,3,…,T, gradually add noise to get the data set ; The forward noise addition process uses a linear noise addition method. The noise variance added at each time step is fixed. The noise amount increases linearly with the increase of time step t. When reaching a larger time step, the original data is close to white noise. The variance of the initial noise is , the variance of the final noise is , since the noise variance increases linearly, the noise variance at time step t is as follows: ; Using the recursive relation, we get: ; is the noise sampled from the standard Gaussian distribution; let , the data of time step t obtained by adding noise from time step t-1 can be expressed as: ; Using the recursive relation, we can easily get Add noise and get Indicates: ; in express ; () indicates Gaussian distribution; In the reverse denoising process, at the current time step t, based on the noisy data Data before sampling and noise addition , gradually denoise to generate the original gait sequence; according to Bayesian theorem, from the known The probability distribution of The probability distribution of can be expressed as: ; Encoding with gait features As the target gait constraint, the clue frame of the sequence to be generated To guide the generation of gait sequences, the probability distribution of the model-based reverse denoising process is expressed as: ; ; Convert the predicted denoised data into the noise for predicting the current time step t as follows: ; Where c is the viewing angle of the target sequence to be generated, are the parameters of the network model, , indicates that both the mean and variance are predicted by the model, is the noise at time step t.

[0011] Preferably, the denoising model includes a noise prediction loss function and an ID gait information constraint term, wherein the noise prediction loss function is calculated as follows: In the current step t, the predicted noise output by the model is arrive The added noise is used as MAE loss to optimize the accuracy of the model in predicting noise, expressed as: ; The calculation steps of ID gait information constraint items are as follows: For the denoising results at all time steps during training, predict , the prediction results and the original gait sequence are input into the gait feature extractor to obtain the gait identity feature embedding of the two sequences, and the Euclidean distance is calculated as the similarity measure for them. The loss function can be expressed as: ; Indicates the extraction of feature embedding, and the overall loss function of the model can be expressed as: ; and are the coefficients of the two losses respectively.

[0012] Preferably, in step S4, it is assumed that a plurality of consecutive gait single-frame images are obtained, which are recorded as , is the s-1th frame image in the gait sequence, and “action” means that this unit responds to the action frame at the current moment based on the state information of the historical frames in the sequence. The “action” information is represented as The “action transfer strategy” indicates the strategy based on which the aforementioned “action” is made, which involves how to obtain the current “action” representation from several known consecutive gait frames. This process is modeled as , "Reward" is used to give feedback to the "action" made by the "action transfer strategy"; in the registration process, the "reward" is calculated by the pixel-level error between the current "action" variable and the given target gait frame, which can be expressed as: ; Based on the "state" information at the current moment, that is, the continuous single-frame gait images in the sequence, a reasonable "action" variable is obtained in response to verify the physical interpretability of the gait frames in the sequence generated by the denoising model and make corrections. This process can be expressed as: ; To generate the s-th frame image in the sequence, is the real / corrected image of the three frames before the sth frame; To generate the corrected s-th frame image in the sequence, the output of the interpretable correction unit is the final generated gait sequence.

[0013] A cross-view gait recognition system based on diffusion model under missing view, comprising a data acquisition module, a gait feature extractor, a feature space transformer, a denoising module and an interpretable correction unit; Data acquisition module: obtain the original data set; Using the original data set to train the gait feature extractor, feature space transformer, and reinforcement learning module; The gait feature extractor includes a spatial encoder and a temporal encoder. For a single gait sequence of each ID in the original data set, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are input into the temporal encoder to fuse the temporal features. After that, the feature dimension is transformed through the feature classification head to obtain the gait identity embedding feature. Feature space transformer: If the gait identity embedding feature pixel of the original gait sequence is greater than the threshold, the size of the single frame image is reduced and then input into the denoising model for training. After the training is completed, the image size is restored to complete the reconstruction of the gait data; Denoising module: Using the existing U-net structure as the basic framework, the gait identity embedding features extracted by the gait feature extractor and the clue frames used to guide gait generation are added, and the gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. After multiple iterations, a gait sequence is generated from Gaussian pure noise sampling; the denoised generated gait sequence is input into the reinforcement learning module in the interpretable correction unit for prediction; Interpretable correction unit: Based on the constructed unified dataset, reinforcement learning is used to align the gait sequence; for a gait sequence containing several consecutive frames, the image of the b-th frame is calibrated based on the limb movement pattern in the previous b-1 frames to correct the physically incoherent movements that may appear in the gait sequence generated by the denoising module; the entire generated sequence is corrected by iterating step by step from the first frame to the last frame, and a generated gait sequence with interpretable continuous limb movements is obtained.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. The cross-view gait sequence generation model proposed in the present invention can use the existing gait information of a certain ID identity in any view as a constraint, and based on the clue frame in any missing view, generate the gait sequence of the same ID identity in the missing view, thereby providing a missing view gait sequence generation method based on a cross-view conversion pathway.

[0015] 2. The present invention proposes a physically interpretable correction unit to correct the gait sequence generated by the generative model in accordance with the objective laws of physics, align the outlier limb movements in the generated sequence to a reasonable position, and realize the correction of the coherence and interpretability of the generated sequence.

[0016] 3. The method proposed in the present invention can generate high-quality and physically interpretable missing gait sequences under the condition of missing perspective. The generated sequences can effectively expand the missing perspective of existing data and supplement the limited real gait sequences. It greatly alleviates the problem that the recognition performance of the cross-perspective gait recognition model drops sharply due to the difficulty in collecting full-perspective gait data, and improves the cross-perspective gait recognition accuracy of the model under the condition of missing perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A cross-view gait recognition framework based on a diffusion model under missing view conditions.

[0018] Figure 2 Schematic diagram of gait feature extractor.

[0019] Figure 3 Schematic diagram of the gait denoising model.

[0020] Figure 4 Schematic diagram of forward denoising and reverse denoising of DDPM.

[0021] Figure 5 Schematic diagram of the interpretable correction unit.

[0022] Figure 6 A framework for interpretable gait sequence generation based on diffusion models.

[0023] Figure 7 This is a comparison chart of the existing perspective sequence and the generated missing perspective sequence of a certain ID in CAISA-B.

[0024] Figure 8 Generate sequence comparison plots before and after application of interpretable correction units. DETAILED DESCRIPTION

[0025] In order to facilitate the understanding of the present invention, the present invention is further described below through embodiments in conjunction with the accompanying drawings, but the present invention is not limited thereto and is not intended to be limited thereto.

[0026] Embodiment 1: A cross-view gait recognition method based on diffusion model under missing view conditions, such as Figure 1 As shown in the figure, the method includes (1) generating missing view sequences by a gait sequence generation method based on a diffusion model, (2) using the existing sequences and the generated sequences under the missing view to jointly train a gait recognition model, and (3) after the training is completed, realizing the cross-view gait recognition task under the missing view condition, which specifically includes the following contents: A. Construction of a unified indoor / outdoor dataset: We select the four most widely used independent datasets: CASIA-B, OUMVLP, GREW, and GAIT3D, and organize them into a unified data structure to construct a unified data set containing multiple fixed indoor perspectives and outdoor random perspectives.

[0027] B. Design of gait feature extractor; including a spatial encoder, a temporal encoder, and a feature classification head. For gait sequences, the spatial encoder is used to extract spatial features, which are then input into the temporal encoder to fuse temporal features. The feature classification head is then used to transform the feature dimension to obtain gait identity feature embedding.

[0028] C. Design of feature space transformer: Based on variational autoencoder, a feature space transformer is constructed. It consists of two parts: the first half is the autoencoder and the second half is the feature decoder. It first encodes and then decodes to restore the original input gait sequence image to achieve gait data reconstruction.

[0029] D. Gait denoising model design: Based on U-net as the basic framework, additional time step information, target sequence view information, gait identity feature embedding extracted from step B, and clue frames used to guide gait generation are added. After multiple iterations, a gait sequence is generated from Gaussian pure noise sampling.

[0030] E. Design of interpretable correction unit: For a gait sequence containing several consecutive frames, the limb movement patterns in the first three frames are used as the reference to calibrate the image of the fourth frame to correct the physically incoherent movements that may appear in the gait sequence generated in step D. Iterate step by step from the first frame to the last frame to complete the correction of the entire generated sequence and obtain a generated gait sequence with interpretable continuous limb movements.

[0031] F. Missing view gait sequence generation: The above-mentioned interpretable gait sequence generation method is used as a gait sequence conversion path under any different perspectives. The gait identity feature embedding is extracted for a complete gait sequence under any existing perspective of a certain ID, and random Gaussian noise is input into the denoising model. The arbitrary missing perspective to be generated is used as the category information, and the clue frame in the target sequence to be generated is used to guide the denoising model to generate the gait sequence under the target missing perspective. After the interpretable correction unit, the gait sequence data of this ID under this missing perspective is generated.

[0032] G. Cross-view gait recognition based on gait sequence generation based on diffusion model under missing view conditions: The restricted gait sequence under the missing view condition is input into the cross-view gait recognition framework, and the gait sequence conversion pathway under any of the above-mentioned viewpoints is used to realize the generation of gait sequences from the existing viewpoint to the missing viewpoint. The gait recognition model is trained using all the expanded gait sequences, and the trained model is used to complete the cross-view gait recognition under the missing view condition.

[0033] Embodiment 2: The difference between the cross-view gait recognition method based on diffusion model for gait sequence generation under the condition of missing view described in Example 1 is that: In step A, the construction of the indoor / outdoor unified dataset includes the following steps: a. Unification of indoor / outdoor data. Four widely used datasets, CASIA-B, OUMVLP, GREW, and GAIT3D, are selected to construct a unified dataset. Among them, CASIA-B and OUMVLP are indoor data, containing gait sequence data of 22 different fixed angles of view, including 0°, 15°, 18°, 30°, 36°, 45°, 48°, 60°, 72°, 75°, 90°, 108°, 126°, 144°, 162°, 180°, 195°, 210°, 225°, 240°, 255°, and 270°. The amount of data from different angles is roughly the same. CASIA-B has different dressing styles such as NM / BG / CL, and OUMVLP only has NM style. The other two datasets, GREW and GAIT3D, are outdoor data, in which the gait sequences of all IDs are at random perspectives, and there is no label information indicating the perspective in the two datasets. The data of different perspectives for each ID are randomly distributed, and the dressing style is also completely random. In order to unify the data labels, the labels of the above 22 different fixed perspectives in the indoor data are set to 0, 1, 2, …, 21, the data perspective labels in GREW are set to 22, and the data perspective labels in GAIT3D are set to 23. Clothing style is not used as auxiliary information, so the unified dataset does not restrict clothing style. The data of different datasets are merged, and the training sets and test sets of all data are unified according to the directory structure of "identity / clothing style / perspective / .pkl". Specifically, one sample is taken from each of the original four datasets, and the unified path is as follows: * / CASIA-B-123 / style / 126 / 126.pkl; * / OUMVLP-10307 / style / 195 / 195.pkl; * / GAIT3D-0145 / style / camid16_videoid3-seq0 / seq0.pkl; * / GREW-18264train / style / jZrDRX4y / jZrDRX4y.pkl.

[0034] In step B, the gait feature extractor is designed, including the following steps: b. Given a gait sequence After being mapped to the token sequence, it is input into the Transformer structure to extract features. Figure 2 Specifically, the model consists of two independent encoders, each of which is implemented using a 6-layer Transformer. The first is a spatial encoder that only models the interaction between different tokens in the same frame to obtain the representation of each frame. All frame-level representations Connected together, through the second part of the temporal encoder, modeling the interaction between tokens from different frames, we get The 6-layer CNN structure is used to implement the feature classification head, and the feature classification head is used to resize the output of the time encoder to obtain the gait feature encoding. , expressed as: .

[0035] In step C, the design of the feature space transformer includes the following steps: c. The feature space transformer is used as a pre-module of the denoising model to reduce the size of a single frame image while retaining the gait sequence information when training the denoising model, making the denoising model lighter and easier to use. The feature space transformer is implemented based on the variational autoencoder structure, which includes an autoencoder and a feature decoder. The original gait sequence Input the autoencoder, and after feature extraction, two feature codes are obtained, called mean coding and variance coding , where the mean encoding represents the mean of the Gaussian distribution that the input gait sequence obeys after being encoded, and the variance encoding is the variance of the Gaussian distribution, which can be expressed as: ; By using the mean and variance to randomly sample the Gaussian distribution, the feature obtained is 2 times smaller than the original sequence single frame size, and the number of channels becomes 4. When training the denoising model, After adding noise, the input is fed into the denoising model, and the output of the denoising model is then partially restored to the original input gait sequence through the decoder. This process can be expressed as: ; When generating gait sequences, the autoencoder is no longer used and the standard Gaussian noise is directly sampled. , the denoising result is generated by the denoising model , and then restored to the original size multi-frame sequence through the decoder , the whole process can be expressed as: .

[0036] In step D, the gait denoising model is designed, including the following steps: Taking U-net as the basic framework, we add additional time step information, target sequence view information, gait identity feature embedding extracted from step B, and clue frames to guide gait generation. The framework is as follows: Figure 3 shown.

[0037] d. The process of denoising diffusion probability model DDPM (Denoising Diffusion Probabilistic Model) is as follows Figure 4 As shown, it includes two processes: forward denoising and reverse denoising. For the convenience of expression, the original data without denoising is expressed as , adding noise step by step according to time t=1,2,3,…,1000 to obtain .

[0038] The forward noise addition process uses a linear noise addition method. The noise variance added at each time step is fixed and increases linearly with time. Specifically, the amount of noise increases linearly with the increase of time step t. When reaching a larger time step, the original data is approximately white noise.

[0039] Define the total number of diffusion steps to 1000 and the variance of the initial noise , the variance of the final noise is Since the noise variance increases linearly, the noise variance at time step t is: ; Using the recursive relation, we can get: ; is the noise sampled from a standard Gaussian distribution. Let , the data of time step t obtained by adding noise from time step t-1 can be expressed as: ;

[0040] Using the recursive relation, we can get Add noise and get Representation: ;

[0041] in express .

[0042] In the reverse denoising process, at the current time step t, based on the noisy data Data before sampling and noise addition , gradually denoising to generate the original gait sequence. According to Bayesian theorem, from the known The probability distribution of The probability distribution of can be expressed as: ; All terms on the right side of the equation obey Gaussian distribution, so the terms on the left side also obey Gaussian distribution. In the gait denoising model, the gait feature encoding extracted in step B is used. As the target gait constraint, the clue frame of the sequence to be generated Guide the generation of gait sequences. Assuming that the target viewing angle to be generated is 0°, the label of the corresponding viewing angle information is 0. When the denoising feature of the 499th time step is obtained from the 500th time step, the probability distribution of the reverse denoising process based on the model can be expressed as: ; ; Furthermore, for the convenience of optimization, the predicted denoised data is converted to predict the noise at the current time step t=500: .

[0043] Among them, Gaitdiff is the denoising model, 500 is the current time step, and 0 is the viewing angle of the target sequence to be generated. are the parameters of the network model, , indicates that both the mean and variance are represented by the model, is the noise at time step 500.

[0044] e. Loss function: There are two loss functions in this generative model. The first is the noise prediction loss. At the current step t=500, the prediction noise output by the model is arrive The added noise The loss optimizes the accuracy of the model’s prediction of noise and is expressed as: ; Another loss function is the ID gait information constraint term. For the denoising results at all time steps during training, the prediction , the prediction results and the original gait sequence are input into the gait feature extractor to obtain the gait identity feature embedding of the two sequences, and the Euclidean distance is calculated as the similarity measure. The better the denoising result of the model, the closer the distance between the two parts of the feature; and the larger the Euclidean distance, the stronger the gait constraint is imposed on the denoising model. This loss function can be expressed as: ; Represents the extracted feature embedding. The overall loss function of the model can be expressed as: .

[0045] and are the coefficients of the two losses, respectively. , .

[0046] In step E, the modified unit design may be explained, including the following steps: f. Use reinforcement learning to perform image registration of multiple consecutive frames in the gait sequence; the gait sequence generation denoising model generates a complete denoised gait sequence at one time , use the explainable correction unit to correct this sequence. This unit consists of four parts: "state, action, action transfer strategy, and reward". , where the "state" variable is , "Action" means that this unit responds to the state of the current frame based on the state information of the 12th, 13th, and 14th frames in the sequence. The "action" information is represented as The “action transfer strategy” indicates the strategy used by this correction unit to make the aforementioned “action” based on the aforementioned state. This involves how to obtain the current “action” representation from several known consecutive gait frames. This process is modeled as , After using this loss to train this correction unit, it can respond to reasonable "action" variables based on the "state" information at the current moment, that is, the continuous single-frame gait images at the historical moment, to verify the physical interpretability of the gait frames in the sequence generated by the denoising model and make corrections. This process can be expressed as: ; Indicates the 15th frame image in the generated sequence. It means the three frames before the 15th frame, namely the 12th, 13th and 14th frames have been corrected images. represents the 15th frame image after correction in the generated sequence. For a generated complete gait sequence ( ), using the clue frame as the first three frames, correcting the 4th frame in the generated sequence, and using it as an auxiliary correction of the 5th frame, and so on, until the 30th frame in the sequence, the output of the interpretable correction unit is the final generated gait sequence. This process is as follows Figure 5 shown.

[0047] In step F, the missing view gait sequence is generated, including the following steps: g. If Figure 6 As shown in the figure, for a certain ID identity, generating its gait sequence under a missing perspective has the following steps: First, use the gait feature extractor to extract the existing gait sequence of the ID Extracting identity embeddings As the gait constraint of the ID identity; then randomly sample from the standard Gaussian distribution to obtain the initial noisy sequence ; The perspective information of the target missing perspective gait sequence to be generated , gait identity feature embedding , the target missing view cue frame generated by the guide sequence , and the sampled noisy sequence The gait sequence is input into the denoising model together, and after multiple iterations, the gait sequence generated under the target missing perspective is obtained after complete denoising. ; For the sequence generated by the model, the physical interpretable correction unit is input for final verification, and the discontinuous limb movements that deviate from the objective physical laws in the generated continuous gait frames are corrected for registration, and finally the ID is output in the target missing perspective Physically interpretable gait sequences .

[0048] In step G, cross-view gait recognition based on the diffusion model under the condition of missing view includes the following steps: For the missing view dataset - ={90°}, where only the pedestrian gait sequence at 90° is available, and other angles are missing. Take the existing gait sequence at 90° , input the cross-view gait recognition framework, use the above-mentioned diffusion model-based interpretable gait sequence generation method to extract gait features from the 90° gait sequence, and assist in completing the generation of gait sequences under 10 missing view angles of 0°, 18°, 36°, 54°, 72°, 108°, 126°, 144°, 162°, and 180°. Four classic gait recognition models GaitSet, GaitPart, Gaitbase, and Deepgaitv2 are selected, and the four gait recognition models are trained using the expanded gait sequences of all 11 angles. The trained models are used to complete the cross-view gait recognition under the missing view condition (only training data of 90° view). First, the model is trained with data from the 90° view and the recognition results are recorded. Subsequently, gait sequences of other view angles are generated by the present invention, the data set is expanded and the model is retrained, and the recognition results are recorded again. Finally, the recognition accuracy of the models trained with the original data and the expanded data is compared to evaluate the performance improvement effect of the present invention. Table 1 is the comparison result of this experiment.

[0049] Table 1 Comparison of cross-view gait recognition accuracy of mainstream gait recognition methods based on missing view and CASIA-B after missing view generation

[0050] Note: The missing viewing angle condition is only 90°.

[0051] In this embodiment, in order to verify the beneficial effect 1 of the present invention, as Figure 7 The gait sequence images of ID=35, clothing style NM, and angle of 126° in the CASIA-B dataset are compared with the gait sequence images with missing angle of 72° generated by the present invention, showing the beneficial effects of the present invention.

[0052] In order to verify the beneficial effect 2 of the present invention, that is, the effectiveness of the interpretable correction unit, the conditional information of ID=35, clothing style NM, and angle 72° in the CASIA-B data set is selected to generate a gait sequence, and the interpretable correction unit is used to compare the generated sequence before and after correction. Figure 8 shown.

[0053] According to Table 1, Figure 7 It can be seen that the cross-view gait recognition method based on the diffusion model under the condition of missing perspective effectively improves the robustness and generalization of the gait recognition model, and improves the recognition accuracy of the four classic gait recognition models GaitSet, GaitPart, GaitBase, and DeepGaitv2 in the NM style under the condition of missing perspective by 103.1%, 106.2%, 90.8%, and 114.4%, respectively. The method proposed in the present invention can generate high-quality and physically interpretable missing gait sequences under the condition of missing perspective. The generated sequences can effectively expand the missing perspective of existing data and supplement the limited real gait sequences, which greatly alleviates the problem of the difficulty in collecting full-view gait data, resulting in a sharp drop in the recognition performance of the cross-view gait recognition model, and improves the cross-view gait recognition accuracy of the model under the condition of missing perspective.

Claims

1. A cross-view gait recognition method based on diffusion model under missing view, characterized in that: The following steps are involved: S1. Obtain the original data set and perform data preprocessing. The processed data set has a uniform directory structure. The data set is used to train the gait feature extractor, feature space transformer and reinforcement learning model. S2. extracting gait identity embedding features in the data set using a gait feature extractor; The gait feature extractor includes a spatial encoder and a temporal encoder. For a single gait sequence of each ID in the data set, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are input into the temporal encoder to fuse the temporal features. After that, the feature dimension is transformed through the feature classification head to obtain the gait identity embedding feature. S3. If the original gait sequence is greater than or equal to the threshold, the size of the single frame image is reduced by using the feature space transformer and then input into the denoising model for training; if the gait identity embedded feature pixel is less than the threshold, then it is input into the denoising model for training; After training, the feature space transformer restores the image size and completes the reconstruction of the gait data; The denoising model uses the existing U-net structure as the basic framework, adds the gait identity embedding features extracted by the gait feature extractor, and the clue frame used to guide gait generation, integrates the gait identity embedding features and clue frames into the cross-attention interaction of the denoising process, and generates gait sequences from Gaussian pure noise sampling after multiple iterations; S4. The denoised generated gait sequence is input into the reinforcement learning module, and the discontinuous limb movements that deviate from the objective physical laws in the generated continuous gait frames under the missing perspective are corrected, and finally the gait sequence image of the ID under this missing perspective is output.

2. The cross-view gait recognition method based on diffusion model under missing view according to claim 1 is characterized in that: The original data set includes indoor data and outdoor data; CASIA-B and OUMVLP are indoor data, which contain gait sequence data of multiple different and fixed perspectives, and the amount of data from different perspectives is roughly the same. CASIA-B has different dressing styles of NM, BG, and CL, and OUMVLP has only NM style; GREW and GAIT3D are outdoor data, in which the gait sequences of all IDs are random perspectives. There is no label information indicating the perspective in the dataset, and the data of different perspectives of each ID are randomly distributed, and the dressing style is also completely random. In order to unify the data labels, the labels of all different fixed perspectives in the indoor data are set to be sorted (0, 1, 2, ..., j), the data perspective label in GREW is set to j+1, and the data perspective label in GAIT3D is set to j+2. The unified dataset has no restrictions on dressing style. The processed data set has a uniformly set directory structure and is divided into training set and test set.

3. The cross-view gait recognition method based on diffusion model under missing view according to claim 1, characterized in that: The gait feature extractor is based on several Transformer layers to design a spatial encoder and a temporal encoder respectively, and a feature classification head based on several CNN layers to design the gait sequence in the dataset. After being mapped to the token sequence, it is input into the Transformer structure to extract features; The spatial encoder only models the interaction between different tokens in the same frame to obtain the representation of each frame. , all frame-level representations Connected together, after the interaction between tokens from different frames of the time encoder, we get ; Finally, the feature classification head is used to resize the output of the encoder to obtain the gait feature encoding , expressed as: ； Among them, S1 is the length of the gait sequence, H is the height of the image, W is the width of the image, and C1 is the channel.

4. The cross-view gait recognition method based on diffusion model under missing view according to claim 1, characterized in that: The feature space transformer consists of two parts. The first part is an autoencoder, which outputs two learnable parameters. The two parameters are sampled from Gaussian noise as mean and variance respectively, and then gradually restored to the original input gait sequence image through the feature decoder in the second part, thereby realizing the reconstruction of gait data.

5. The cross-view gait recognition method based on diffusion model under missing view according to claim 4, characterized in that: S31. If the original gait sequence is greater than or equal to the threshold, the gait sequence Input the autoencoder, and after feature extraction, two feature codes are obtained, namely, mean code and variance coding , where the mean encoding represents the mean of the Gaussian distribution that the input gait sequence obeys after being encoded, and the variance encoding is the variance of the Gaussian distribution, which can be expressed as: ； By using the mean and variance to randomly sample the Gaussian distribution, the obtained features are reduced by n times compared to the original sequence single frame size; ； S32. When generating gait sequences, the autoencoder is no longer used and is directly sampled from standard Gaussian noise. ; At the same time, the gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. The output of the denoising model is then partially restored to the original input gait sequence through the decoder. This process is expressed as: 。 6. The cross-view gait recognition method based on diffusion model under missing view according to claim 5, characterized in that: The denoising model includes two processes: forward denoising and reverse denoising. Assume that the original denoised data is expressed as , according to time t=1,2,3,…,T, gradually add noise to get the data set ; The forward noise addition process uses a linear noise addition method. The noise variance added at each time step is fixed. The noise amount increases linearly with the increase of time step t. When reaching a larger time step, the original data is close to white noise. The variance of the initial noise is , the variance of the final noise is , since the noise variance increases linearly, the noise variance at time step t is as follows: ； Using the recursive relation, we get: ； is the noise sampled from the standard Gaussian distribution; let , the data of time step t obtained by adding noise from time step t-1 can be expressed as: ； Using the recurrence relation, we can easily get Add noise and get Indicates: ； in express ; () indicates Gaussian distribution; In the reverse denoising process, at the current time step t, based on the noisy data Data before sampling and noise addition , gradually denoise to generate the original gait sequence; according to Bayesian theorem, from the known The probability distribution of The probability distribution of can be expressed as: ； Encoding with gait features As the target gait constraint, the clue frame of the sequence to be generated To guide the generation of gait sequences, the probability distribution of the model-based reverse denoising process is expressed as: ； ； Convert the predicted denoised data into the noise for predicting the current time step t as follows: ； Where c is the viewing angle of the target sequence to be generated, are the parameters of the network model, , indicates that both the mean and variance are predicted by the model, is the noise at time step t.

7. The cross-view gait recognition method based on diffusion model under missing view according to claim 6, characterized in that: The denoising model includes a noise prediction loss function and an ID gait information constraint term, where the noise prediction loss function is calculated as follows: In the current step t, the predicted noise output by the model is arrive The added noise is used as MAE loss to optimize the accuracy of the model in predicting noise, expressed as: ； The calculation steps of ID gait information constraint items are as follows: For the denoising results at all time steps during training, predict , the prediction results and the original gait sequence are input into the gait feature extractor to obtain the gait identity feature embedding of the two sequences, and the Euclidean distance is calculated as the similarity measure for them. The loss function can be expressed as: ； Indicates the extraction of feature embedding, and the overall loss function of the model can be expressed as: ； and are the coefficients of the two losses respectively.

8. The cross-view gait recognition method based on diffusion model under missing view according to claim 6, characterized in that: In step S4, it is assumed that a number of consecutive gait single-frame images are obtained, denoted as , is the s-1th frame image in the gait sequence. “Action” means that this unit responds to the action frame at the current moment based on the state information of the historical frames in the sequence. The “action” information is represented as , the "action transfer strategy" indicates the strategy based on which the aforementioned "action" is made, which involves how to obtain the current "action" representation from several known continuous gait frames. This process is modeled as , "reward" is used to give feedback to the "action" made by the "action transfer strategy"; in the registration process, the "reward" is calculated by the pixel-level error between the current "action" variable and the given target gait frame, which can be expressed as: ； Based on the "state" information at the current moment, that is, the previous continuous single-frame gait images in the sequence, a reasonable "action" variable is obtained in response to verify the physical interpretability of the gait frames in the sequence generated by the denoising model and make corrections. This process can be expressed as: ； To generate the s-th frame image in the sequence, is the real / corrected image of the three frames before the sth frame; To generate the corrected s-th frame image in the sequence, the output of the interpretable correction unit is the final generated gait sequence.

9. A cross-view gait recognition system based on a diffusion model under missing view, adopting the cross-view gait recognition method based on a diffusion model under missing view according to any one of claims 1 to 8, characterized in that: It includes a data acquisition module, a gait feature extractor, a feature space transformer, a denoising module and an interpretable correction unit; Data acquisition module: obtain the original data set; Using the original data set to train the gait feature extractor, feature space transformer, and reinforcement learning module; The gait feature extractor includes a spatial encoder and a temporal encoder. For a single gait sequence of each ID in the original data set, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are input into the temporal encoder to fuse the temporal features. After that, the feature dimension is transformed through the feature classification head to obtain the gait identity embedding feature. Feature space transformer: If the gait identity embedding feature pixel of the original gait sequence is greater than the threshold, the size of the single frame image is reduced and then input into the denoising model for training. After the training is completed, the image size is restored to complete the reconstruction of the gait data; Denoising module: Using the existing U-net structure as the basic framework, the gait identity embedding features extracted by the gait feature extractor and the clue frames used to guide gait generation are added, and the gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. After multiple iterations, a gait sequence is generated from Gaussian pure noise sampling; the denoised generated gait sequence is input into the reinforcement learning module in the interpretable correction unit for prediction; Interpretable correction unit: Based on the constructed unified dataset, reinforcement learning is used to align the gait sequence; for a gait sequence containing several consecutive frames, the image of the bth frame is calibrated based on the limb movement pattern in the previous b-1 frames to correct the physically incoherent movements that may appear in the gait sequence generated by the denoising module; The entire generated sequence is corrected by iterating step by step from the first frame to the last frame, and a generated gait sequence with interpretable continuous limb movements is obtained.

Images