Cross-view gait recognition method and system based on diffusion model in missing perspective

Abstract

The present application belongs to the field of gait recognition technology, and specifically relates to a cross-perspective gait recognition method and system based on a diffusion model under missing perspective. A restricted gait sequence under missing perspective conditions is input into the cross-perspective gait recognition framework, and the gait sequence conversion pathway under any perspective described above is used to generate a gait sequence from the existing perspective to the missing perspective. A gait recognition model is trained using all the expanded gait sequences, and the trained model is used to complete cross-perspective gait recognition under missing perspective conditions. The advantage of this application is that the cross-perspective gait sequence generation model can use the existing gait information of a certain ID identity under any perspective as a constraint, and based on the clue frame under any missing perspective, generate a gait sequence for the same ID identity under the missing perspective, thereby providing a missing perspective gait sequence generation method based on a cross-perspective conversion pathway.

Description

Technical Field

[0001] The present application belongs to the technical field of gait recognition, and specifically relates to a cross-view gait recognition method and system based on a diffusion model in the absence of a view. Background Art

[0002] The scarcity of gait recognition data is one of the main challenges facing the development of this field. Since gait data collection requires specific environments and equipment, such as high-definition cameras or specialized motion sensing devices (such as Kinect), and involves multiple gait perspectives (e.g., different angles, lighting, and clothing), acquiring large-scale gait datasets is costly and time-consuming. Furthermore, gait data involves personal privacy, requiring consideration of issues such as pedestrian privacy protection and data security. This is especially true when data is collected in public places, which can easily lead to privacy disputes. High-quality gait recognition data is often difficult to obtain in large quantities, limiting the construction of large-scale datasets.

[0003] Due to various factors, the existing gait datasets are insufficient in diversity, making it difficult to cover people of different ages, genders, body shapes, and walking habits. Training based on these data affects the generalization ability of the recognition model. The generation of gait data can effectively improve the cross-perspective recognition ability of the gait recognition model under the condition of missing perspective. The current gait data generation methods can be mainly divided into two types. One is to generate a Gait Energy Image (GEI). The GEI image is obtained by averaging all single-frame images in a gait sequence of a person. This type of image completely loses the temporal information in the gait sequence. Therefore, the current gait recognition model no longer uses GEI data for training; the other gait data generation method is to establish a mapping relationship between different perspectives of a person through a generative model. Given a gait sequence of a person shot at perspective 1, the gait sequence of the person shot at perspective 2 is generated. This gait data generation method has great limitations, mainly reflected in two aspects: First, when gait data from a certain perspective needs to be generated, the model needs to input gait data from another specific perspective with this ID (ID is used to represent the identity information of the person in this invention). The specific perspective required depends on how the model divides all different perspectives during training. Once a certain perspective information is missing, the model cannot generate data from another perspective with the corresponding mapping relationship for that perspective. Second, for outdoor gait data, there is no longer explicit perspective label information in the dataset. Gait data from any perspective may exist in all the data. At this time, the generation model that establishes a mapping relationship between different perspectives is completely inapplicable. Summary of the Invention

[0004] This invention focuses on how to improve the cross-view gait recognition performance of the model when any view is missing, especially when the view is missing at extremes. To address the shortcomings of the existing technology, this invention provides a cross-view gait recognition method and system based on a diffusion model. Its technical solution is as follows:

[0005] A cross-view gait recognition method based on a diffusion model under missing view includes the following steps:

[0006] S1. Obtain the original data set and perform data preprocessing. The processed data set has a unified directory structure and is used to train the gait feature extractor, feature space transformer, and reinforcement learning model.

[0007] S2. extracting gait identity embedding features from the data set using a gait feature extractor;

[0008] The gait feature extractor includes a spatial encoder and a temporal encoder. For each gait sequence of each ID in the dataset, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are then fed into the temporal encoder to fuse the temporal features. The feature classification head then transforms the feature dimension to obtain the gait identity embedding feature.

[0009] S3. If the original gait sequence is greater than or equal to the threshold, the feature space transformer is used to reduce the size of the single frame image and then input it into the denoising model for training. If the gait identity embedded feature pixels are less than the threshold, the denoising model is also input for training. After training, the feature space transformer restores the image size to complete the gait data reconstruction.

[0010] The denoising model uses the existing U-net structure as the basic framework, adds the gait identity embedding features extracted by the gait feature extractor and the clue frame used to guide gait generation. The gait identity embedding features and clue frame are integrated into the cross-attention interaction of the denoising process. After multiple iterations, gait sequences are generated from Gaussian pure noise samples.

[0011] S4. Input the denoised generated gait sequence into the reinforcement learning module, and perform registration and correction on the discontinuous limb movements that deviate from the objective physical laws in the continuous gait frames generated under the missing perspective, and finally output the gait sequence image of the ID under this missing perspective.

[0012] Preferably, the original data set includes indoor data and outdoor data; CASIA-B and OUMVLP are indoor data, which contain gait sequence data of multiple different and fixed perspectives, and the amount of data at different perspectives is roughly the same. CASIA-B has different dressing styles of NM, BG, and CL, and OUMVLP has only NM style; GREW and GAIT3D are outdoor data, in which the gait sequences of all IDs are random perspectives, there is no label information indicating the perspective in the data set, and the different perspective data of each ID are randomly distributed, and the dressing style is also completely random. In order to unify the data labels, the labels of all different fixed perspectives in the indoor data are set to be sorted (0, 1, 2, ..., j), the data perspective label in GREW is set to j+1, and the data perspective label in GAIT3D is set to j+2. The unified data set does not restrict the dressing style. The processed data set has a uniformly set directory structure and is divided into a training set and a test set.

[0013] Preferably, the gait feature extractor is based on several Transformer layers to design a spatial encoder and a temporal encoder respectively, and based on several CNN layers to design a feature classification head, and the gait sequence in the dataset is extracted. After being mapped to the token sequence, it is input into the Transformer structure to extract features;

[0014] The spatial encoder only models the interaction between different tokens in the same frame to obtain the representation of each frame. , all frame-level representations Connected together, after the interaction between tokens from different frames of the time encoder, we get Finally, the feature classification head is used to resize the output of the encoder to obtain the gait feature encoding , expressed as:

[0015] ;

[0016] Among them, S1 is the length of the gait sequence, H is the height of the image, W is the width of the image, and C1 is the channel.

[0017] Preferably, the feature space transformer includes a two-part structure. The first half is an autoencoder, which outputs two learnable parameters. The two parameters are respectively used as the mean and variance, sampled from Gaussian noise, and passed through the feature decoder in the second half to gradually restore the original input gait sequence image to achieve gait data reconstruction.

[0018] Preferably, S31. If the original gait sequence is greater than or equal to the threshold, the gait sequence Input the automatic encoder, and after feature extraction, two feature codes are obtained, namely mean code and variance coding , where the mean encoding represents the mean of the Gaussian distribution that the input gait sequence obeys after being encoded, and the variance encoding is the variance of the Gaussian distribution, which can be expressed as:

[0019] ;

[0020] By randomly sampling the Gaussian distribution using the mean and variance, the resulting features are reduced by n times compared to the original sequence single frame size;

[0021] ;

[0022] S32. When generating gait sequences, the autoencoder is no longer used and the gaussian noise is directly sampled. At the same time, the gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. The output of the denoising model is then partially restored to the original input gait sequence through the decoder. This process is expressed as:

[0023] .

[0024] Preferably, the denoising model includes two processes: forward denoising and reverse denoising. Assume that the original denoised data is represented as , according to time t=1,2,3,…,T, gradually add noise to get the data set ;

[0025] The forward noise addition process uses a linear noise addition method. The noise variance added at each time step is fixed. The noise amount increases linearly with the increase of time step t. When the time step is larger, the original data is approximately white noise. The variance of the initial noise is , the variance of the final noise is , since the noise variance increases linearly, the noise variance at time step t is as follows:

[0026] ;

[0027] Using the recurrence relation, we get:

[0028] ;

[0029] is the noise sampled from the standard Gaussian distribution; let , the data of time step t obtained by adding noise from time step t-1 can be expressed as:

[0030] ;

[0031] Using the recurrence relation, we can easily get Add noise and get Indicates:

[0032] ;

[0033] in express ; () indicates Gaussian distribution;

[0034] In the reverse denoising process, at the current time step t, based on the noisy data Data before sampling and noise addition , gradually denoising to generate the original gait sequence; according to Bayesian theorem, from the known The probability distribution of the previous step is obtained The probability distribution of can be expressed as:

[0035] ;

[0036] Encoding with gait features As the target gait constraint, the clue frame of the sequence to be generated To guide the generation of gait sequences, the probability distribution of the model-based reverse denoising process is expressed as:

[0037] ;

[0038] ;

[0039] Convert the predicted denoised data into the noise for predicting the current time step t as follows:

[0040] ;

[0041] Where c is the viewing angle of the target sequence to be generated, are the parameters of the network model, , indicates that both the mean and variance are predicted by the model, is the noise at time step t.

[0042] Preferably, the denoising model includes a noise prediction loss function and an ID gait information constraint term, wherein the noise prediction loss function is calculated as follows: In the current step t, the predicted noise output by the model is arrive The added noise is used as MAE loss to optimize the model’s prediction accuracy, which is expressed as:

[0043] ;

[0044] The calculation steps of ID gait information constraint items are as follows:

[0045] For the denoising results at all time steps during training, the prediction , the prediction results and the original gait sequence are input into the gait feature extractor to obtain the gait identity feature embedding of the two sequences, and the Euclidean distance is calculated as the similarity measure for them. The loss function can be expressed as:

[0046] ;

[0047] Indicates the extracted feature embedding, and the overall loss function of the model can be expressed as:

[0048] ;

[0049] and are the coefficients of the two losses respectively.

[0050] Preferably, in step S4, it is assumed that a plurality of consecutive gait single-frame images are obtained, which are recorded as , is the s-1th frame image in the gait sequence, and “action” means that this unit responds to the action frame at the current moment based on the state information of the historical frames in the sequence. The “action” information is represented as The “action transfer strategy” indicates the strategy based on which the aforementioned “action” is made, which involves how to obtain the current “action” representation from several known consecutive gait frames. This process is modeled as , “reward” is used to give feedback on the “action” made by the “action transfer strategy”; in the registration process, “reward” is calculated using the pixel-level error between the current “action” variable and the given target gait frame, which can be expressed as:

[0051] ;

[0052] Based on the "state" information at the current moment, that is, the previous continuous single-frame gait images in the sequence, a reasonable "action" variable is obtained in response to verify the physical interpretability of the gait frames in the sequence generated by the denoising model and make corrections. This process can be expressed as:

[0053] ;

[0054] To generate the s-th frame image in the sequence, is the real / corrected image of the three frames before the sth frame; To generate the corrected s-th frame image in the sequence, the output of the interpretable correction unit is the final generated gait sequence.

[0055] A cross-view gait recognition system based on a diffusion model under missing view, including a data acquisition module, a gait feature extractor, a feature space transformer, a denoising module and an interpretable correction unit;

[0056] Data acquisition module: obtain the original data set;

[0057] The original data set is used to train the gait feature extractor, feature space transformer, and reinforcement learning module;

[0058] The gait feature extractor includes a spatial encoder and a temporal encoder. For each gait sequence of each ID in the original data set, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are input into the temporal encoder to fuse the temporal features. The feature classification head then transforms the feature dimension to obtain the gait identity embedding feature.

[0059] Feature Space Transformer: If the gait identity embedding feature pixels of the original gait sequence are larger than the threshold, the size of the single frame image is reduced and input into the denoising model for training. After training, the image size is restored to complete the reconstruction of the gait data.

[0060] Denoising module: Using the existing U-net structure as the basic framework, the gait identity embedding features extracted by the gait feature extractor and the clue frames used to guide gait generation are added. The gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. After multiple iterations, a gait sequence is generated from Gaussian pure noise samples. The denoised gait sequence is input into the reinforcement learning module in the interpretable correction unit for prediction.

[0061] Interpretable correction unit: Based on the constructed unified dataset, reinforcement learning is used to align gait sequences. For a gait sequence containing several consecutive frames, the image of the b-th frame is calibrated based on the limb movement pattern in the previous b-1 frames to correct the physical incoherence that may appear in the gait sequence generated by the denoising module. The entire generated sequence is corrected by iterating step by step from the first frame to the last frame, resulting in a generated gait sequence with interpretable continuous limb movements.

[0062] Compared with the prior art, this application has the following beneficial effects:

[0063] 1. The cross-view gait sequence generation model proposed in the present invention can use the existing gait information of a certain ID identity under any view as a constraint, and based on the clue frames under any missing view, generate the gait sequence of the same ID identity under the missing view, providing a missing view gait sequence generation method based on a cross-view conversion pathway.

[0064] 2. The present invention proposes a physically interpretable correction unit to correct the gait sequence generated by the generative model in accordance with the objective laws of physics, align the outlier limb movements in the generated sequence to reasonable positions, and achieve correction of the coherence and interpretability of the generated sequence.

[0065] 3. The method proposed in the present invention can generate high-quality and physically interpretable missing gait sequences under the condition of missing perspective. The generated sequences can effectively expand the missing perspective of existing data and supplement the limited real gait sequences. It greatly alleviates the problem of the difficulty in collecting full-perspective gait data, which leads to a sharp drop in the recognition performance of the cross-perspective gait recognition model, and improves the cross-perspective gait recognition accuracy of the model under the condition of missing perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A cross-view gait recognition framework based on diffusion model under missing view conditions.

[0067] Figure 2 Schematic diagram of gait feature extractor.

[0068] Figure 3 Schematic diagram of the gait denoising model.

[0069] Figure 4 Schematic diagram of DDPM's forward denoising and reverse denoising.

[0070] Figure 5 Schematic diagram of the interpretable correction unit.

[0071] Figure 6 A framework for interpretable gait sequence generation based on diffusion models.

[0072] Figure 7 This is a comparison chart of the sequence under the existing perspective and the generated sequence under the missing perspective for a certain ID in CAISA-B.

[0073] Figure 8 Generate sequence comparison plots before and after application of interpretable correction units. DETAILED DESCRIPTION

[0074] In order to facilitate understanding of the present invention, the present invention is further described below through examples in conjunction with the accompanying drawings, but the present invention is not limited thereto and is not intended to be limited thereto.

[0075] Example 1:

[0076] A cross-view gait recognition method based on diffusion model under missing view conditions, such as Figure 1As shown in the figure, the method includes (1) generating missing view sequences based on the diffusion model gait sequence generation method, (2) using the existing sequence and the generated missing view sequence to jointly train the gait recognition model, and (3) achieving the cross-view gait recognition task under the missing view condition after the training is completed. Specifically, the method includes the following contents:

[0077] A. Construction of a unified indoor / outdoor dataset: We select the four most widely used independent datasets: CASIA-B, OUMVLP, GREW, and GAIT3D, and organize them into a unified data structure to construct a unified data set that includes multiple fixed indoor viewpoints and random outdoor viewpoints.

[0078] B. Design of the gait feature extractor; it consists of a spatial encoder, a temporal encoder, and a feature classification head. For gait sequences, the spatial encoder extracts spatial features, which are then fed into the temporal encoder to fuse temporal features. The feature classification head then transforms the feature dimensions to produce a gait identity embedding.

[0079] C. Design of the Feature Space Transformer: Based on a variational autoencoder, a feature space transformer is constructed. This structure consists of two parts: the first half is the autoencoder, and the second half is the feature decoder. This part first encodes and then decodes the original input gait sequence image to reconstruct the gait data.

[0080] D. Gait denoising model design: Using U-net as the basic framework, we additionally incorporate time step information, target sequence viewpoint information, the gait identity feature embedding extracted in step B, and clue frames to guide gait generation. After multiple iterations, we generate a gait sequence from Gaussian pure noise samples.

[0081] E. Design of an interpretable correction unit: For a gait sequence consisting of several consecutive frames, the limb motion patterns in the first three frames are used as a reference to calibrate the image of the fourth frame to correct any physical incoherence that may occur in the gait sequence generated in step D. This correction is completed by iterating stepwise from the first frame to the last frame, resulting in a generated gait sequence with interpretable continuous limb motions.

[0082] F. Generation of gait sequences with missing perspectives:

[0083] The above-mentioned interpretable gait sequence generation method is used as a pathway for converting gait sequences from any different perspectives. The gait identity feature embedding is extracted from a complete gait sequence of a certain ID from any existing perspective. Random Gaussian noise is input into the denoising model. The missing perspective to be generated is used as the category information. The clue frames in the target sequence to be generated are used to guide the denoising model to generate the gait sequence from the target missing perspective. After passing through the interpretable correction unit, the gait sequence data for this ID from this missing perspective is generated.

[0084] G. Cross-view gait recognition based on gait sequence generation based on diffusion model under missing view conditions:

[0085] The restricted gait sequence under the missing view condition is input into the cross-view gait recognition framework, and the gait sequence conversion pathway under any view mentioned above is used to realize the generation of gait sequence from the existing view to the missing view. The gait recognition model is trained with all the expanded gait sequences, and the trained model is used to complete the cross-view gait recognition under the missing view condition.

[0086] Example 2:

[0087] The difference between the cross-view gait recognition method based on diffusion model gait sequence generation under the condition of missing view described in Example 1 is that:

[0088] In step A, the construction of the unified indoor / outdoor dataset includes the following steps:

[0089] a. Unification of indoor and outdoor data. We selected four widely used datasets, CASIA-B, OUMVLP, GREW, and GAIT3D, to construct a unified dataset. CASIA-B and OUMVLP are indoor datasets, containing gait sequence data at 22 different fixed view angles: 0°, 15°, 18°, 30°, 36°, 45°, 48°, 60°, 72°, 75°, 90°, 108°, 126°, 144°, 162°, 180°, 195°, 210°, 225°, 240°, 255°, and 270°. The amount of data from different view angles is roughly the same. CASIA-B includes different clothing styles, such as NM / BG / CL, while OUMVLP only has NM style. The other two datasets, GREW and GAIT3D, are outdoor data. The gait sequences for all IDs are randomized, and neither dataset contains any labels indicating the viewpoint. The data for each ID is randomly distributed across different viewpoints, and the clothing style is also completely random. To unify the data labels, the labels for the 22 different fixed viewpoints in the indoor data are set to 0, 1, 2, …, 21. The viewpoint labels for GREW and GAIT3D are set to 22 and 23, respectively. Clothing style is not used as auxiliary information, so the unified dataset does not impose any restrictions on clothing style. The data from the different datasets is merged, and the training and test sets for all data are unified according to the directory structure of "identity / clothing style / viewpoint / .pkl". Specifically, one example is taken from each of the four original datasets, and the unified path is as follows:

[0090] * / CASIA-B-123 / style / 126 / 126.pkl;

[0091] * / OUMVLP-10307 / style / 195 / 195.pkl;

[0092] * / GAIT3D-0145 / style / camid16_videoid3-seq0 / seq0.pkl;

[0093] * / GREW-18264train / style / jZrDRX4y / jZrDRX4y.pkl.

[0094] In step B, the gait feature extractor is designed, including the following steps:

[0095] b. Given a gait sequence After being mapped to the token sequence, it is input into the Transformer structure to extract features. Figure 2 Specifically, the model consists of two independent encoders, each implemented with a 6-layer Transformer. The first is a spatial encoder that only models the interaction between different tokens in the same frame to obtain the representation of each frame. All frame-level representations Connected together, through the second part of the time encoder, modeling the interaction between tokens from different frames, we get The 6-layer CNN structure is used to implement the feature classification head, and the feature classification head is used to resize the output of the time encoder to obtain the gait feature encoding. , expressed as:

[0096] .

[0097] In step C, the design of the feature space transformer includes the following steps:

[0098] c. The feature space transformer is used as a pre-module for the denoising model. It is used to reduce the size of a single frame image while retaining the gait sequence information when training the denoising model, making the denoising model lighter and easier to use. The feature space transformer is implemented based on the variational autoencoder structure and consists of an autoencoder and a feature decoder. The original gait sequence Input the autoencoder, and after feature extraction, two feature codes are obtained, called mean codes and variance coding , where the mean encoding represents the mean of the Gaussian distribution that the input gait sequence obeys after being encoded, and the variance encoding is the variance of the Gaussian distribution, which can be expressed as:

[0099] ;

[0100] By randomly sampling the Gaussian distribution using the mean and variance, the feature obtained is 2 times smaller than the original sequence single frame size, and the number of channels becomes 4. When training the denoising model, After adding noise, the input is fed into the denoising model, and the output of the denoising model is then passed through the decoder to restore the original input gait sequence. This process can be expressed as:

[0101] ;

[0102] When generating gait sequences, the autoencoder is no longer used and the gaussian noise is directly sampled. , the denoising result is generated by the denoising model , and then restored to the original size multi-frame sequence through the decoder , the whole process can be expressed as:

[0103] .

[0104] In step D, the gait denoising model is designed, including the following steps:

[0105] Taking U-net as the basic framework, we add time step information, target sequence view information, gait identity feature embedding extracted from step B, and clue frames to guide gait generation. The framework is as follows: Figure 3 shown.

[0106] d. The process of denoising diffusion probability model DDPM (Denoising Diffusion Probabilistic Model) is as follows Figure 4 As shown, it includes two processes: forward denoising and reverse denoising. For the convenience of expression, the original data without noise is expressed as , according to the time t=1,2,3,…,1000, the noise is gradually added to obtain .

[0107] The forward noise addition process uses a linear noise addition method. The noise variance added at each time step is fixed and increases linearly with time. Specifically, the amount of noise increases linearly with the increase of time step t. When the original data reaches a larger time step, it is close to white noise.

[0108] Define the total number of diffusion steps to 1000 and the variance of the initial noise to be , the variance of the final noise is Since the noise variance increases linearly, the noise variance at time step t is:

[0109] ;

[0110] Using the recurrence relation, we can get:

[0111] ;

[0112] is the noise sampled from the standard Gaussian distribution. Let , the data of time step t obtained by adding noise from time step t-1 can be expressed as:

[0113] ;

[0114] Using the recursive relation, we can get Add noise and get Indicates:

[0115] ;

[0116] in express .

[0117] In the reverse denoising process, at the current time step t, based on the noisy data Data before sampling and noise addition , gradually denoising to generate the original gait sequence. According to Bayesian theorem, from the known The probability distribution of the previous step is obtained The probability distribution of can be expressed as:

[0118] ;

[0119] All terms on the right side of the equation obey Gaussian distribution, so the terms on the left side also obey Gaussian distribution. In the gait denoising model, the gait feature encoding extracted in step B is used. As the target gait constraint, the clue frame of the sequence to be generated Guide the generation of gait sequences. Assuming that the target viewing angle to be generated is 0°, the label of the corresponding viewing angle information is 0. When the denoising feature of the 499th time step is obtained from the 500th time step, the probability distribution of the model-based reverse denoising process can be expressed as:

[0120] ;

[0121] ;

[0122] Furthermore, for the convenience of optimization, the predicted denoised data is converted into the noise of the current time step t=500:

[0123] .

[0124] Among them, Gaitdiff is the denoising model, 500 is the current time step, and 0 is the viewing angle of the target sequence to be generated. are the parameters of the network model, , Indicates that both the mean and variance are represented by the model, is the noise at time step 500.

[0125] e. Loss function: There are two loss functions in this generative model. The first is the noise prediction loss. At the current step t=500, the prediction noise output by the model is arrive The added noise The loss, which optimizes the accuracy of the model’s prediction of noise, is expressed as:

[0126] ;

[0127] Another loss function is the ID gait information constraint term. For the denoising results at all time steps during training, the prediction The predicted results and the original gait sequence are input into the gait feature extractor to obtain the gait identity feature embeddings for the two sequences. The Euclidean distance between them is calculated as a similarity measure. The better the denoising result of the model, the closer the distance between the two features; the larger the Euclidean distance, the stronger the gait constraints imposed on the denoising model. This loss function can be expressed as:

[0128] ;

[0129] Represents the extracted feature embedding. The overall loss function of the model can be expressed as:

[0130] .

[0131] and are the coefficients of the two losses, respectively. , .

[0132] In step E, the modified cell design may be explained, including the following steps:

[0133] f. Use reinforcement learning to perform image registration of multiple consecutive frames in a gait sequence; the gait sequence generation and denoising model generates a complete denoised gait sequence at one time , use the interpretable correction unit to correct this sequence. This unit consists of four parts: "state, action, action transfer strategy, and reward". , where the "state" variable is , “Action” means that this unit responds to the state of the current frame based on the state information of the 12th, 13th, and 14th frames in the sequence. The “action” information is represented as The “action transfer strategy” indicates the strategy by which this correction unit makes the aforementioned “action” based on the aforementioned state. This involves how to obtain the current “action” representation from several known consecutive gait frames. This process is modeled as After using this loss to train this correction unit, it is possible to obtain reasonable "action" variables based on the "state" information at the current moment, that is, the continuous single-frame gait images at the historical moment, to verify the physical interpretability of the gait frames in the sequence generated by the denoising model and make corrections. This process can be expressed as:

[0134] ;

[0135] Indicates the 15th frame image in the generated sequence, Indicates that the three frames before the 15th frame, namely the 12th, 13th, and 14th frames have been corrected. Represents the 15th frame image after correction in the generated sequence. For a generated complete gait sequence ( ), using the clue frame as the first three frames, correct the 4th frame in the generated sequence, and use it as an auxiliary correction for the 5th frame, and so on, until the 30th frame in the sequence, the output of the interpretable correction unit is the final generated gait sequence. This process is as follows Figure 5 shown.

[0136] In step F, the missing view gait sequence is generated, including the following steps:

[0137] g. If Figure 6 As shown in the figure, for a certain ID identity, the steps to generate its gait sequence under a missing perspective are as follows: First, use the gait feature extractor to extract the existing gait sequence of the ID Extracting identity embeddings As the gait constraint of the ID identity; then randomly sample from the standard Gaussian distribution to obtain the initial noisy sequence ; The perspective information of the target missing perspective gait sequence to be generated , gait identity feature embedding , target missing view clue frame generated by the guide sequence , and the sampled noisy sequence The gait sequence is input together to generate the denoising model. After multiple iterations, the gait sequence generated under the target missing perspective is obtained after complete denoising. For the sequence generated by the model, the physical interpretable correction unit is input for final verification, and the discontinuous limb movements that deviate from the objective physical laws in the generated continuous gait frames are corrected for registration, and finally the ID is output in the target missing perspective. Physically interpretable gait sequences .

[0138] In step G, cross-view gait recognition based on the diffusion model under the condition of missing view includes the following steps:

[0139] For the missing view dataset - ={90°}, where only the pedestrian gait sequence at 90° is available and other angles are missing. Take the existing gait sequence at 90° The cross-view gait recognition framework is then fed into the framework. The aforementioned interpretable gait sequence generation method based on the diffusion model is used to extract gait features from the 90° gait sequence, assisting in the generation of gait sequences for 10 missing view angles: 0°, 18°, 36°, 54°, 72°, 108°, 126°, 144°, 162°, and 180°. Four classic gait recognition models, GaitSet, GaitPart, Gaitbase, and Deepgaitv2, are selected and trained using the expanded gait sequences from all 11 view angles. The trained models are then used to perform cross-view gait recognition under the missing view angle condition (training data from only the 90° view angle). First, the models are trained using only the 90° view angle data, and the recognition results are recorded. Subsequently, gait sequences for other view angles are generated using the present invention, the dataset is expanded, the models are retrained, and the recognition results are again recorded. Finally, the recognition accuracy of the models trained with the original and expanded data is compared to evaluate the performance improvement achieved by the present invention. Table 1 shows the comparison results from this experiment.

[0140] Table 1 Comparison of cross-view gait recognition accuracy of mainstream gait recognition methods based on CASIA-B with missing view and missing view generation

[0141]

[0142] Note: The missing viewing angle condition is only 90°.

[0143] In this embodiment, in order to verify the beneficial effect 1 of the present invention, as Figure 7 The comparison of the gait sequence images of ID=35, clothing style NM, and angle of 126° in the CASIA-B dataset with the gait sequence images generated by the present invention with a missing angle of 72° shows the beneficial effects of the present invention.

[0144] In order to verify the beneficial effect 2 of the present invention, that is, the effectiveness of the interpretable correction unit, the conditional information of ID=35, clothing style NM, and angle 72° in the CASIA-B dataset is selected to generate a gait sequence, and the interpretable correction unit is applied to compare the generated sequence before and after correction. Figure 8 shown.

[0145] According to Table 1, Figure 7 It can be seen that the diffusion model-based cross-view gait recognition method under missing view conditions effectively improves the robustness and generalization of the gait recognition model, improving the recognition accuracy of the four classic gait recognition models, GaitSet, GaitPart, GaitBase, and DeepGaitv2, in the NM style under missing view conditions by 103.1%, 106.2%, 90.8%, and 114.4%, respectively. The method proposed in this invention can generate high-quality and physically interpretable missing gait sequences under missing view conditions. The generated sequences can effectively expand the missing view of existing data and supplement the limited real gait sequences. This greatly alleviates the problem of the difficulty in collecting full-view gait data, which leads to a sharp drop in the recognition performance of the cross-view gait recognition model, and improves the cross-view gait recognition accuracy of the model under missing view conditions.

Claims

1. A cross-view gait recognition method based on diffusion model under missing view, characterized by: The following steps are involved: S1. Obtain the original data set and perform data preprocessing. The processed data set has a unified directory structure and is used to train the gait feature extractor, feature space transformer, and reinforcement learning model. S2. extracting gait identity embedding features from the data set using a gait feature extractor; The gait feature extractor includes a spatial encoder and a temporal encoder. For each gait sequence of each ID in the dataset, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are then fed into the temporal encoder to fuse the temporal features. The feature classification head then transforms the feature dimension to obtain the gait identity embedding feature. S3. If the original gait sequence is greater than or equal to the threshold, the single-frame image is reduced in size using the feature space transformer and then input into the denoising model for training. If the gait identity embedded feature pixels are less than the threshold, the denoising model is then input into the denoising model for training. After training, the feature space transformer restores the image size and completes the reconstruction of the gait data; The denoising model uses the existing U-net structure as the basic framework, adds the gait identity embedding features extracted by the gait feature extractor and the clue frame used to guide gait generation. The gait identity embedding features and clue frame are integrated into the cross-attention interaction of the denoising process. After multiple iterations, gait sequences are generated from Gaussian pure noise samples. S4. Input the denoised generated gait sequence into the reinforcement learning module, and perform registration and correction on the discontinuous limb movements that deviate from the objective physical laws in the continuous gait frames generated under the missing perspective, and finally output the gait sequence image of the ID under this missing perspective.

2. The cross-view gait recognition method based on diffusion model under missing view according to claim 1 is characterized in that: The original data set includes indoor data and outdoor data; CASIA-B and OUMVLP are indoor data, containing gait sequence data from multiple different and fixed perspectives, and the amount of data from different perspectives is roughly the same. CASIA-B has different dressing styles (NM, BG, and CL), while OUMVLP only has NM style. GREW and GAIT3D are outdoor data, in which the gait sequences of all IDs are random perspectives. There is no label information indicating the perspective in the dataset, and the data from different perspectives of each ID are randomly distributed, and the dressing style is also completely random. In order to unify the data labels, the labels of all different fixed perspectives in the indoor data are set to be sorted (0, 1, 2, ..., j), the data perspective label in GREW is set to j+1, and the data perspective label in GAIT3D is set to j+2. The unified dataset does not restrict the dressing style. The processed data set has a uniformly set directory structure and is divided into training and test sets.

3. The cross-view gait recognition method based on diffusion model in the absence of view according to claim 1, characterized in that: The gait feature extractor is based on several Transformer layers to design a spatial encoder and a temporal encoder respectively, and a feature classification head based on several CNN layers to design a gait sequence in the dataset. After being mapped to the token sequence, it is input into the Transformer structure to extract features; The spatial encoder only models the interaction between different tokens in the same frame to obtain the representation of each frame. , all frame-level representations Connected together, after the interaction between tokens from different frames of the time encoder, we get Finally, the feature classification head is used to resize the output of the encoder to obtain the gait feature encoding , expressed as: ； Among them, S1 is the length of the gait sequence, H is the height of the image, W is the width of the image, and C1 is the channel.

4. The cross-view gait recognition method based on diffusion model under missing view according to claim 1, characterized in that: The feature space transformer consists of two parts. The first half is an autoencoder, which outputs two learnable parameters. The two parameters are used as the mean and variance, sampled from Gaussian noise, and then passed through the feature decoder in the second half to gradually restore the original input gait sequence image, thereby realizing gait data reconstruction.

5. The cross-view gait recognition method based on diffusion model under missing view according to claim 4 is characterized in that: S31. If the original gait sequence is greater than or equal to the threshold, the gait sequence Input the automatic encoder, and after feature extraction, two feature codes are obtained, namely mean code and variance coding , where the mean encoding represents the mean of the Gaussian distribution that the input gait sequence obeys after being encoded, and the variance encoding is the variance of the Gaussian distribution, which can be expressed as: ； By randomly sampling the Gaussian distribution using the mean and variance, the resulting features are reduced by n times compared to the original sequence single frame size; ； S32. When generating gait sequences, the autoencoder is no longer used and the gaussian noise is directly sampled. At the same time, the gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. The output of the denoising model is then partially restored to the original input gait sequence through the decoder. This process is expressed as: 。 6. The cross-view gait recognition method based on diffusion model in the absence of view according to claim 5, characterized in that: The denoising model includes two processes: forward denoising and reverse denoising. Assume that the original denoised data is represented as , according to time t=1,2,3,…,T, gradually add noise to get the data set ; The forward noise addition process uses a linear noise addition method. The noise variance added at each time step is fixed. The noise amount increases linearly with the increase of time step t. When the time step is larger, the original data is approximately white noise. The variance of the initial noise is , the variance of the final noise is , since the noise variance increases linearly, the noise variance at time step t is as follows: ； Using the recurrence relation, we get: ； is the noise sampled from the standard Gaussian distribution; let , the data of time step t obtained by adding noise from time step t-1 can be expressed as: ； Using the recurrence relation, we can easily get Add noise and get Indicates: ； in express ; () indicates Gaussian distribution; In the reverse denoising process, at the current time step t, based on the noisy data Data before sampling and noise addition , gradually denoising to generate the original gait sequence; according to Bayesian theorem, from the known The probability distribution of the previous step is obtained The probability distribution of can be expressed as: ； Encoding with gait features As the target gait constraint, the clue frame of the sequence to be generated To guide the generation of gait sequences, the probability distribution of the model-based reverse denoising process is expressed as: ； ； Convert the predicted denoised data into the noise for predicting the current time step t as follows: ； Where c is the viewing angle of the target sequence to be generated, are the parameters of the network model, , Indicates that both the mean and variance are predicted by the model, is the noise at time step t.

7. The cross-view gait recognition method based on diffusion model under missing view according to claim 6, characterized in that: The denoising model includes a noise prediction loss function and an ID gait information constraint term, wherein the noise prediction loss function is calculated as follows: In the current step t, the predicted noise output by the model is arrive The added noise is used as MAE loss to optimize the model’s prediction accuracy, which is expressed as: ； The calculation steps of ID gait information constraint items are as follows: For the denoising results at all time steps during training, the prediction , the prediction results and the original gait sequence are input into the gait feature extractor to obtain the gait identity feature embedding of the two sequences, and the Euclidean distance is calculated as the similarity measure for them. The loss function can be expressed as: ； Indicates the extracted feature embedding, and the overall loss function of the model can be expressed as: ； and are the coefficients of the two losses respectively.

8. The cross-view gait recognition method based on diffusion model under missing view according to claim 6, characterized in that: In step S4, it is assumed that a number of consecutive gait single-frame images are obtained, which are recorded as , is the s-1th frame image in the gait sequence. “Action” means that this unit responds to the action frame at the current moment based on the state information of the historical frames in the sequence. The “action” information is represented as , the "action transfer strategy" indicates the strategy based on which the aforementioned "action" is made, which involves how to obtain the current "action" representation from several known consecutive gait frames. This process is modeled as , "reward" is used to give feedback on the "action" made by the "action transfer strategy"; during the registration process, the "reward" is calculated using the pixel-level error between the current "action" variable and the given target gait frame, which can be expressed as: ； Based on the "state" information at the current moment, that is, the previous continuous single-frame gait images in the sequence, a reasonable "action" variable is obtained in response to verify the physical interpretability of the gait frames in the sequence generated by the denoising model and make corrections. This process can be expressed as: ； To generate the s-th frame image in the sequence, is the real / corrected image of the three frames before the sth frame; To generate the corrected s-th frame image in the sequence, the output of the interpretable correction unit is the final generated gait sequence.

9. A cross-view gait recognition system based on a diffusion model in the absence of a view, using the cross-view gait recognition method based on a diffusion model in the absence of a view according to any one of claims 1 to 8, characterized in that: It includes data acquisition module, gait feature extractor, feature space transformer, denoising module and interpretable correction unit; Data acquisition module: obtain the original data set; The original data set is used to train the gait feature extractor, feature space transformer, and reinforcement learning module; The gait feature extractor includes a spatial encoder and a temporal encoder. For each gait sequence of each ID in the original data set, the spatial encoder is used to extract spatial features from each frame of the sequence. The spatial features of all frames are input into the temporal encoder to fuse the temporal features. The feature classification head then transforms the feature dimension to obtain the gait identity embedding feature. Feature Space Transformer: If the gait identity embedding feature pixels of the original gait sequence are larger than the threshold, the size of the single frame image is reduced and input into the denoising model for training. After training, the image size is restored to complete the reconstruction of the gait data. Denoising module: Using the existing U-net structure as the basic framework, the gait identity embedding features extracted by the gait feature extractor and the clue frames used to guide gait generation are added. The gait identity embedding features and clue frames are integrated into the cross-attention interaction of the denoising process. After multiple iterations, a gait sequence is generated from Gaussian pure noise samples. The denoised gait sequence is input into the reinforcement learning module in the interpretable correction unit for prediction. Interpretable Correction Unit: Based on the constructed unified dataset, reinforcement learning is used to align gait sequences. For a gait sequence consisting of several consecutive frames, the image of the bth frame is calibrated based on the limb movement patterns in the previous b-1 frames to correct for any physical incoherence that may occur in the gait sequence generated by the denoising module. The entire generated sequence is modified by iterating step by step from the first frame to the last frame, and a generated gait sequence with interpretable continuous limb movements is obtained.

Images