Gait recognition method and device based on causal intervention mechanism

Abstract

The application discloses a gait recognition method based on a cause-effect intervention mechanism, and relates to the technical field of gait recognition. The gait recognition method comprises the following steps: firstly, low-level gait features are obtained through feature extraction and time domain downsampling for a gait sequence to be detected; then, a cause-effect intervention learner composed of multiple fact and counter-fact modules is used for feature mapping to generate gait features. The fact and counter-fact modules can maximize the distribution difference between fact and counter-fact feature expressions, so that the gait model pays more attention to the area reflecting the gait features, thereby reducing the interference of mixed factors on gait recognition, and the modules have strong portability and are suitable for various mainstream gait recognition networks.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically, relates to a gait recognition method and device based on a causal intervention mechanism. Background Art

[0002] Gait recognition is a biometric technology that identifies individuals based on their unique walking pattern. Compared to other biometric technologies such as face, iris, fingerprint, and vein recognition, gait is difficult to spoof and does not require the subject's cooperation during the recognition process. Therefore, gait recognition has been widely used in fields such as intelligent security systems, video surveillance, sports science, and crime prevention. However, gait sequences contain confounding factors, such as walking speed and clothing changes. These confounding factors affect the recognition performance of gait models. Summary of the Invention

[0003] In response to the shortcomings of existing methods, the present invention provides a gait recognition method based on a causal intervention mechanism, which enables the gait model to focus more on the area reflecting gait characteristics, thereby reducing the interference of confounding factors on gait recognition. In addition, the module is highly portable and applicable to a variety of mainstream gait recognition networks.

[0004] To achieve the above object, according to one aspect of the present invention, a gait recognition method based on a causal intervention mechanism is provided, comprising the following steps:

[0005] Step 1: For the gait sequence to be trained and tested, first crop the image to a resolution of 64×44. Sample 30 frames of the training sequence continuously, and input all of them during testing.

[0006] Step 2: The gait sequence processed in step 1 is input into the low-level feature extraction network, which performs downsampling in the time domain to obtain low-level semantic feature representation;

[0007] Step 3: Design a causal intervention learner based on the fact and counterfactual modules, and input the low-level semantic feature expression obtained in step 2 into several consecutive causal intervention learners to obtain high-level semantic feature expression. Assume that the input of the causal intervention learner is First, the fact module generates the fact feature X f ; In the counterfactual module, the feature X is first generated by the convolution module, and the input f i Get the time domain correlation matrix and spatiotemporal characteristics The two matrices are multiplied to obtain the weight W, and then the scope feature X is used to obtain the counterfactual feature X cf , during training, X f and X cfThe causal intervention learner makes the gait model focus more on the area reflecting the gait characteristics, thereby reducing the interference of confounding factors on gait recognition;

[0008] Step 4: Perform feature mapping on the high-level semantic feature expression obtained in step 3 to obtain the final gait feature for training;

[0009] Step 5: Use the gait features mapped in step 4 to predict the pedestrian identity of the gait sequence to be tested.

[0010] In one embodiment of the present invention, the low-level feature extraction network in step 2 is composed of two layers of 3D convolutional neural networks, wherein the output channels of the two layers of convolutional networks are both 32, the convolution kernel size of the first layer is 3×3×3, and the convolution kernel size of the second layer is 3×1×1.

[0011] In one embodiment of the present invention, the number of fact and counterfactual modules in the causal intervention learner in step three is 3.

[0012] In one embodiment of the present invention, for the causal intervention learner, the input f i Generate time domain correlation matrix and spatiotemporal characteristics The function expression is:

[0013]

[0014]

[0015] Where SP represents spatial pooling, CP represents channel pooling, FC represents fully connected layer, SA represents self-attention operation, and σ represents Softmax function.

[0016] In one embodiment of the present invention, the mechanism of the fact and counterfactual module in step 3 is to maximize the distribution difference between the fact and counterfactual feature expressions, and adopt counterfactual loss for supervised training, and its function expression is:

[0017]

[0018] where X f is the factual feature expression, X cf is a counterfactual feature expression.

[0019] In one embodiment of the present invention, the output feature expression of the fact and counterfactual module is X o The expression is:

[0020] X o =X f -Gate(X cf)

[0021] Among them, Gate is a gating mechanism, which mainly enhances the expression of fact features X f , highlighting the characteristic areas of interest in gait.

[0022] In one embodiment of the present invention, the specific expression of the feature mapping in step 4 is:

[0023] y=FC(GeM(TP(x)))

[0024] TP is the time domain pooling, specifically the MaxPooling operation, and GeM is the horizontal aggregation operation. The expression of GeM is: FC fully connected layer, the output dimension is 128.

[0025] In one embodiment of the present invention, in step 4, the network is trained using triplet loss and cross entropy loss, wherein the triplet loss function is expressed as:

[0026]

[0027] where d(·) represents the Euclidean distance, x a and x p are samples from the same object, and x a and x n are samples from different objects, and m is the optimization interval of the triplet loss function.

[0028] In one embodiment of the present invention, the optimization interval m=0.2.

[0029] In one embodiment of the present invention, the testing method in step 5 is specifically as follows:

[0030] The Euclidean distance between the gait feature to be tested and all gait sequence features in the library is calculated, and then sorted from low to high according to the distance size. The one with the smallest distance is the matching sequence of the test gait sequence.

[0031] According to another aspect of the present invention, a gait recognition device based on a causal intervention mechanism is also provided, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the gait recognition method based on the causal intervention mechanism.

[0032] In general, the above technical solutions conceived by the present invention have the following advantages compared with the prior art:

[0033] (1) Strong novelty: Compared with previous gait recognition methods, this invention introduces a causal intervention mechanism and designs factual and counterfactual modules to maximize the distribution difference between factual and counterfactual feature expressions, thereby reducing the interference of confounding factors on gait recognition and helping to improve gait recognition performance;

[0034] (2) Robustness: This method can minimize the influence of confounding factors such as walking speed and clothing changes, thereby improving the performance of the gait recognition model and making it more robust in practical applications such as changing clothes.

[0035] (3) Strong portability: The module designed in the present invention is highly portable and applicable to a variety of mainstream gait recognition networks without changing the original network. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a gait recognition method based on a causal intervention mechanism of the present invention;

[0037] Figure 2 Schematic diagram of the fact and counterfactual modules in the present invention. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0039] The following first explains and illustrates the technical terms of the present invention:

[0040] CASIA-B dataset: This dataset is a large multi-view human gait dataset established by the Institute of Automation, Chinese Academy of Sciences (CASIA), shot in an indoor environment. The dataset contains video sequences from 124 subjects from 11 perspectives, where each perspective contains 6 gait sequences of normal walking (NM), 2 backpack walking (BG), and 2 coat walking (CL). The present invention divides the dataset, and the gait sequences of the first 74 subjects are used for training, and the gait sequences of the last 50 subjects are used for testing. During the test, the first four sequences (NM#1-4) in the NM state are used as the gallery set (gallery), and the remaining sequences are probe sets (prob). The probe sets are divided into 3 subsets according to the walking state, where the NM subset contains sequences NM#5-6, the BG subset contains sequences BG#1-2, and the CL subset contains sequences CL#1-2.

[0041] In order to solve the problems existing in the prior art, the present invention designs a gait recognition method based on a causal intervention mechanism. By maximizing the distribution difference between factual and counterfactual feature expressions, the gait model focuses more on the area reflecting gait characteristics, thereby reducing the interference of confounding factors on gait recognition. In addition, the module is highly portable and applicable to a variety of mainstream gait recognition networks.

[0042] like Figure 1 As shown, the present invention provides a gait recognition method based on a causal intervention mechanism, comprising the following steps:

[0043] Step 1: For the gait sequence to be trained and tested, first crop the image to a resolution of 64×44. Sample 30 frames of the training sequence continuously, and input all of them during testing.

[0044] Step 2: The gait sequence processed in step 1 is input into a low-level feature extraction network. This network consists of two layers of 3D convolutional neural networks. The output channels of both layers are 32. The convolution kernel size of the first layer is 3×3×3, and the convolution kernel size of the second layer is 3×1×1. Therefore, downsampling is performed in the time domain to obtain low-level semantic feature representation.

[0045] Step 3: Figure 2 This is a schematic diagram of the fact and counterfactual module. The module uses counterfactual loss for supervised training by maximizing the distribution difference between fact and counterfactual feature expressions. Its function expression is:

[0046]

[0047] where X f is the factual feature expression, X cf is the counterfactual feature expression. cf The function expression is:

[0048] X cf =X⊙W

[0049] Where W is the weight, which is the time domain correlation matrix and spatiotemporal characteristics Matrix multiplication is obtained. and The expressions are Where SP represents spatial pooling, CP represents channel pooling, FC represents fully connected layer, SA represents self-attention operation, and σ represents Softmax function; the output feature expression of fact and counterfactual module is X o The expression is:

[0050] X o =X f-Gate(X cf )

[0051] Among them, Gate is a gating mechanism, which mainly enhances the expression of fact features X f , which makes the gait model focus more on the areas reflecting gait characteristics, thereby reducing the interference of confounding factors on gait recognition. The module is highly portable and applicable to various mainstream gait recognition networks. Based on the fact and counterfactual modules, a causal intervention learner is designed, which contains a total of three fact and counterfactual modules. The low-level semantic feature expression obtained in step 2 is input into the causal intervention learner to obtain a high-level semantic feature expression;

[0052] Step 4: Perform feature mapping on the high-level semantic feature expression obtained in step 3. The specific expression is:

[0053] y=FC(GeM(TP(x)))

[0054] TP is the time domain pooling, specifically the MaxPooling operation, GeM is the horizontal aggregation operation, and the GeM expression is Y=(AvgPool 1×w (X p )) 1 / p , FC fully connected layer, the output dimension is 128. The final gait features are obtained by mapping for training, where the loss function is triple loss and cross entropy loss. The triple loss function expression is:

[0055] L tri =∑[d(x a ,x p )-d(x a ,x n )+m] +

[0056] where d(·) represents the Euclidean distance, x a and x p are samples from the same object, and x a and x n are samples from different objects, m is the optimization interval of the triplet loss function is 0.2;

[0057] Step 5: Use the gait features mapped in step 4 to predict the pedestrian identity of the gait sequence to be tested. The specific method is to calculate the Euclidean distance between the gait features to be tested and the features of all gait sequences in the library, and then sort them from low to high according to the distance. The sequence with the smallest distance is the matching sequence of the test gait sequence.

[0058] Furthermore, the present invention also provides a gait recognition device based on a causal intervention mechanism, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the gait recognition method based on the causal intervention mechanism.

[0059] The effectiveness of the present invention is demonstrated below through experimental examples, and the experimental results show that the present invention can improve the recognition accuracy of gait recognition.

[0060] The present invention was experimented on the CASIA-B dataset and compared with five existing gait recognition methods. Table 1 compares the results of the present invention with those of the five methods. The results of the present invention were compared under three conditions: NM (normal walking), BG (walking with a backpack), and CL (walking with a coat). Larger numerical values ​​indicate higher gait recognition accuracy. As can be seen from the table, the present invention significantly improves performance, indicating that the causal intervention mechanism can reduce the interference of confounding factors on gait recognition, allowing the gait model to focus more on areas that reflect gait characteristics.

[0061] Table 1 Accuracy of different methods on the CASIA-B dataset

[0062] method NM BG CL average GaitSet 95.0 87.2 70.4 84.2 GaitPart 96.2 91.5 78.7 88.8 GaitGL 97.4 94.5 83.6 91.8 CSTL 97.8 93.6 84.2 91.9 3DLocal 97.5 94.3 83.7 91.8 The present invention 97.9 95.4 85.3 92.9

[0063] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A gait recognition method based on causal intervention mechanism, characterized in that: The method comprises the following steps: Step 1: For the gait sequence to be trained and tested, first crop the image into The training sequence is sampled continuously for 30 frames, and all frames are input during testing. Step 2: The gait sequence processed in step 1 is input into the low-level feature extraction network, which performs downsampling in the time domain to obtain low-level semantic feature representation; Step 3: Design a causal intervention learner based on the factual and counterfactual modules. Input the low-level semantic feature expression obtained in step 2 into several consecutive causal intervention learners to obtain a high-level semantic feature expression. Assume that the input of the causal intervention learner is , firstly, the fact module generates fact features ; Input in the counterfactual module First, the convolution module generates features , and by input Get the time domain correlation matrix and spatiotemporal characteristics , the weights are obtained by multiplying the two matrices , and then through the characteristics Get counterfactual features , during training and The causal intervention learner makes the gait model focus more on the area reflecting the gait characteristics, thereby reducing the interference of confounding factors on gait recognition. Generate time domain correlation matrix and spatiotemporal characteristics The function expression is: , ,in represents spatial pooling, represents channel pooling, represents the fully connected layer, represents the self-attention operation, express function; Step 4: Perform feature mapping on the high-level semantic feature expression obtained in step 3 to obtain the final gait feature for training; Step 5: Use the gait features mapped in step 4 to predict the pedestrian identity of the gait sequence to be tested.

2. The gait recognition method based on causal intervention mechanism according to claim 1 is characterized in that: The low-level feature extraction network in step 2 is composed of two layers of 3D convolutional neural networks, where the output channels of the two layers of convolutional networks are both 32, and the convolution kernel size of the first layer is , the convolution kernel size of the second layer is .

3. The gait recognition method based on causal intervention mechanism according to claim 1, characterized in that: The number of fact and counterfactual modules in the causal intervention learner in step 3 is 3.

4. The gait recognition method based on causal intervention mechanism according to claim 1, characterized in that: The mechanism of the fact and counterfactual module in step 3 is to maximize the distribution difference between the fact and counterfactual feature expressions, and use counterfactual loss for supervised training. Its function expression is: in To express factual characteristics, is a counterfactual feature expression.

5. The gait recognition method based on causal intervention mechanism according to claim 1 or 4, characterized in that: The output feature expression of the fact and counterfactual module The expression is: in It is a gating mechanism that enhances the expression of factual features. , highlighting the characteristic areas of interest in gait.

6. The gait recognition method based on causal intervention mechanism according to claim 1, characterized in that: The specific expression of the feature mapping in step 4 is: in It is time domain pooling, specifically MaxPooling operation, For horizontal aggregation operation, The expression is , Fully connected layer, the output dimension is 128.

7. The gait recognition method based on causal intervention mechanism according to claim 1, characterized in that: In step 4, triplet loss and cross entropy loss are used for training, where the triplet loss function is expressed as: in represents the Euclidean distance, and are samples from the same subject, and and are samples from different objects, is the optimization interval of the triplet loss function.

8. The gait recognition method based on causal intervention mechanism according to claim 1, characterized in that: The testing method in step 5 is specifically as follows: The Euclidean distance between the gait feature to be tested and all gait sequence features in the library is calculated, and then sorted from low to high according to the distance size. The one with the smallest distance is the matching sequence of the gait sequence to be tested.

9. A gait recognition device based on a causal intervention mechanism, characterized by: The method comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the gait recognition method based on the causal intervention mechanism according to any one of claims 1 to 8.

Images