A multi-view gait recognition method based on joint loss function

Abstract

The present invention provides a multi-view gait recognition method based on a joint loss function, which includes: using a GaitSet model structure, only needing to calculate the representation of each sample once, and being able to perform classification and recognition by measuring the Euclidean distance between the representations of different samples; using a joint loss function combining the two loss functions of Softmax and Triplet for joint training, increasing the distance between gait example classes, and reducing the distance within the gait example class, so that the model improves the representation ability under the optimization of multiple loss functions. This application proposes a gait recognition method with loss joint supervision, which enhances the generalization ability of the model, and improves the recognition accuracy in the two walking states of carrying packages and wearing coats without increasing the amount of model calculation. When the suspect covers his face, camouflages himself, and the lighting conditions are poor, face recognition technology is limited, and gait recognition technology can be indirectly applied to public security practice as an important supplement to face recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of video detection, and in particular to a multi-view gait recognition method based on a joint loss function. Background Art

[0002] Video investigation technology has the advantages of breaking through time and space limitations and fixing evidence, and has gradually become a new means for public security to handle cases. In video investigation work, target identification and tracking have always been an important part of case detection, and its efficiency and accuracy are crucial. Criminals are often fully disguised during the crime process, and because of their increasing awareness of anti-detection, many technical means are limited. Existing face recognition technology is mainly used in portrait camera mounts or scenes equipped with high-definition monitoring equipment. In the process of committing crimes, criminal suspects will deliberately use camera blind spots and disguise facial image information to make their identities unrecognizable. Gait recognition can use existing cameras for long-distance identification, and it is difficult for suspects to hide and disguise. It has obvious advantages in video investigation and has become a hot topic in the field of identity recognition research in recent years.

[0003] Gait recognition is a method that analyzes the walking posture of pedestrians and uses algorithms to identify them from the gait database. Gait recognition has a high recognition accuracy when pedestrians do not change their outer clothes, carry nothing, and the camera direction does not change. However, in the face of complex real-life scenarios, this technology still has many difficult issues to address, such as uneven walking surfaces and changes in camera angles that can cause perspective problems, and whether or not a person is wearing a coat or carrying a package that can cause occlusion problems. These factors result in a low recognition accuracy in complex actual walking processes.

[0004] At present, there are two main methods for gait recognition. The first method regards gait as an image, and a typical representative is to extract the gait energy map. The gait energy map is the most commonly used feature map in gait recognition tasks. Its main work is to align and compress a set of silhouette images into an image, and use the gait template to classify pedestrians. In the gait energy map, by calculating the average gait silhouette over a long period of time, the noise remaining due to incomplete processing in the preprocessing stage can be more fully reduced. The gait energy map is one of the most commonly used gait recognition methods at present. It can achieve a relative balance between the amount of calculation and the recognition accuracy, and has always been regarded as a synonym for gait. However, it cannot be ignored that since this method does not take into account the temporal information and spatial information in the gait, its gait recognition accuracy is not high.

[0005] The second method treats gait as a video sequence and directly extracts gait features from the initial silhouette sequence. LSTM, 3D-CNN or two stream methods are often used. Both video and speech signals contain temporal information, and LSTM models are usually used to process this information with temporal order. First, convolutional neural networks are used to extract gait features from each frame of video, and then LSTM is used to model the temporal relationship. 3D-CNN models can process the extra time dimension of videos compared to images, so they are often used in video classification. However, due to their heavy computational tasks and long computational cycles, they are often difficult to be truly applied in practice.

[0006] The dual-stream method is a video behavior recognition method that includes two channels, RGB image and optical flow. The optical flow channel is used to model temporal information, and the RGB image channel is used to model spatial information. This method fuses the information of the two channels and conducts joint training, so that the temporal and spatial information in the gait sequence can be well modeled, but it is often limited by some external factors.

[0007] In order to balance the amount of computation and the accuracy of recognition, Chao et al. proposed a new solution: consider the gait features as a set of image sequences composed of video frames. Walking is a periodic movement, so gait can be represented by one of the cycles. In a gait sequence of a cycle, it can be observed that the silhouettes at different positions have shapes that can be distinguished from other silhouettes, such as Figure 1 shown.

[0008] Just by observing their general shapes, they can be distinguished one by one. Even when faced with gait silhouettes that have been completely disrupted, they can be reassembled into their original correct order. Therefore, it is reasonable to assume that the appearance of the gait silhouette itself contains the order, and then the order information of the silhouette will no longer be a necessary input. The gait can be directly regarded as a set of images to extract the time sequence information. Based on this theory, Chao et al. proposed a GaitSet deep learning network model. The network framework is as follows Figure 2 .

[0009] This application is based on deep learning gait recognition technology, and uses the trplet and softmax joint loss function in the GaitSet model to improve the gait recognition accuracy of pedestrians in the two situations of wearing coats and carrying packages.

[0010] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0011] The purpose of the present invention is to provide a multi-view gait recognition method based on a joint loss function to improve the accuracy of gait recognition of pedestrians in the two situations of wearing coats and carrying packages.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions:

[0013] The present invention provides a multi-view gait recognition method based on a joint loss function, characterized in that it comprises the following steps:

[0014] S1: Using the GaitSet model structure, the representation of each sample only needs to be calculated once, and classification and recognition can be performed by measuring the Euclidean distance between the representations of different samples;

[0015] S2: A joint loss function combining Softmax and Triplet loss functions is used for joint training to increase the distance between gait example classes and reduce the distance within gait example classes, so that the model can improve its representation ability under the optimization of multiple loss functions.

[0016] Preferably, the Triplet loss function in step S2 is used to train examples with small differences, and its specific formula is (1):

[0017]

[0018] Among them, L t Represents the loss, a represents the anchor example, p represents the positive example, n represents the negative example, and N represents the number of all examples in a Batch.

[0019] Preferably, the Softmax loss function in step S2 is used to process the image multi-classification task, and its specific formula is (2):

[0020]

[0021] Where Ls represents the loss, Xi represents the feature vector of the i-th gait image, yi is the true label of Xi, b is the bias, and They represent the weight vectors for distinguishing Xi into categories yi and j respectively, C represents the total number of categories, represents the score of the gait image in category yi, and N represents the total number of training samples.

[0022] Preferably, the specific formula of the joint loss function in step S2 is (3):

[0023] Loss = αL t +βL s (3)

[0024] Among them, α and β represent the weights of the Triplet and Softmax loss functions respectively. When α=0 and β≠0, it is the Softmax loss function, when β=0 and α≠0, it is the Triplet loss function, and when α≠0 and β≠0, it is the joint loss function.

[0025] By adopting the above technical solution, the present invention has the following beneficial effects:

[0026] The present invention proposes a gait recognition method jointly supervised by Triplet loss and Softmax loss, which enhances the generalization ability of the model and improves the recognition accuracy in the two walking states of carrying packages and wearing coats without increasing the model calculation amount. Due to the complexity of the actual scene, gait recognition technology is currently difficult to directly apply to video investigation. However, when the suspect covers his face, disguises, or the lighting conditions are poor, face recognition technology will be limited. Gait recognition technology can be indirectly applied to public security practice as an important supplement to face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A gait silhouette for a cycle;

[0029] Figure 2 This is a framework diagram of the GaitSet model structure;

[0030] Figure 3 This is an example of NM type gait image;

[0031] Figure 4 An example of image sequence after preprocessing of CASIA-B dataset;

[0032] Figure 5 It is the data storage structure after CASIA-B preprocessing;

[0033] Figure 6 is the loss function curve;

[0034] Figure 7 Comparison of test results of two types of loss functions under different weights;

[0035] Figure 8This is the accuracy curve of GaitSet trained using only Softmax loss. DETAILED DESCRIPTION

[0036] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The present invention adopts the GaitSet model structure. The GaitSet model is very flexible, and the input will only limit the size of the contour map, which means that the input sequence can contain different angles, walking states with arbitrary combinations, and unlimited and discontinuous silhouettes. This model does not calculate the similarity between a pair of gait sequences, but directly learns the representation of gait. Therefore, it is only necessary to calculate the representation of each sample once, and classification and recognition can be performed by measuring the Euclidean distance between the representations of different samples. GaitSet greatly improves the recognition performance of the two major public data sets CASIA-B and OUMVLP, showing its strong robustness under different viewing angles and multi-walking conditions and its high generalization ability for large data sets.

[0038] The joint loss function combines multiple loss functions to achieve the effect of joint optimization. The present invention adopts the method of joint training of two loss functions, Softmaxloss and Triplet loss, to increase the distance between gait example classes and reduce the distance within gait example classes, so that the model can improve its representation ability under the optimization of multiple loss functions.

[0039] The Triplet loss function is usually used to train examples with small differences. Its specific formula is (1). t Represents the loss, a represents the anchor example, p represents the positive example, n represents the negative example, and N represents the number of all examples in a Batch.

[0040]

[0041] The Softmax loss function is most commonly used to process image multi-classification tasks, and its specific formula is (2). Where Ls represents the loss, Xi represents the feature vector of the i-th gait image, yi is the true label of Xi, and b is the bias. and They represent the weight vectors for distinguishing Xi into categories yi and j respectively, C represents the total number of categories, represents the score of the gait image in category yi, and N represents the total number of training samples.

[0042]

[0043] The specific formula of the final joint loss function is (3). In it, α and β represent the weights of the Triplet and Softmax loss functions respectively. When α=0 and β≠0, it is the Softmax loss function; when β=0 and α≠0, it is the Triplet loss function; when α≠0 and β≠0, it is the joint loss function.

[0044] Loss = αL t +βL s (3)

[0045] The present invention is further described below in conjunction with specific experimental data:

[0046] CASIA-B is a free and open human gait dataset proposed by the Institute of Automation, Chinese Academy of Sciences in 2005. This dataset is a general dataset with advantages such as large scale, multi-angle, and multi-walking conditions, and it has a high status in the field of gait recognition. The gait information in the dataset contains a total of 124 pedestrians with different identities (ID numbers are 1-124). Each pedestrian is recorded from 11 perspectives (0°-180°) in three walking conditions: normal walking (NM), wearing a coat (CL), and carrying a package (BG). The NM walking condition is recorded 6 times, and the CL and BG walking conditions are recorded 2 times each. The gait images of a pedestrian with a certain ID in the video under normal walking conditions and different perspectives are as follows: Figure 3 shown.

[0047] In principle, each data set has 13,640 gait videos and 1,364 background videos, totaling 15,004. The length of the video frames ranges from a few to several hundred, occupying a total of 16GB of memory. During the experiment, in the process of sorting and analyzing the information of the above data sets, it was found that the gait videos of the two pedestrian IDs 026 and 027 could not be used normally because of defects in the data itself, and the outline of the pedestrian in the silhouette image basically did not appear in video No. 109. Therefore, there are a total of 14,614 videos that can be used, of which gait videos account for about 91% of the total, and the rest of the videos account for 9%. After the above data is preprocessed, about 1,116,300 human gait images are generated, occupying about 3.2GB of memory. Figure 4 The following is an example of a silhouette image of a pedestrian with ID 025 after preprocessing, recorded 01 times under normal walking conditions, and with a viewing angle of 90 degrees. The image storage directory is in the format of "CASIA-B / ID / walking status / recording angle / gait image", such as Figure 5 shown.

[0048] Based on the GaitSet network model, the CASIA-B data set is divided in the form of large sample training. As a convention, the divided training set part should be fully input into the GaitSet network for training. However, during the verification process, since some gait images of all pedestrians need to be used as the control group, the other part of the gait images that are not used as the control but used for identification are called probes. Because it is necessary to balance the amount between NM, BG, and CL to prevent too much or too little information on one of the walking conditions, during the experiment, the silhouette images of nm-05, 06, cl-01, bg-01, cl-02, and bg-02 in all pedestrian IDs are used for identification, and the remaining nm-01, 02, 03, and 04 silhouette images constitute the gait library.

[0049] This experiment uses 8 NVIDIA Tesla K80 12G graphics cards, the number of fully connected hidden layers is 256, the learning rate is 0.0001, the number of video extraction frames is 30, the batch size is 8×16, the triple loss Margin is 0.2, and the number of training rounds is 80,000.

[0050] During the training process, 100 rounds are divided into a group, and a total of 800 groups are trained. The average loss function of each group is as follows: Figure 6 As shown in the figure, it can be observed that, unlike the monotonically increasing trend of the triplet loss function alone, the joint loss function shows a monotonically decreasing trend as a whole. Since all the batch data for training are composed of random types of random target IDs, random angles, and random frame sequences, there is a certain degree of oscillation between each adjacent step, so it is reasonable for the loss value to eventually converge to the margin value.

[0051] In the setting of the weight of the joint loss function, different values ​​of α and β in formula (3) will affect the final recognition effect of the model. In order to achieve a higher recognition accuracy in the two cases of wearing a coat and carrying a package, it is necessary to select the optimal α and β values. The present invention uses 11 groups of different weight allocation methods for α and β, namely 1:0, 0.9:0.1, 0.8:0.2, 0.7:0.3, 0.6:0.4, 0.5:0.5, 0.4:0.6, 0.3:0.7, 0.2:0.8, 0.1:0.9, and 0:1, to conduct experiments. The comparison results are shown in FIG. Figure 7 shown.

[0052] When α=0 and β=1, it means that the Softmax loss function is used alone to train GaitSet. The accuracy curve is as follows Figure 8 shown.

[0053] It can be seen that although the final classification accuracy is very high when the Softmax loss function is used alone to train the GaitSet network, Figure 7 In the test results, the model trained with α=0, β=1 has poor recognition accuracy in the three walking conditions, which shows that the Triplet loss function is indispensable. In the test results of the joint training of the Triplet loss function and the Softmax loss function, the walking conditions of wearing a coat and carrying a package have different degrees of improvement under all weights, among which the recognition accuracy of the weight of 0.1:0.9 has the largest improvement. In the test results of the weights of 0.2:0.8, 0.3:0.7, 0.4:0.6, 0.5:0.5, 0.6:0.4, 0.7:0.3, 0.8:0.2, and 0.9:0.1, the recognition accuracy under normal walking conditions has a slight decrease compared with the original GaitSet network, and only the recognition accuracy of the weight of 0.1:0.9 has a slight increase, so we select α=0.1, β=0.9, and the final loss function is shown in formula (4).

[0054] Loss = 0.1L t +0.9L s (4)

[0055] Under the training set division form of LT, the experimental results of the present invention are compared with GaitSet and CNN-LB according to walking conditions, the recognition accuracy under 11 different viewing angles is recorded, and the average of the recognition accuracy of all viewing angles under each walking condition is calculated respectively. The results are shown in Table 1.

[0056] Table 1 Comparison of gait recognition experimental results under multiple perspectives

[0057]

[0058] The recognition performance of the three algorithms in Table 1 under normal walking conditions is not much different, and the recognition accuracy rates are all higher than 91%, especially the GaitSet algorithm and the method of the present invention, both of which are as high as 95%; under the condition of walking with a package, it can be inferred that the three algorithms are affected by the partial occlusion of the pedestrian's body, but relatively speaking, the method of the present invention that uses the joint loss function to train the GaitSet network is the least affected, and the gait recognition accuracy rate exceeds 90%; the three algorithms have unsatisfactory recognition effects for the situation where the pedestrian's body is largely obscured due to wearing a coat, while the method of the present invention achieves a recognition accuracy rate of 75.1%, with relatively better recognition results.

[0059] In summary, in multi-view surveillance videos, the gait recognition effect of carrying packages and wearing coats is generally poor. In view of the above shortcomings, the present invention proposes a gait recognition method jointly supervised by Triplet loss and Softmax loss to enhance the generalization ability of the model. While almost no increase in the amount of model calculation, the recognition accuracy of the two walking states of carrying packages and wearing coats is improved. Due to the complexity of the actual scene, gait recognition technology is currently difficult to directly apply to actual video investigation. However, when the suspect covers his face, disguises, or the lighting conditions are poor, face recognition technology will be limited. Gait recognition technology can be indirectly applied to public security practice as an important supplement to face recognition.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-view gait recognition method based on a joint loss function, characterized in that: The steps include: S1: Using the GaitSet model structure, the representation of each sample only needs to be calculated once, and classification and recognition can be performed by measuring the Euclidean distance between the representations of different samples; S2: A joint loss function combining Softmax and Triplet loss functions is used for joint training to increase the distance between gait examples and reduce the distance within gait examples, so that the model can improve its representation ability under the optimization of multiple loss functions. The Triplet loss function in step S2 is used to train examples with small differences. Its specific formula is (1): Among them, L t represents the loss, a represents the anchor example, p represents the positive example, n represents the negative example, and N represents the number of all examples in a batch; The Softmax loss function in step S2 is used to process the image multi-classification task, and its specific formula is (2): Among them, L s represents the loss, X i Represents the feature vector of the i-th gait image, y i is the true label of Xi, b is the bias, and Respectively represent X i Identify as y i The weight vector of class and class j, C represents the total number of categories, Indicates that the gait image is in category y i The score on, N represents the total number of training samples; The specific formula of the joint loss function in step S2 is (3): Loss=αL t +βL s (3) Among them, α and β represent the weights of the Triplet and Softmax loss functions respectively. When α=0 and β≠0, it is the Softmax loss function, when β=0 and α≠0, it is the Triplet loss function, and when α≠0 and β≠0, it is the joint loss function; Select α=0.1 and β=0.9.

Images