A gait recognition method based on multi-level features of convolutional neural network

Abstract

A gait recognition method based on multi-level features of convolutional neural network includes: reading an input gait image sequence and preprocessing the image sequence; extracting features of different levels from the image sequence through feature extraction branches of different convolution sizes; fusing the extracted features of different levels in different ways to obtain the features finally used for gait recognition and store them; calculating the loss using a loss function according to feature graphs and labels, and updating the convolutional neural network model using a back propagation algorithm until a final recognition model is generated when preset conditions are met; inputting the gait target to be recognized into the final recognition model, and matching the model output result with the stored feature values, and taking the result with the highest similarity to the stored feature values ​​as the recognition result. The present invention performs online data enhancement on the input gait profile graph, increases the diversity of input data, and improves the robustness of the algorithm to pedestrian profiles in actual scenarios.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a gait recognition method based on multi-level features of convolutional neural networks. Background Art

[0002] Gait is a physiological and biological feature that describes a person's walking pattern. Compared with biometric features such as face, fingerprint, and iris, gait recognition technology has the advantages of non-contact, long-distance, difficult to disguise, and no need for active cooperation. Therefore, gait recognition technology is widely used in access control systems, security monitoring, human-computer interaction and other fields.

[0003] With the development of deep learning and convolutional neural network technology, gait recognition technology has also made great progress. The current mainstream gait recognition methods are generally divided into template-based methods and sequence-based methods, both of which use pedestrian gait profiles for recognition. Among them, sequence-based gait recognition methods such as GaitSet and GaitPart algorithms can achieve good results on public data sets, but the GaitSet algorithm is based on the entire profile for recognition and does not capture local features. The GaitPart algorithm can extract local features, but the extracted feature level is relatively simple. The recognition accuracy of both algorithms is somewhat insufficient in actual scenarios. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a gait recognition method based on multi-level features of convolutional neural networks, which overcomes the above problems or at least partially solves the above problems.

[0005] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0006] A gait recognition method based on multi-level features of convolutional neural network, comprising:

[0007] S100. Reading an input gait image sequence and preprocessing the image sequence;

[0008] S200. extracting features of different levels from the image sequence through feature extraction branches with different convolution sizes;

[0009] S300. Fusing the extracted features at different levels in different ways to obtain the final features for gait recognition and store them;

[0010] S400. According to the feature graph and the label, the loss is calculated using the loss function, and the convolutional neural network model is updated using the back propagation algorithm until the preset conditions are met to generate the final recognition model;

[0011] S500. Input the gait target to be identified into the final recognition model, and match the model output result with the stored feature values, and take the result with the highest similarity to the stored feature values ​​as the recognition result.

[0012] Furthermore, the specific method of S100 is: download the CASIA-B open source gait recognition dataset, judge the length of the gait sequence in the dataset, take the middle 30 images if the sequence length is greater than 150, and discard the sequence if the sequence length is less than 30; crop the input gait sequence to 64*64 size according to the edge and center line principles in the height and width directions.

[0013] Furthermore, the specific method of S100 also includes: randomly performing morphological opening or closing operations on the image according to a certain ratio, and the kernel size of the opening and closing operations is set to 3. During training, the ratio is set to 0.2, and during inference, the ratio is set to 0; the number of input sequences each time is P*K, where P is the number of pedestrians in each batch during training or inference, and K is the number of gait sequences of each selected pedestrian under different viewing angles, clothing, backpacks or walking conditions.

[0014] Furthermore, the specific method of S200 includes: using three types of convolutions, 1x1, 3x3, and 5x5, to perform feature extraction of different granularities on the input image sequence respectively. The number of input channels of the three convolutions is 1, the number of output channels is 32, the step size is 2, and the input length and width are 64*64. The n image sequences of length s=30 are extracted for different granularity features; then the three 32*32 feature maps are concatenated in the channel dimension through the Concat layer, and finally the feature maps containing information of different levels are obtained.

[0015] Furthermore, the Block layer consists of two convolution layers with a convolution size of 3 and a stride of 1 and a maximum pooling layer with a stride of 2. The number of input channels of the first convolution layer is 96, and the number of output channels is 64. The number of input channels of the second convolution layer is 64, and the number of output channels is 128. The feature map containing different levels of information extracted by the Block layer is output as n*30*128*16*16.

[0016] Furthermore, the specific method of S300 includes: using Block_W layer, Block_H layer, HPM layer, VPM layer and Concat2 layer to fuse the multi-level features extracted by S200 in the w and h directions of the feature map; wherein, the Block_W layer and the Block_H layer are both composed of 2 layers of convolution with a convolution size of 3 and a stride of 1, the input and output channel numbers of the first layer of convolution are 128 and 256 respectively, and the input and output channel numbers of the second layer of convolution are 256 and 256 respectively.

[0017] Furthermore, the specific method of S300 also includes: the Block_W layer divides the features output by the feature extraction layer into n equal parts in the w direction, and uses a convolution operation to extract local features from the n strip feature maps segmented in the w direction, and then concats these features together, and then uses the global maximum pooling to extract the strip local information in the w direction; the Block_H layer divides the features output by the feature extraction layer into n equal parts in the h direction, and uses a convolution operation to extract local features from the n strip feature maps segmented in the h direction, and then concats these features together, and then uses the global maximum pooling to extract the strip local information in the h direction; the HPM layer uses an independent fully connected layer to map each h-direction pooled feature to the discriminant space; the VPM layer uses an independent fully connected layer to map each w-direction pooled feature to the discriminant space; the Concat2 layer contains a concat operation to splice the output feature maps of the HPM layer and the VPM layer together in the channel dimension to form a feature map containing local features in both the w and h directions.

[0018] Furthermore, in S400, Treplitloss and Centerloss are used for joint training, and the loss function is designed as follows:

[0019]

[0020]

[0021] LOSS=L tri +β·L Cen

[0022] Among them, the joint loss is composed of the ternary loss L tri and the center loss L Cen and the weight balance factor β of the two losses; the ternary loss L tri In the above equation, N is the number of samples in a training batch. is a sample randomly selected during training. The positive sample corresponding to the training sample is is the negative sample corresponding to the training sample, is the intra-class Euclidean distance between the randomly selected sample and similar samples, is the inter-class Euclidean distance between the randomly selected sample and samples of different classes, α is the minimum difference between the inter-class distance and the intra-class distance corresponding to the sample, and the center loss L Cen Medium i is a sample randomly selected during training, and c is the class center of all samples, which is obtained during the training process.

[0023] Furthermore, the training process is optimized using the Adam algorithm, with the learning rate set to 0.0001 and the momentum to 0.9. If the loss of the training process is less than 0.0000001 or the number of iterations is greater than 100,000, the training is stopped.

[0024] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0025] The present invention discloses a gait recognition method based on multi-level features of convolutional neural networks, which performs online data enhancement on an input gait profile map, increases the diversity of input data, and improves the robustness of the algorithm to pedestrian profiles in actual scenarios; uses convolutions with different receptive fields to extract gait information and expresses the gait with different local features in the height h and width w directions, and uses ternary loss and center loss simultaneously in the training process of the algorithm, which increases the distance between different classes and reduces the distance between the same class, thereby improving the accuracy of gait recognition.

[0026] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0028] Figure 1 This is a flow chart of a gait recognition method based on multi-level features of a convolutional neural network in Example 1 of the present invention;

[0029] Figure 2 This is a structural diagram of a convolutional neural network in a gait recognition method based on multi-level features of a convolutional neural network in Example 1 of the present invention. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] In order to solve the problems existing in the prior art, an embodiment of the present invention provides a gait recognition method based on multi-level features of a convolutional neural network.

[0032] Example 1

[0033] This embodiment discloses a gait recognition method based on multi-level features of convolutional neural network. Figure 1 ,include:

[0034] S100. Reading an input gait image sequence and preprocessing the image sequence;

[0035] In this embodiment, the specific method of S100 is: download the CASIA-B open source gait recognition data set, judge the gait sequence length in the data set, if the sequence length is greater than 150, take the middle 30 images, if the sequence length is less than 30, discard the sequence; the input gait sequence is cropped to 64*64 size according to the edge and center line principles in the height and width directions.

[0036] In some preferred embodiments, the specific method of S100 further includes: randomly performing morphological opening or closing operations on the image at a certain ratio, wherein the kernel size of the opening and closing operations is set to 3, and during training, the ratio is set to 0.2, and during inference, the ratio is set to 0; the number of input sequences each time is P*K, where P is the number of pedestrians in each batch during training or inference, and K is the number of gait sequences of each selected pedestrian under different viewing angles, clothing, backpacks, or walking conditions. As an embodiment, P can be set to 8, and K can be set to 16.

[0037] S200. Perform feature extraction of different levels on the image sequence through feature extraction branches of different convolution sizes; in this embodiment, Figure 2 The specific method of S200 includes: using three types of convolutions of 1x1, 3x3, and 5x5 to extract features of different granularities from the input image sequence respectively. The number of input channels of the three convolutions is 1, the number of output channels is 32, and the step size is 2. After the n image sequences with a length of s=30 and a length of 64*64 are extracted with different granularities, the feature map is n*30*32*32*32; then the three 32*32 feature maps are concatenated in the channel dimension through the Concat layer, and the output is n*30*96*32*32; the Block layer is composed of two convolution layers with a convolution size of 3 and a step size of 1 and a maximum pooling layer with a step size of 2. The number of input channels of the first convolution layer is 96, and the number of output channels is 64. The number of input channels of the second convolution layer is 64, and the number of output channels is 128. The feature map containing information of different levels extracted through the Block layer is output as n*30*128*16*16.

[0038] S300. The features extracted at different levels are fused in different ways to obtain the final features used for gait recognition and store them; specifically, in this embodiment, the Block_W layer, Block_H layer, HPM layer, VPM layer and Concat2 layer are used to fuse the multi-level features extracted by S200 in the w and h directions of the feature map; wherein, the Block_W layer and the Block_H layer are both composed of 2 layers of convolution with a convolution size of 3 and a stride of 1, the input and output channel numbers of the first layer of convolution are 128 and 256 respectively, and the input and output channel numbers of the second layer of convolution are 256 and 256 respectively.

[0039] The Block_W layer divides the features output by the feature extraction layer into n equal parts in the w direction, and uses the convolution operation to extract local features from the n strip feature maps segmented in the w direction, then concats these features together, and then uses the global maximum pooling to extract the strip local information in the w direction; the Block_H layer divides the features output by the feature extraction layer into n equal parts in the h direction, and uses the convolution operation to extract local features from the n strip feature maps segmented in the h direction, then concats these features together, and then uses the global maximum pooling to extract the strip local information in the h direction; the HPM layer uses an independent fully connected layer to map each h-direction pooled feature to the discriminant space; the VPM layer uses an independent fully connected layer to map each w-direction pooled feature to the discriminant space; the Concat2 layer contains a concat operation to splice the output feature maps of the HPM layer and the VPM layer together in the channel dimension to form a feature map containing local features in both w and h directions.

[0040] S400. According to the feature graph and the label, the loss is calculated using the loss function, and the convolutional neural network model is updated using the back propagation algorithm until the preset conditions are met to generate the final recognition model;

[0041] In this embodiment, Treplitloss and Centerloss are used for joint training, and the loss function is designed as follows:

[0042]

[0043]

[0044] LOSS=L tri +β·L Cen

[0045] Among them, the joint loss is composed of the ternary loss L tri and the center loss L Cen and the weight balance factor β of the two losses; the ternary loss L triIn the above equation, N is the number of samples in a training batch. is a sample randomly selected during training. The positive sample corresponding to the training sample is is the negative sample corresponding to the training sample, is the intra-class Euclidean distance between the randomly selected sample and similar samples, is the inter-class Euclidean distance between the randomly selected sample and samples of different classes, α is the minimum difference between the inter-class distance and the intra-class distance corresponding to the sample, and the center loss L Cen Medium i is a sample randomly selected during training, and c is the class center of all samples, which is obtained during the training process. The Adam algorithm is used for optimization during the training process, with the learning rate set to 0.0001 and the momentum to 0.9. If the loss of the training process is less than 0.0000001 or the number of iterations is greater than 100,000, the training is stopped.

[0046] S500. Input the gait target to be identified into the final recognition model, and match the model output result with the stored feature value, and take the result with the highest similarity to the stored feature value as the recognition result. When performing gait recognition, first establish a query set, that is, store the gait feature values ​​of each target; for a gait target to be identified, calculate its feature value and the Euclidean distance with the stored target one by one according to S100-S300, and take the result with the closest Euclidean distance to the stored target as the recognition result.

[0047] The present embodiment discloses a gait recognition method based on multi-level features of a convolutional neural network, which performs online data enhancement on an input gait profile map, increases the diversity of input data, and improves the robustness of the algorithm to pedestrian profiles in actual scenarios; uses convolutions with different receptive fields to extract gait information and expresses the gait with different local features in the height h and width w directions, and uses ternary loss and center loss simultaneously in the training process of the algorithm, which increases the distance between different classes and reduces the distance between the same class, thereby improving the accuracy of gait recognition.

[0048] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0049] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0050] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0051] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0052] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0053] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A gait recognition method based on multi-level features of convolutional neural network, characterized in that: include: S100. Reading an input gait image sequence and preprocessing the image sequence; S200. extracting features of different levels from the image sequence through feature extraction branches with different convolution sizes; S300. The features extracted at different levels are fused in different ways to obtain the final features for gait recognition and store them; the specific method of S300 includes: using Block_W layer, Block_H layer, HPM layer, VPM layer and Concat2 layer to fuse the multi-level features extracted by S200 in the w and h directions of the feature map; wherein, the Block_W layer and the Block_H layer are composed of 2 layers of convolution with a convolution size of 3 and a step size of 1, the input and output channels of the first layer of convolution are 128 and 256 respectively, and the input and output channels of the second layer of convolution are 256 and 256 respectively; the Block_W layer divides the features output by the feature extraction layer into n equal parts in the w direction, and uses the convolution operation to extract local features from the n strip feature maps divided in the w direction, and then combines these The features are concat-aggregated, and then the global maximum pooling is used to extract the strip local information in the w direction; the Block_H layer divides the features output by the feature extraction layer into n equal parts in the h direction, and uses the convolution operation to extract local features from the n h-direction segmented strip feature maps, and then concat-aggregates these features, and then uses the global maximum pooling to extract the strip local information in the h direction; the HPM layer uses an independent fully connected layer to map the features after each h-direction pooling to the discriminant space; the VPM layer uses an independent fully connected layer to map the features after each w-direction pooling to the discriminant space; the Concat2 layer contains a concat operation to splice the output feature maps of the HPM layer and the VPM layer together in the channel dimension to form a feature map containing both w and h-direction local features; S400. According to the feature graph and label, the loss is calculated using the loss function, and the convolutional neural network model is updated using the back propagation algorithm until the preset conditions are met to generate the final recognition model; in S400, Treplitloss and Centerloss are used for joint training, and the loss function is designed as follows: ； ； ； Among them, the joint loss is composed of three-dimensional loss and center loss and the weight balancing factor of the two losses Together they constitute; three-way loss In the above equation, N is the number of samples in a training batch. is a sample randomly selected during training. is the positive sample corresponding to the training sample, is the negative sample corresponding to the training sample, is the intra-class Euclidean distance between the randomly selected sample and similar samples, is the inter-class Euclidean distance between the randomly selected sample and samples of different classes, is the minimum difference between the inter-class distance and the intra-class distance corresponding to the sample, and the center loss middle is a sample randomly selected during training, and c is the class center of all samples, obtained during the training process; S500. Input the gait target to be identified into the final recognition model, and match the model output result with the stored feature values, and take the result with the highest similarity to the stored feature values ​​as the recognition result.

2. A gait recognition method based on multi-level features of convolutional neural network as claimed in claim 1, characterized in that: The specific method of S100 is as follows: download the CASIA-B open source gait recognition dataset, judge the length of the gait sequence in the dataset, and extract the middle 30 images if the sequence length is greater than 150; If the sequence length is less than 30, the sequence is discarded; the input gait sequence is cut into 64 according to the edge and center line principle in the height and width directions. 64 size.

3. A gait recognition method based on multi-level features of convolutional neural network as claimed in claim 1, characterized in that: The specific method of S100 also includes: randomly performing morphological opening or closing operations on the image according to a certain ratio, wherein the kernel size of the opening and closing operations is set to 3, and during training, the ratio is set to 0.2, and during inference, the ratio is set to 0; the number of input sequences each time is P K, where P is the number of pedestrians in each batch during training or inference, and K is the number of gait sequences of each selected pedestrian under different viewing angles, clothing, backpacks, or walking conditions.

4. A gait recognition method based on multi-level features of convolutional neural network as claimed in claim 1, characterized in that: The specific method of S200 includes: using 1x1, 3x3, and 5x5 convolutions to extract features of different granularities from the input image sequence. The number of input channels of the three convolutions is 1, the number of output channels is 32, the step size is 2, and the input length and width are 64. 64-sized n image sequences with length s=30 are extracted with different granularity features; then the three 32 The feature maps of size 32 are concatenated in the channel dimension, and finally feature maps containing information at different levels are obtained.

5. A gait recognition method based on multi-level features of convolutional neural network as claimed in claim 4, characterized in that: The Block layer consists of two convolution layers with a convolution size of 3 and a stride of 1, and a maximum pooling layer with a stride of 2. The first convolution layer has 96 input channels and 64 output channels. The second convolution layer has 64 input channels and 128 output channels. The feature maps containing different levels of information extracted by the Block layer are output as n 30 128 16 16.

6. A gait recognition method based on multi-level features of convolutional neural network as claimed in claim 1, characterized in that: The training process is optimized using the Adam algorithm, with the learning rate set to 0.0001 and the momentum to 0.

9. If the loss of the training process is less than 0.0000001 or the number of iterations is greater than 100,000, the training is stopped.

Images