Gait recognition method, device, and intelligent terminal based on improved convolutional neural network

Abstract

The present invention discloses a gait recognition method, device, and intelligent terminal based on an improved convolutional neural network. The method comprises: sequentially connecting a convolution layer, a batch normalization layer, and an activation layer with an activation function to form an improved basic convolution unit, and introducing sequence attention to form a basic sequence attention convolution module; selecting a plurality of basic sequence attention convolution modules according to the size of a video image, sequentially connecting them, and connecting them with an average pooling layer to build a gait recognition convolutional neural network; using the built gait recognition convolutional neural network to train a complete gait recognition model; inputting the video data to be recognized into the trained gait recognition model for gait recognition. The gait recognition convolutional neural network of the present invention integrates temporal features at each layer, and the network model has a strong ability to suppress noise. At the same time, the features of the previous and next sequence graphs can be used to restore missing information, so that the network model has a better effect and the average accuracy rate is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of identity verification and recognition technology, and in particular to a gait recognition method, device, intelligent terminal and storage medium based on an improved convolutional neural network. Background Art

[0002] Biometric authentication technology has been a hot research topic in the field of computer vision in recent years. This technology uses facial images, fingerprints, voiceprints, gait, and other biometric information to identify or verify identities, and has broad application prospects in security and access management. Compared to other biometric information, gait-based authentication technology differs significantly in its data collection and processing methods. During authentication, a camera typically captures a video of the verifier walking from a distance, then analyzes their gait characteristics to identify the person being authenticated. In terms of data collection, gait-based authentication technology avoids close contact with the verifier, thus preventing them from being disturbed and being less noticeable, allowing authentication to be performed without the person actively participating. In terms of data processing, gait features are extracted from video data and include both appearance and temporal information. The inclusion of temporal information facilitates the extraction of gait features even with low camera resolution and makes them more difficult to imitate. Due to these characteristics, gait-based authentication technology has found practical application in criminal investigations.

[0003] Existing convolutional neural network-based feature extraction methods use a consistent computational process for features at every location in GEI images. However, due to significant variations in the size, shape, and trajectory of various body parts during walking, their importance to gait identification varies. Ignoring this spatial variability reduces the quality of feature extraction.

[0004] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a gait recognition method, device, intelligent terminal and storage medium based on an improved convolutional neural network to address the above-mentioned defects of the prior art. The present invention can improve the quality of feature extraction and the accuracy of extraction.

[0006] In order to solve the above technical problems, the first aspect of the present invention provides a gait recognition method based on an improved convolutional neural network, the method comprising:

[0007] The convolution layer, batch normalization layer, and activation layer with activation function are connected in sequence to form an improved basic convolution unit. Sequential attention is introduced into the improved basic convolution unit to form a basic sequence attention convolution module.

[0008] According to the size of the video image, multiple basic sequence attention convolution modules are selected and connected in sequence. After the multiple basic sequence attention convolution modules are connected in sequence, an average pooling layer is connected to build a gait recognition convolutional neural network.

[0009] Use the built gait recognition convolutional neural network to train a complete gait recognition model;

[0010] The video data to be recognized is input into the trained gait recognition model for gait recognition, and the recognition results are output.

[0011] The gait recognition method based on the improved convolutional neural network, wherein the steps of sequentially connecting the convolution layer, the batch normalization layer, and the activation layer with the activation function to form an improved basic convolution unit, and introducing sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module include:

[0012] The convolution layer is used to extract features. During the convolution operation, the neurons in the current layer are connected to the local neurons in the feature map of the previous layer only through the convolution kernel.

[0013] The activation layer is an activation layer using the ReLU activation function; the activation layer is used to increase the nonlinearity of the neural network.

[0014] In the gait recognition method based on the improved convolutional neural network, the step of introducing sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module includes:

[0015] The input feature map is subjected to three basic convolution operations through the basic convolution unit to obtain the query feature map, key feature map, and value feature map for spatial feature extraction;

[0016] Multiply the query feature map with the key feature map to obtain the sequence heat map;

[0017] Map the sequence heat map to between 0 and 1 through the Softmax function;

[0018] Multiply the sequence heat map after Softmax with the value feature map, weight the features at each moment, and obtain the output feature map;

[0019] Adding the output feature map to the input feature map forms the building steps of the basic sequence attention convolution module.

[0020] The gait recognition method based on the improved convolutional neural network, wherein the steps of selecting multiple basic sequence attention convolution modules according to the size of the video image and connecting them in sequence, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network include:

[0021] According to the size of the video image, multiple basic sequence attention convolution modules are selected and connected in sequence. Each connected basic sequence attention convolution module gradually reduces the height and width of the feature map and gradually increases the number of channels to compress and extract the features step by step.

[0022] An average pooling layer is connected after the multiple basic sequence attention convolution modules are connected in sequence, and a gait feature vector with an output length of 256 is obtained through the average pooling layer;

[0023] The average pooling layer is sequentially connected to the fully connected layer and the Softmax layer to build a gait recognition convolutional neural network.

[0024] The gait recognition method based on the improved convolutional neural network, wherein the step of selecting multiple basic sequence attention convolution modules according to the size of the video image and connecting them in sequence, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network also includes:

[0025] During the testing phase, the k-nearest neighbor algorithm is used to calculate the category to which the gait features obtained by the gait recognition convolutional neural network belong.

[0026] The gait recognition method based on the improved convolutional neural network, wherein the step of using the constructed gait recognition convolutional neural network to train a complete gait recognition model includes:

[0027] The first 74 people in the CASIA standard gait recognition dataset were selected as training samples;

[0028] The gait graph is resized to 64x64 and used as the input of the gait recognition convolutional neural network, and the category of the gait sequence is the target output;

[0029] The basic settings for training the gait recognition convolutional neural network are as follows: Adam optimizer, cross entropy loss function, learning rate of 4e-4, weight decay of 1e-5, training sequence length of 32, and batch size of 32.

[0030] The model training of the set gait recognition convolutional neural network is started, and a total of 500 generations are trained to obtain the final network model, which is the trained gait recognition model.

[0031] In the gait recognition method based on the improved convolutional neural network, the step of inputting the video data to be recognized into the trained gait recognition model for gait recognition and outputting the recognition result includes:

[0032] Testing steps, the testing steps comprising:

[0033] Half of the samples in the CASIA standard gait recognition dataset of 74-124 people in normal state were selected as benchmark samples, and the other half in normal state, wrapped state, and coat state were selected as test samples;

[0034] Unify the size of the gait graph to 64x64;

[0035] The gait sequence is input into the trained gait recognition model to obtain a feature vector with an output length of 256;

[0036] The k-nearest neighbor algorithm is used to calculate the category to which the test sample belongs and the accuracy rate.

[0037] A gait recognition device based on an improved convolutional neural network, wherein the device comprises:

[0038] The basic sequence attention convolution module building module is used to connect the convolution layer, batch normalization layer, and activation layer with activation function in sequence to form an improved basic convolution unit, and introduce sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module;

[0039] A gait recognition convolutional neural network building module is used to select multiple basic sequence attention convolution modules according to the size of the video image and connect them in sequence. After the multiple basic sequence attention convolution modules are connected in sequence, an average pooling layer is connected to build a gait recognition convolutional neural network;

[0040] The training module is used to train a complete gait recognition model using the built gait recognition convolutional neural network;

[0041] The recognition module is used to input the video data to be recognized into the trained gait recognition model for gait recognition and output the recognition results.

[0042] An intelligent terminal includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including the method for executing any one of the methods described above.

[0043] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any one of the methods described above.

[0044] Beneficial effects: Compared with the existing technology, the present invention provides a gait recognition method based on an improved convolutional neural network. The present invention inputs the video data to be recognized into a trained gait recognition model for gait recognition and outputs the recognition result. The test results show that the average accuracy is improved. The accuracy under normal conditions is mainly affected by image noise or missing. Since the gait recognition convolutional neural network improved by the present invention integrates the time features at each layer, the network model has a strong ability to suppress noise. At the same time, the features of the previous and next sequence graphs can be used to restore the missing, so that the network model has a better effect and the average accuracy is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a specific implementation of a gait recognition method based on an improved convolutional neural network provided in an embodiment of the present invention.

[0046] Figure 2 Schematic diagram of the basic convolution unit construction of the gait recognition method based on the improved convolutional neural network provided in an embodiment of the present invention.

[0047] Figure 3 Schematic diagram of the construction of the basic sequence attention convolution module of the gait recognition method based on the improved convolutional neural network provided in an embodiment of the present invention.

[0048] Figure 4 Schematic diagram of building a gait recognition convolutional neural network for the gait recognition method based on an improved convolutional neural network provided in an embodiment of the present invention.

[0049] Figure 5 This is a principle block diagram of a gait recognition device based on an improved convolutional neural network provided by an embodiment of the present invention.

[0050] Figure 6 This is a block diagram of the internal structure of the smart terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] Biometric authentication technology has been a hot research topic in the field of computer vision in recent years. This technology uses facial images, fingerprints, voiceprints, gait, and other biometric information to identify or verify identities, and has broad application prospects in security and access management. Compared to other biometric information, gait-based authentication technology differs significantly in its data collection and processing methods. During authentication, a camera typically captures a video of the verifier walking from a distance, then analyzes their gait characteristics to identify the person being authenticated. In terms of data collection, gait-based authentication technology avoids close contact with the verifier, thus preventing them from being disturbed and being less noticeable, allowing authentication to be performed without the person actively participating. In terms of data processing, gait features are extracted from video data and include both appearance and temporal information. The inclusion of temporal information facilitates the extraction of gait features even with low camera resolution and makes them more difficult to imitate. Due to these characteristics, gait-based authentication technology has found practical application in criminal investigations. However, compared with biometric identification technologies such as face recognition and fingerprint recognition, gait identification technology still has a lot of room for improvement. How to solve the problem of gait identification under different perspectives, different clothing and holding objects still requires further research.

[0053] Video-based gait identification technology can generally be divided into two categories: model-based methods and image-free methods. Model-based gait identification Model-based methods generally construct a human body model by extracting skeletal points or dividing the regions of each limb, and then extract gait information from the time series of the human body model. Image-based methods do not require a complex modeling process. They generally use contour images to calculate feature maps such as GEI (Gait Energy Image) images, GFI (Gait Flow Image) images, and MGEI (Masked GEI based on GEnI) images, and directly extract gait features from the feature maps.

[0054] In recent years, due to the tremendous success of deep learning in computer vision, several algorithms using neural networks for gait identification have been proposed. Some use generative adversarial networks (GANs), resulting in trained network models that can remove knapsacks from GEI images and transform GEI images from different perspectives to a unified perspective, thereby eliminating the influence of both the knapsack and the perspective. Following the same principle, some have employed auto-encoder models to perform perspective conversion and knapsack removal, further improving recognition performance. With the rapid development of convolutional neural networks (CNNs) in image recognition, face recognition, and action classification, many researchers have also attempted to use CNNs to further extract and match gait features, achieving promising results. Some have used dual-input CNNs to directly calculate the similarity between two GEI images, integrating feature extraction and matching into a single network. However, the dual-input network parameters are not shared, resulting in high model complexity. Furthermore, gait identification verification requires pairing the gait data to be tested with the data in the database to calculate similarity, which is computationally expensive and slows verification. Others have used twin networks with shared parameters instead of dual-input networks to reduce model complexity. For classification and recognition, Alotaibi et al. further extracted GEI features and trained a classifier using a single-input convolutional neural network, avoiding the need to calculate the similarities of all possible GEI image pairs and significantly improving computational efficiency.

[0055] Due to the weight-sharing strategy of convolution operations, the aforementioned convolutional neural network-based method uses a consistent computational process for feature extraction at all locations in the GEI image. However, as people walk, body parts vary significantly in size, shape, and trajectory, and their importance to gait identification varies. Ignoring this spatial variability will reduce the quality of feature extraction.

[0056] In order to solve the problems of the prior art, this embodiment provides a gait recognition method based on an improved convolutional neural network. The present invention addresses the recognition problem faced by the gait recognition algorithm under conditions where the status of pedestrians such as backpacks and coats changes. First, a sequence attention convolution module is improved. This structure can more comprehensively integrate the features of all video sequences. Then, this module is used to build an improved convolutional neural network and train a complete gait recognition model. After testing with the CASIA standard gait dataset, the recognition accuracy has been greatly improved, providing convenience for users.

[0057] Exemplary Methods

[0058] The gait recognition method based on the improved convolutional neural network of this embodiment can be applied to smart terminals. When it is implemented, Figure 1 As shown in , the gait recognition method based on the improved convolutional neural network specifically includes the following steps:

[0059] Step S100: Connect the convolution layer, batch normalization layer, and activation layer with activation function in sequence to form an improved basic convolution unit, introduce sequence attention into the improved basic convolution unit, and form a basic sequence attention convolution module;

[0060] Step S200: Select multiple basic sequence attention convolution modules according to the size of the video image and connect them in sequence. After the multiple basic sequence attention convolution modules are connected in sequence, an average pooling layer is connected to build a gait recognition convolutional neural network.

[0061] Step S300: using the constructed gait recognition convolutional neural network to train a complete gait recognition model;

[0062] Step S400: Input the video data to be recognized into the trained gait recognition model to perform gait recognition, and output the recognition result.

[0063] In this embodiment, the existing basic convolution unit is improved, and sequence attention is introduced into the improved basic convolution unit to form a basic sequence attention convolution module; then, multiple basic sequence attention convolution modules are selected according to the size of the video image and connected in sequence, and an average pooling layer is connected after the multiple basic sequence attention convolution modules are connected in sequence to build a gait recognition convolutional neural network; the constructed gait recognition convolutional neural network is used to train a complete gait recognition model, and gait recognition is performed based on the trained gait recognition model; after testing with the CASIA standard gait dataset, the recognition accuracy is greatly improved.

[0064] In this embodiment of the present invention, regarding step S100: the convolution layer, the batch normalization layer, and the activation layer with the activation function are sequentially connected to form an improved basic convolution unit, and the sequence attention is introduced into the improved basic convolution unit to form a basic sequence attention convolution module. Specifically:

[0065] like Figure 2 As shown in FIG, the improved basic convolution unit of the embodiment of the present invention is composed of a convolution layer, a batch normalization layer, and an activation layer with an activation function connected in sequence. The improved basic convolution unit of the embodiment of the present invention is mainly used for feature extraction.

[0066] Among them, regarding the convolution layer (Conv): The convolution layer of the embodiment of the present invention is the basic module of the convolutional neural network (CNN) and is used for feature extraction. When performing a convolution operation, the neurons in the current layer are only connected to the local neurons in the feature map of the previous layer through the convolution kernel. In the embodiment of the present invention, preferably, the convolution kernel size is 3x3, the padding is 1, the step size of the last layer is 1, and the rest are 2.

[0067] The batch normalization layer of the basic convolutional unit in this embodiment normalizes the features extracted by the convolutional layer. Using batch normalization, the layer is an effective neural network optimization technique. Batch normalization not only solves the "vanishing gradient" problem in deep learning but also stabilizes training and accelerates network convergence.

[0068] Regarding the activation layer of the basic convolutional unit in this embodiment, the use of the activation layer in this embodiment is to increase the nonlinearity of the neural network and enhance the expressive power of the model. The activation function used in this invention is ReLU. In this embodiment of the present invention, the use of an activation layer with an activation function facilitates the construction of a gait recognition convolutional neural network based on a multi-layer basic sequence attention convolution module, allowing for deeper feature extraction.

[0069] In the embodiment of the present invention, the basic sequence attention convolution module is constructed, such as Figure 3 As shown in FIG, a basic sequence attention convolution module is formed according to an embodiment of the present invention; specifically, it is implemented by the following steps:

[0070] First, the input feature map is subjected to three basic convolution operations through the basic convolution unit to obtain the query feature map, key feature map, and value feature map. The basic convolution unit has a strong spatial feature extraction capability, so it is used for spatial feature extraction.

[0071] Then, the query feature map is multiplied by the key feature map to obtain the sequence heat map;

[0072] Secondly, the sequence heat map is mapped to the range of 0-1 through the Softmax function to indicate the contribution ratio of the feature map at different moments to the output feature map;

[0073] Then, the sequence heat map after Softmax is multiplied with the value feature map, the features at each moment are weighted, and the output feature map is obtained to achieve the purpose of time feature extraction.

[0074] Finally, the output feature map is added to the input feature map to form the construction steps of the basic sequence attention convolution module, so as to further improve the performance of the module.

[0075] In an embodiment of the present invention, the step S200 of selecting multiple basic sequence attention convolution modules according to the size of the video image and sequentially connecting them, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network specifically includes:

[0076] In the present invention, the Figure 3 The basic sequence attention convolution module shown in the figure is then used to build a gait recognition convolutional neural network. The present invention is specifically described as follows about building a gait recognition convolutional neural network model:

[0077] like Figure 4 As shown, the present invention builds a convolutional neural network for gait recognition. It selects a number of pre-built basic sequence attention convolution modules according to the size of the video image and connects them in sequence. It achieves a step-by-step input of 64*64 and output of 1*1. Specifically:

[0078] In step S201, the basic sequence attention convolution module constructed in the above steps is selected, and multiple basic sequence attention convolution modules are connected in sequence according to the size of the video image. Each basic sequence attention convolution module connected in sequence gradually reduces the height and width of the feature map, and gradually increases the number of channels, compressing and extracting the features step by step. Each layer of basic attention convolution module integrates the features of space and time, and the features are expressed more fully.

[0079] like Figure 4 In the embodiment shown, the seven basic sequence attention convolution modules constructed in the above steps are connected in sequence, and the input gait picture, the number of channels is 1, and the seven basic sequence attention convolution modules connected in sequence are: the first basic sequence attention convolution module, the number of channels is 32, the second basic sequence attention convolution module, the number of channels is 64, the third basic sequence attention convolution module, the number of channels is 128, the fourth basic sequence attention convolution module, the number of channels is 256, the fifth basic sequence attention convolution module, the number of channels is 256, the sixth basic sequence attention convolution module, the number of channels is 256, and the seventh basic sequence attention convolution module, the number of channels is 256; the features are compressed and extracted step by step, and each layer of the basic attention convolution module integrates the features of space and time, and the features are expressed more fully.

[0080] Step S202: After the multiple basic sequence attention convolution modules are connected in sequence, an average pooling layer is connected to form an initialization, and a gait feature vector with an output length of 256 is obtained through the average pooling layer.

[0081] Step S203: Then, the average pooling layer is sequentially connected to the fully connected layer and the Softmax layer to build a gait recognition convolutional neural network for easy training.

[0082] Step S04, testing phase, uses the k-nearest neighbor algorithm to calculate the category to which the gait features obtained by the gait recognition convolutional neural network belong.

[0083] For example, the category to which the gait features obtained by the gait recognition convolutional neural network are calculated using the k-nearest neighbor algorithm.

[0084] like Figure 4 As shown, in the embodiment of the present invention, the step S300 of using the constructed gait recognition convolutional neural network to train a complete gait recognition model specifically includes:

[0085] Step S301: First, select the first 74 people in the CASIA standard gait recognition data set as training samples.

[0086] Step S302: Then, the gait graph is resized to 64x64 and used as the input of the gait recognition convolutional neural network, and the category to which the gait sequence belongs is the target output.

[0087] Step S303: Then, basic settings are made for the training of the gait recognition convolutional neural network, including the optimizer being Adam, the loss function being the cross entropy loss function, the learning rate being 4e-4, the weight decay being 1e-5, the training sequence length being 32, and the batch size being 32.

[0088] Step S304: Finally, the model training of the set gait recognition convolutional neural network is started, and a total of 500 generations are trained to obtain the final network model, which is the trained gait recognition model.

[0089] In a further embodiment of the present invention, regarding step S400 , the video data to be recognized is input into a trained gait recognition model for gait recognition, and a recognition result is output.

[0090] That is, when the present invention is applied, the video data to be recognized is input into the trained gait recognition model for gait recognition, and the recognition result is output. Figure 4As shown, for example, the video data to be recognized can be first obtained and organized into gait sequences. All gait graphs in the gait sequence are then resized to 64x64. The gait sequence is then fed into a trained gait recognition model. The model's basic sequence attention convolution module performs three basic convolution operations on the input gait graph, generating a query feature graph, a key feature graph, and a value feature graph. The basic convolution unit has strong spatial feature extraction capabilities and is therefore used for spatial feature extraction. The query feature graph is then multiplied with the key feature graph to generate a sequence heatmap. Next, the sequence heatmap is mapped to a value between 0 and 1 using a Softmax function to represent the contribution of feature graphs at different moments to the output feature graph. The Softmaxed sequence heatmap is then multiplied with the value feature graph to weight the features at each moment, achieving temporal feature extraction. This is then fed into the next basic sequence attention convolution module for further feature extraction. The present invention gradually reduces the height and width of the feature map through multiple layers of basic sequence attention convolution modules, gradually increases the number of channels, and compresses and extracts the features step by step. Each layer of basic attention convolution modules integrates the features of space and time, and the expression of features is more sufficient. Then, through the average pooling layer, a gait feature vector with an output length of 256 is obtained. The gait recognition convolutional neural network of the present invention integrates the temporal features at each layer. The network model has a strong ability to suppress noise. At the same time, the features of the previous and next sequence maps can be used to restore the missing, so that the network model is more effective and the average accuracy is greatly improved.

[0091] Furthermore, in order to test the application effect of the trained gait recognition model of the present invention, in specific applications, the present invention performs a model test on the trained gait recognition model to test the effect of the trained gait recognition model of the present invention on gait recognition of video data. The details are as follows:

[0092] Half of the samples in the CASIA standard gait recognition dataset of 74-124 people in normal state were selected as benchmark samples, and the other half in normal state, wrapped state, and coat state were selected as test samples;

[0093] Then, the gait graph is resized to 64x64;

[0094] Then, the gait sequence is input into the trained gait recognition model to obtain a feature vector with an output length of 256, such as Figure 4 shown.

[0095] Then, the k-nearest neighbor algorithm is used to calculate the category to which the test sample belongs, and the accuracy is calculated. The accuracy calculated by the embodiment of the present invention is shown in the following table:

[0096]

[0097] As shown in the table above, the present invention inputs the video data to be recognized into a trained gait recognition model for gait recognition and outputs the recognition results. Test results show that the accuracy in normal conditions reaches 100% except for the 180° viewing angle, and the average accuracy is improved by 5%. The accuracy in normal conditions is mainly affected by image noise or missing information. Because the improved gait recognition convolutional neural network of the present invention integrates temporal features at each layer, the network model has a strong ability to suppress noise. At the same time, the features of the previous and next sequence images can be used to recover missing information, making the network model more effective and significantly improving the average accuracy.

[0098] In the backpack state, the gait recognition convolutional neural network of the present invention outperforms other algorithms, with an average accuracy increase of 4.9%. The accuracy in the backpack state is mainly affected by the image differences caused by the backpack and the occlusion of leg information by the backpack. The gait recognition convolutional neural network model of the present invention filters the influence of the backpack state on the classification results through multi-layer convolution connections. At the same time, the enhancement of sequential attention can restore the partial occlusion of leg information by the backpack. Therefore, the performance of the gait recognition convolutional neural network model of the present invention is greatly improved; in the coat state; although it is lower than other algorithms at angles of 18, 36, and 162, the average accuracy is improved by 4.2%.

[0099] The accuracy in the coat state is mainly affected by the occlusion of leg information by the coat. The sequence attention convolution module used in this invention integrates gait information at all times, has a certain feature recovery ability, and the accuracy is further improved.

[0100] As can be seen from the above, the gait recognition method based on the improved convolutional neural network provided by the embodiment of the present invention has the following advantages:

[0101] The improved sequence attention convolution module used in the present invention has the ability to combine the temporal and spatial features of the gait sequence. Unlike some existing technologies that extract spatial features and temporal features separately, the feature fusion of the present invention is more complete.

[0102] The gait recognition convolutional neural network constructed by the present invention performs sequential attention enhancement on the features of each layer. Unlike the existing technology that only uses the deeper layers of the convolutional neural network, the feature extraction of the present invention is more comprehensive and more robust.

[0103] Exemplary devices

[0104] like Figure 5 As shown in , an embodiment of the present invention provides a gait recognition device based on an improved convolutional neural network, the device comprising:

[0105] A basic sequence attention convolution module building module 610 is used to sequentially connect a convolution layer, a batch normalization layer, and an activation layer with an activation function to form an improved basic convolution unit, and introduce sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module;

[0106] A gait recognition convolutional neural network building module 620 is used to select multiple basic sequence attention convolution modules according to the size of the video image and connect them in sequence, and then connect an average pooling layer after the multiple basic sequence attention convolution modules to build a gait recognition convolutional neural network.

[0107] A training module 630 is used to train a complete gait recognition model using the constructed gait recognition convolutional neural network;

[0108] The recognition module 640 is used to input the video data to be recognized into the trained gait recognition model to perform gait recognition and output the recognition result, as described above.

[0109] Based on the above embodiment, the present invention also provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 6 As shown. The intelligent terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a gait recognition method based on an improved convolutional neural network is implemented. The display screen of the intelligent terminal can be a liquid crystal display or an electronic ink display.

[0110] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention and does not constitute a limitation on the smart terminal to which the solution of the present invention is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0111] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0112] The convolution layer, batch normalization layer, and activation layer with activation function are connected in sequence to form an improved basic convolution unit. Sequential attention is introduced into the improved basic convolution unit to form a basic sequence attention convolution module.

[0113] According to the size of the video image, multiple basic sequence attention convolution modules are selected and connected in sequence. After the multiple basic sequence attention convolution modules are connected in sequence, an average pooling layer is connected to build a gait recognition convolutional neural network.

[0114] Use the built gait recognition convolutional neural network to train a complete gait recognition model;

[0115] The video data to be recognized is input into the trained gait recognition model for gait recognition, and the recognition results are output.

[0116] The steps of sequentially connecting the convolution layer, the batch normalization layer, and the activation layer with the activation function to form an improved basic convolution unit, and introducing sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module include:

[0117] The convolution layer is used to extract features. During the convolution operation, the neurons in the current layer are connected to the local neurons in the feature map of the previous layer only through the convolution kernel.

[0118] The activation layer is an activation layer using the ReLU activation function; the activation layer is used to increase the nonlinearity of the neural network.

[0119] The step of introducing sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module includes:

[0120] The input feature map is subjected to three basic convolution operations through the basic convolution unit to obtain the query feature map, key feature map, and value feature map for spatial feature extraction;

[0121] Multiply the query feature map with the key feature map to obtain the sequence heat map;

[0122] Map the sequence heat map to between 0 and 1 through the Softmax function;

[0123] Multiply the sequence heat map after Softmax with the value feature map, weight the features at each moment, and obtain the output feature map;

[0124] Adding the output feature map to the input feature map forms the building steps of the basic sequence attention convolution module.

[0125] The steps of selecting multiple basic sequence attention convolution modules according to the size of the video image and sequentially connecting them, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network include:

[0126] According to the size of the video image, multiple basic sequence attention convolution modules are selected and connected in sequence. Each connected basic sequence attention convolution module gradually reduces the height and width of the feature map and gradually increases the number of channels to compress and extract the features step by step.

[0127] An average pooling layer is connected after the multiple basic sequence attention convolution modules are connected in sequence, and a gait feature vector with an output length of 256 is obtained through the average pooling layer;

[0128] The average pooling layer is sequentially connected to the fully connected layer and the Softmax layer to build a gait recognition convolutional neural network.

[0129] The step of selecting multiple basic sequence attention convolution modules according to the size of the video image and connecting them in sequence, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network also includes:

[0130] During the testing phase, the k-nearest neighbor algorithm is used to calculate the category to which the gait features obtained by the gait recognition convolutional neural network belong.

[0131] The step of using the constructed gait recognition convolutional neural network to train a complete gait recognition model includes:

[0132] The first 74 people in the CASIA standard gait recognition dataset were selected as training samples;

[0133] The gait graph is resized to 64x64 and used as the input of the gait recognition convolutional neural network, and the category of the gait sequence is the target output;

[0134] The basic settings for training the gait recognition convolutional neural network are as follows: Adam optimizer, cross entropy loss function, learning rate of 4e-4, weight decay of 1e-5, training sequence length of 32, and batch size of 32.

[0135] The model training of the set gait recognition convolutional neural network is started, and a total of 500 generations are trained to obtain the final network model, which is the trained gait recognition model.

[0136] The step of inputting the video data to be recognized into the trained gait recognition model for gait recognition and outputting the recognition result includes:

[0137] Testing steps, the testing steps comprising:

[0138] Half of the samples in the CASIA standard gait recognition dataset of 74-124 people in normal state were selected as benchmark samples, and the other half in normal state, wrapped state, and coat state were selected as test samples;

[0139] Unify the size of the gait graph to 64x64;

[0140] The gait sequence is input into the trained gait recognition model to obtain a feature vector with an output length of 256;

[0141] The k-nearest neighbor algorithm is used to calculate the category to which the test sample belongs and the accuracy rate.

[0142] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0143] In summary, the present invention discloses a gait recognition method, device, intelligent terminal and storage medium based on an improved convolutional neural network. The method includes: connecting a convolution layer, a batch normalization layer, and an activation layer with an activation function in sequence to form an improved basic convolution unit, and introducing sequence attention to form a basic sequence attention convolution module; selecting multiple basic sequence attention convolution modules according to the size of the video image and connecting them in sequence, and connecting them with an average pooling layer to build a gait recognition convolutional neural network; using the built gait recognition convolutional neural network to train a complete gait recognition model; inputting the video data to be recognized into the trained gait recognition model for gait recognition. The gait recognition convolutional neural network of the present invention integrates the temporal features at each layer, and the network model has a strong ability to suppress noise. At the same time, the features of the previous and next sequence graphs can be used to restore the missing information, so that the network model has a better effect and the average accuracy is greatly improved.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A gait recognition method based on an improved convolutional neural network, characterized in that: The method comprises: The convolution layer, batch normalization layer, and activation layer with activation function are connected in sequence to form an improved basic convolution unit. Sequential attention is introduced into the improved basic convolution unit to form a basic sequence attention convolution module. According to the size of the video image, multiple basic sequence attention convolution modules are selected and connected in sequence. After the multiple basic sequence attention convolution modules are connected in sequence, an average pooling layer is connected to build a gait recognition convolutional neural network. Use the built gait recognition convolutional neural network to train a complete gait recognition model; Input the video data to be recognized into the trained gait recognition model for gait recognition and output the recognition results; The gait recognition convolutional neural network integrates temporal features at each layer and uses the features of the previous and next sequence graphs to recover the missing features. The steps of selecting multiple basic sequence attention convolution modules according to the size of the video image and sequentially connecting them, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network include: According to the size of the video image, multiple basic sequence attention convolution modules are selected and connected in sequence. Each connected basic sequence attention convolution module gradually reduces the height and width of the feature map and gradually increases the number of channels to compress and extract the features step by step. An average pooling layer is connected after the multiple basic sequence attention convolution modules are connected in sequence, and a gait feature vector with an output length of 256 is obtained through the average pooling layer; The average pooling layer is sequentially connected to the fully connected layer and the Softmax layer to build a gait recognition convolutional neural network; The step of inputting the video data to be recognized into the trained gait recognition model for gait recognition and outputting the recognition result includes: Testing steps, the testing steps comprising: Half of the samples in the CASIA standard gait recognition dataset of 74-124 people in normal state were selected as benchmark samples, and the other half in normal state, wrapped state, and coat state were selected as test samples; Unify the size of the gait graph to 64x64; The gait sequence is input into the trained gait recognition model to obtain a feature vector with an output length of 256; Use the k-nearest neighbor algorithm to calculate the category to which the test sample belongs and calculate the accuracy; In the backpack state, the gait recognition convolutional neural network model filters the influence of the backpack state on the classification results through multi-layer convolution connections. At the same time, the enhancement of sequential attention can restore the partial occlusion of the leg information by the backpack. In the coat state, the sequence attention convolution module used integrates the gait information at all times and has a certain feature recovery ability.

2. The gait recognition method based on the improved convolutional neural network according to claim 1, characterized in that: The steps of sequentially connecting the convolution layer, the batch normalization layer, and the activation layer with the activation function to form an improved basic convolution unit, and introducing sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module include: The convolution layer is used to extract features. During the convolution operation, the neurons in the current layer are connected to the local neurons in the feature map of the previous layer only through the convolution kernel. The activation layer is an activation layer using the ReLU activation function; the activation layer is used to increase the nonlinearity of the neural network.

3. The gait recognition method based on the improved convolutional neural network according to claim 1, characterized in that: The step of introducing sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module includes: The input feature map is subjected to three basic convolution operations through the basic convolution unit to obtain the query feature map, key feature map, and value feature map for spatial feature extraction; Multiply the query feature map with the key feature map to obtain the sequence heat map; Map the sequence heat map to between 0 and 1 through the Softmax function; Multiply the sequence heat map after Softmax with the value feature map, weight the features at each moment, and obtain the output feature map; Adding the output feature map to the input feature map forms the building steps of the basic sequence attention convolution module.

4. The gait recognition method based on the improved convolutional neural network according to claim 1, characterized in that: The step of selecting multiple basic sequence attention convolution modules according to the size of the video image and connecting them in sequence, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network also includes: During the testing phase, the k-nearest neighbor algorithm is used to calculate the category to which the gait features obtained by the gait recognition convolutional neural network belong.

5. The gait recognition method based on the improved convolutional neural network according to claim 1, characterized in that: The steps of using the constructed gait recognition convolutional neural network to train a complete gait recognition model include: The first 74 people in the CASIA standard gait recognition dataset were selected as training samples; The gait graph is resized to 64x64 and used as the input of the gait recognition convolutional neural network, and the category of the gait sequence is the target output; The basic settings for training the gait recognition convolutional neural network are as follows: Adam optimizer, cross entropy loss function, learning rate of 4e-4, weight decay of 1e-5, training sequence length of 32, and batch size of 32. The model training of the set gait recognition convolutional neural network is started, and a total of 500 generations are trained to obtain the final network model, which is the trained gait recognition model.

6. A gait recognition device based on an improved convolutional neural network, characterized in that: The device comprises: The basic sequence attention convolution module building module is used to connect the convolution layer, batch normalization layer, and activation layer with activation function in sequence to form an improved basic convolution unit, and introduce sequence attention into the improved basic convolution unit to form a basic sequence attention convolution module; A gait recognition convolutional neural network building module is used to select multiple basic sequence attention convolution modules according to the size of the video image and connect them in sequence. After the multiple basic sequence attention convolution modules are connected in sequence, an average pooling layer is connected to build a gait recognition convolutional neural network; The training module is used to train a complete gait recognition model using the built gait recognition convolutional neural network; The recognition module is used to input the video data to be recognized into the trained gait recognition model for gait recognition and output the recognition results; The gait recognition convolutional neural network integrates temporal features at each layer and uses the features of the previous and next sequence graphs to recover the missing features. The steps of selecting multiple basic sequence attention convolution modules according to the size of the video image and sequentially connecting them, and connecting an average pooling layer after the multiple basic sequence attention convolution modules connected in sequence to build a gait recognition convolutional neural network include: According to the size of the video image, multiple basic sequence attention convolution modules are selected and connected in sequence. Each connected basic sequence attention convolution module gradually reduces the height and width of the feature map and gradually increases the number of channels to compress and extract the features step by step. An average pooling layer is connected after the multiple basic sequence attention convolution modules are connected in sequence, and a gait feature vector with an output length of 256 is obtained through the average pooling layer; The average pooling layer is sequentially connected to the fully connected layer and the Softmax layer to build a gait recognition convolutional neural network; The step of inputting the video data to be recognized into the trained gait recognition model for gait recognition and outputting the recognition result includes: Testing steps, the testing steps comprising: Half of the samples in the CASIA standard gait recognition dataset of 74-124 people in normal state were selected as benchmark samples, and the other half in normal state, wrapped state, and coat state were selected as test samples; Unify the size of the gait graph to 64x64; The gait sequence is input into the trained gait recognition model to obtain a feature vector with an output length of 256; Use the k-nearest neighbor algorithm to calculate the category to which the test sample belongs and calculate the accuracy; In the backpack state, the gait recognition convolutional neural network model filters the influence of the backpack state on the classification results through multi-layer convolution connections. At the same time, the enhancement of sequential attention can restore the partial occlusion of the leg information by the backpack. In the coat state, the sequence attention convolution module used integrates the gait information at all times and has a certain feature recovery ability.

7. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 5.

Images