Graph capsule based dual-stream gait recognition method and device, and medium

Abstract

The present disclosure provides a kind of double-flow gait recognition method, device and medium based on graph capsule, it is related to image recognition technical field, comprising: the skeleton data of the object to be identified is obtained, then the multi-layer feature extraction module in the first feature extractor of the trained gait recognition network is used to carry out feature extraction to skeleton data, and the second feature data output in each layer of multi-layer feature extraction module is obtained;Again, a predetermined number of second feature data is obtained, and the capsule module of the trained gait recognition network is used to carry out vectorization processing to the predetermined number of second feature data, and obtain first feature data, in the present disclosure, the skeleton data is processed into the first feature data of vector data by capsule module, vector data can represent the spatial relationship and posture information between different body parts, can more accurately capture and express gait mode, to improve the accuracy of gait recognition.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image recognition, in particular to a dual-flow gait recognition method based on graph capsules, a device and a medium. BACKGROUND

[0002] Gait recognition is identity recognition based on human walking posture. Compared with fingerprint recognition, voice recognition, face recognition and other technologies, gait recognition has the advantages of easy collection, long distance, non-contact, difficult disguise and non-invasive, and is a research hotspot in the fields of biometric recognition, computer vision and information security. In the gait recognition task, due to the interference of factors such as clothing, backpack and body shielding, the accuracy of the recognition model is affected. SUMMARY

[0003] The purpose of the present disclosure is to provide a dual-flow gait recognition method based on graph capsules, a device and a medium, which aims to solve the above problems.

[0004] The present disclosure provides a dual-flow gait recognition method based on graph capsules, the method comprising: obtaining skeleton data of a to-be-recognized object; performing feature extraction on the skeleton data through a trained gait recognition network to obtain first feature data; and obtaining a gait recognition result for the to-be-recognized object according to the first feature data; wherein the trained gait recognition network comprises a first feature extractor and a capsule module, the first feature extractor comprises a plurality of layers of feature extraction modules connected in sequence, and the trained gait recognition network performs feature extraction in the following manner: performing feature extraction on the skeleton data through the plurality of layers of feature extraction modules in the first feature extractor to obtain second feature data output by each layer of the plurality of layers of feature extraction modules, wherein the input of a first layer of feature extraction module of the first feature extractor is the skeleton data, and the input of the remaining layers of feature extraction modules of the first feature extractor is the second feature data output by the previous layer of feature extraction module of the layer of feature extraction module; obtaining a preset number of second feature data, and performing vectorization processing on the preset number of second feature data through the capsule module to obtain the first feature data, wherein the first feature data is vector data.

[0005] Optionally, the preset number is 2, and the preset number of second feature data is the second feature data output by the last two layers of the plurality of layers of feature extraction modules.

[0006] Optionally, the capsule module vectorizes the preset number of second feature data to obtain the first feature data, including: the capsule module vectorizes each of the preset number of second feature data to obtain third feature data; and the capsule module performs dynamic routing calculation on the third feature data to obtain the first feature data.

[0007] Optionally, the trained gait recognition network further comprises a second feature extractor, and the method further comprises: obtaining contour data of the to-be-identified object; performing feature extraction on the contour data by the second feature extractor to obtain fourth feature data; and obtaining the gait recognition result for the to-be-identified object according to the first feature data, including: obtaining the gait recognition result for the to-be-identified object according to the first feature data and the fourth feature data.

[0008] Optionally, the trained gait recognition network further comprises a feature splicing module, and obtaining the gait recognition result for the to-be-identified object according to the first feature data and the fourth feature data, including: performing feature splicing on the first feature data and the fourth feature data by the feature splicing module to obtain spliced feature data; and obtaining the gait recognition result for the to-be-identified object according to the spliced feature data.

[0009] Optionally, the trained gait recognition network is trained in the following manner: a plurality of training samples are obtained, wherein the plurality of training samples include a plurality of input samples and an output sample corresponding to each input sample; a target input sample is taken as an output of a gait recognition network, and a target output sample corresponding to the target input sample is taken as an output of the gait recognition network, and the gait recognition network is trained, wherein the target input sample is any input sample in the plurality of input samples; a supervised contrast loss function is obtained according to the target input sample and the target output sample; a graph reconstruction loss function is obtained according to the input sample and a capsule module output of the gait recognition network; a target loss function is obtained according to the supervised contrast loss function and the graph reconstruction loss function; and the gait recognition network is trained by the target loss function to obtain the trained gait recognition network.

[0010] Optionally, the supervised contrast loss function is represented by the following formula:

[0011]

[0012]

[0013] wherein, L SupConLtarget is a supervision contrast loss function, Ltarget is a loss function for a target sample, v i xtarget is a target input sample, v j(i) ytarget is a target output sample, τ is a temperature parameter, and 2N is a total number of training samples, v k xi is any one of the 2N samples, M is an indicator function, M is 1 when k is not equal to i, and M is 0 when k is equal to i, which is used to exclude positive sample pairs.

[0014] Optionally, the graph reconstruction loss function is represented by the following formula:

[0015]

[0016] wherein L Recon Lgraph is a graph reconstruction loss function, N is a total number of training samples, x i xtarget is a target input sample, W1 and W2 are both weight matrices, and y i is a capsule.

[0017] The present disclosure also provides a graph capsule-based dual-flow gait recognition device, the device comprising: a device acquisition module configured to acquire skeleton data of an object to be recognized; a feature extraction module configured to perform feature extraction on the skeleton data by using a trained gait recognition network to obtain first feature data; and a recognition module configured to obtain a gait recognition result for the object to be recognized according to the first feature data.

[0018] The trained gait recognition network comprises a first feature extractor and a capsule module, the first feature extractor comprises a plurality of feature extraction modules connected in sequence, and the trained gait recognition network performs feature extraction in the following manner:

[0019] The skeleton data is subjected to feature extraction by using the plurality of feature extraction modules in the first feature extractor to obtain second feature data output by each layer of the plurality of feature extraction modules, wherein the input of a first layer feature extraction module of the first feature extractor is the skeleton data, and the input of each layer feature extraction module other than the first layer feature extraction module in the first feature extractor is second feature data output by a previous layer feature extraction module of the layer feature extraction module.

[0020] A preset number of second feature data are acquired, and the preset number of second feature data are subjected to vectorization processing by using the capsule module to obtain the first feature data, wherein the first feature data is vector data.

[0021] Optionally, the present disclosure also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the foregoing method.

[0022] The method comprises the following steps: obtaining skeleton data of an object to be recognized; performing feature extraction on the skeleton data by a plurality of layers of feature extraction modules in a first feature extractor of a trained gait recognition network, to obtain second feature data output by each layer of the plurality of layers of feature extraction modules, wherein an input of a first layer of feature extraction module of the first feature extractor is the skeleton data, and an input of each layer of feature extraction module other than the first layer of feature extraction module in the first feature extractor is second feature data output by a previous layer of feature extraction module; obtaining a preset number of second feature data, and performing vectorization processing on the preset number of second feature data by a capsule module of the trained gait recognition network, to obtain first feature data, wherein the first feature data is vector data; and obtaining a gait recognition result for the object to be recognized according to the first feature data.

[0023] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, and are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation of the present disclosure. In the drawings:

[0025] Figure 1 is a flowchart of a graph capsule-based dual-flow gait recognition method shown in the present disclosure;

[0026] Figure 2 is a schematic diagram of a trained gait recognition network;

[0027] Figure 3 is a flowchart of step S120 in Figure 1

[0028] Figure 4 is a schematic diagram of a trained gait recognition network;

[0029] Figure 5 is a structural schematic diagram of a BottleNeck GCN module;

[0030] Figure 6 is a schematic diagram of a trained gait recognition network;

[0031] Figure 7 is a structural block diagram of a graph capsule-based dual-flow gait recognition device; ​

[0032] Figure 8 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] The specific embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the present disclosure, and are not intended to limit the present disclosure.

[0034] It should be noted that all actions of obtaining signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0035] Gait recognition is to identify the identity according to the posture of human walking. Compared with fingerprint recognition, voice recognition, face recognition and other technologies, gait recognition has the advantages of easy collection, long distance, non-contact, difficult disguise, non-invasive, etc. It is a research hotspot in the fields of biometric recognition, computer vision, information security, etc. In the gait recognition task, due to the interference of factors such as dress, backpack, limb occlusion, etc., the accuracy of the recognition model recognition will be affected.

[0036] In existing research, gait recognition is mainly developed from two angles:

[0037] (1) The gait contour graph is used to construct a template that maintains the time and space information in the gait sequence, and a machine learning method is used to learn a discriminative representation.

[0038] (2) A human pose generation model is used to generate a two-dimensional graph structure of a plurality of human joint nodes, and the attributes of each node include horizontal and vertical coordinates and confidence to save spatial information. By constructing a plurality of two-dimensional joint node graphs, the time information is maintained.

[0039] In early research, in the research process of angle (1), two commonly used gait templates, gait energy map and gait entropy map, were proposed. They have low computational complexity and effectively maintain the spatial information in the gait sequence, but lose the timing information. Since the gait template cannot completely save the time and space information in the gait sequence, and the machine learning method lacks the ability of end-to-end modeling of sequence data, the gait contour graph is used to construct the gait sequence, which becomes the mainstream gait feature. The deep learning model based on human contour gait sequence as input has developed rapidly since then.

[0040] In angle (2), the gait of the pedestrian is easily affected by various covariates, and the extraction of identity features is therefore quite challenging. The covariates that affect the performance of gait recognition include camera view, clothing, walking speed, object occlusion, and resolution, etc. Among them, the relative view angle between the camera and the pedestrian is the most influential covariate on the performance of gait recognition. Large view angle changes will cause high non-linear correlation between the gait features at different view angles. Compared with the gait silhouette, the two-dimensional skeleton joint is more robust to the above covariates, and is a more suitable gait feature for real-world scenarios.

[0041] Gait recognition, as a task with image sequences as input, is also suitable for modeling using deep learning methods to some extent. Non-linear models based on neural networks also provide an effective solution to eliminate the influence of covariates in gait recognition. The recent work on gait recognition using deep neural networks is divided into three categories: pre-trained model-based methods, energy gait map-based methods, 3D convolution-based methods, and human pose joint-based methods, and each method is introduced.

[0042] Discriminative methods include pre-trained model-based methods, gait energy map-based methods, 3D convolution-based methods, and metric learning-based methods. First, the pre-trained model-based method, Zhang et al. proposed DeepGait, which uses a pre-trained model VGG-16 (Visual Geometry Group-16) to obtain a deep convolutional feature representation based on gait silhouettes. First, the normalized silhouette sequence is obtained by extracting the region of interest (RoI) in the silhouette sequence. Then, the cycle detection method is used to detect the cycle of the silhouette sequence. Finally, the silhouette of a single cycle is used as the input of the VGG-16 model pre-trained based on the ImageNet dataset, and a set of feature representations corresponding to the sequence length is obtained.

[0043] Unlike directly using gait silhouettes as the input of a deep convolutional network, Shiraga et al. use gait energy maps as the input of the model. The gait energy map is a gait template that mixes static and dynamic information in the gait silhouette sequence. The energy of each pixel in the template is obtained by calculating the average intensity of the silhouette pixels in a gait cycle, as shown in equation (1):

[0044]

[0045] where L is the number of frames in a single gait cycle, B tis the silhouette of the t-th frame in the gait cycle. To solve the cross-view problem in gait recognition, a network structure with two convolutional layers is proposed, and a feature representation invariant to view is obtained through a fully connected layer. During the training process, the recognition problem is regarded as a classification problem on the training dataset, and the cross-entropy loss of multi-classification is calculated using the Softmax function in the last layer of the network. In the test stage, the obtained view-invariant features are used to identify the nearest neighbor classifier.

[0046] For the method of 3D convolution, Wolf et al. proposed to capture the spatio-temporal information in the gait sequence using 3D convolution, and used a multi-view 3D convolutional neural network (MV3DCNN) to input the gait image sequence with gray information into the network, and introduced the optical flow image to process the influence of clothing and color on the recognition performance. In order to solve the problem that the convolutional network cannot process indefinite length gait sequence, a gait sequence is cut into several short sequences of fixed length for processing.

[0047] Thapar et al. also used 3D convolution to extract features from gait silhouette sequences. Different from Wolf et al., for a gait sequence, this method first predicts the view of the gait sequence through a view classifier, and then selects a corresponding 3D convolutional network for recognition according to the predicted view. The same as Wolf et al., this method also cuts a gait sequence into short sequences of fixed length. In the test stage, the identity estimate corresponding to each short sequence can be obtained, and finally the majority voting method is used to judge the identity of the original gait sequence.

[0048] The method based on human pose joint is a very novel and pioneering method. Liao et al. used an open-source pose estimation algorithm to extract human pose information from the original video sequence, which contains the positions of 6 human key points (left and right hips, left and right knees, and left and right ankles) in each frame of the original video sequence. In order to eliminate the influence of the scale change of the distance between the camera and the person during walking, first, the coordinates of each key point are normalized, as shown in formula (2):

[0049]

[0050] where p z is the coordinate of the z-th key point, p' z is the normalized coordinate of the z-th key point, p neck is the coordinate of the key point at the neck of the human body, H nhThe height is from the center of the hip to the neck. After obtaining the normalized gait key point sequence, a pose-based temporal-spatial network (PTSN) is proposed to learn the gait feature representation. Specifically, the spatial information in the key point sequence is extracted by using a convolutional neural network, and the temporal information is extracted by using a long short-term memory (LSTM) module. The LSTM has been proven to be effective in modeling sequence data. For the selection of the loss function, a multi-classification cross-entropy loss and a binary tuple loss are used simultaneously, and the model is trained in a weighted summation manner.

[0051] Yan et al. first applied a graph convolutional network (GCN) to the field of skeleton-based action recognition. First, the skeleton joints are taken as vertices and the skeleton is taken as edges to construct a skeleton graph, and a division strategy is designed according to the distance and spatial configuration. Then, a spatio-temporal graph convolutional layer is built by convolution and two-dimensional time domain convolution to extract spatio-temporal information.

[0052] Existing pose estimation algorithms have high tolerance for self-occlusion, clothing and carrying changes. Compared with using gait images, using pose key points for gait recognition can effectively alleviate the influence of covariant changes on the performance of gait recognition. However, one of the shortcomings is that the effectiveness of the model is not verified in the cross-view scene, and the prediction accuracy of the joint pose estimation network is too dependent, resulting in the effect of the gait recognition network using the joint is not as good as the effect of the network using the contour.

[0053] Therefore, the present disclosure provides a dual-flow gait recognition method based on a graph capsule, please refer to Figure 1 , the dual-flow gait recognition method based on a graph capsule can be applied to Figure 7 the dual-flow gait recognition device 400 based on a graph capsule shown in Figure 8 the electronic device 700 and the computer readable storage medium. In this embodiment, the electronic device is taken as an example, which can be a mobile phone, a computer, a server, a monitoring device, a vehicle-mounted controller on a vehicle, an access control device, etc. The following will be described in detail with respect to Figure 1 the flow shown in, the dual-flow gait recognition method based on a graph capsule can specifically include the following steps:

[0054] Step S110, obtaining skeleton data of a to-be-identified object.

[0055] The skeleton data is obtained by tracking and positioning the joints of the to-be-identified object to represent the pose and action of the to-be-identified object. The joints can be the head, shoulders, arms, legs, etc. of the to-be-identified object.

[0056] Optionally, the image of the object to be recognized is captured by the camera, and then the image is processed into the skeleton data. In order to improve the accuracy of gait recognition, a preset number of images can be extracted from the video captured by the camera, and then the preset number of images are processed into the skeleton data sequence x0, x1, …, xn. n-1 wherein n is a preset number, x0 represents the skeleton data corresponding to the first image, x1 represents the skeleton data corresponding to the second image, and xn represents the skeleton data corresponding to the nth image. n-1 wherein n is a preset number, x0 represents the skeleton data corresponding to the first image, x1 represents the skeleton data corresponding to the second image, and xn represents the skeleton data corresponding to the nth image.

[0057] Optionally, the skeleton data can also be obtained by an inertial measurement unit, a sensor array, etc.

[0058] In step S120, the trained gait recognition network is used to extract features from the skeleton data, and first feature data is obtained.

[0059] Referring to Figure 2 , the trained gait recognition network 100 includes a first feature extractor 120 and a capsule module 130. The first feature extractor includes a plurality of feature extraction modules connected in sequence.

[0060] Referring to Figure 3 , the trained gait recognition network extracts features in the following manner:

[0061] In step S121, the skeleton data is extracted by the plurality of feature extraction modules in the first feature extractor, and second feature data output by each layer of the plurality of feature extraction modules is obtained. The input of the first layer feature extraction module of the first feature extractor is the skeleton data, and the input of the remaining layer feature extraction modules of the first feature extractor is the second feature data output by the previous layer feature extraction module.

[0062] Optionally, in order to improve the generalization ability of the trained gait recognition network, referring to Figure 2 , the trained gait recognition network 100 can also include a normalization module 110. The skeleton data 200 is input into the trained gait recognition network 100, and the normalization module 110 in the network is used to normalize the skeleton data 200, thereby obtaining normalized skeleton data. The normalized skeleton data is input into the first feature extractor 120, and the first feature extractor 120 is used to extract features from the skeleton data, thereby obtaining a plurality of second feature data.

[0063] In step S122, a preset number of second feature data is obtained, and the preset number of second feature data is subjected to vectorization processing by the capsule module to obtain the first feature data, wherein the first feature data is vector data.

[0064] Different second feature data provides different receptive fields. In this embodiment, the preset number can be 1. The preset number can also be at least 2, for example, the preset number is 2, 3, etc. At least two second feature data can provide a larger receptive field, which facilitates the model to better understand and extract the content of the image.

[0065] In an embodiment, each of the preset number of second feature data is subjected to vectorization processing by the capsule module to obtain third feature data. It can be understood that the third feature data is obtained by vectorizing the second feature data, and the third feature data is vector data. For example, the vectorization processing is performed in the manner of formula (3) as follows:

[0066] Z i =W i H i (3)

[0067] Wherein, i is the layer number in the multi-layer feature extraction module, W i is a weight matrix, H i is the second feature data output by the i-th layer in the multi-layer feature extraction module, and Z i is the third feature data.

[0068] The third feature data is further subjected to dynamic routing calculation by the capsule module to obtain the first feature data.

[0069] As a way, the dynamic routing calculation on the third feature data can be divided into two stages, as follows, the dynamic routing calculation on the third feature data is performed to obtain capsule vectors h0,..., hn. p-1 Wherein, p is a preset number, and p is a positive integer less than or equal to n. Then the above capsule vectors h0,..., hn p-1 are subjected to dynamic routing calculation again to generate final class capsules C0,..., CK. K-1 Wherein, K is the output dimension. The final class capsules C0,..., CK K-1 are the first feature data.

[0070] It should be noted that each of the selected preset number of second feature data needs to be vectorized by the above-mentioned manner before dynamic routing calculation. It is not difficult to understand that one first feature data can be obtained by performing the above-mentioned calculation on each second feature data in the preset number. Then, a plurality of first feature data can be obtained by performing the above-mentioned calculation on the preset number of second feature data. In this step, the above-mentioned calculation is performed on the second feature data, which can be understood as the capsule of the second feature data.

[0071] In step S130, a gait recognition result of the to-be-recognized object is obtained according to the first feature data.

[0072] For example, the gait database stores pre-stored feature data and pre-stored labels corresponding to the pre-stored feature data. The pre-stored feature data is obtained from the gait database, and the similarity between the first feature data and the pre-stored feature data is calculated. If the similarity is greater than a preset similarity, the pre-stored label corresponding to the pre-stored feature data is taken as the gait recognition result. The gait recognition result is used to represent the identity of the to-be-recognized object.

[0073] Alternatively, when the gait recognition method based on the graph capsule is applied to the intelligent door lock, if the recognition result matches the pre-stored user information, it indicates that the to-be-recognized object is a family member, and the intelligent door lock is controlled to be opened. Or, if the recognition result does not match the pre-stored user information, it indicates that the to-be-recognized object is not a family member, and the image information captured by the camera on the intelligent door lock is sent to the user terminal corresponding to the family member.

[0074] The gait recognition method based on the graph capsule provided by the present disclosure obtains the skeleton data of the to-be-recognized object, and then performs feature extraction on the skeleton data through the multi-layer feature extraction module in the first feature extractor of the trained gait recognition network to obtain the second feature data output by each layer of the multi-layer feature extraction module. The input of the first layer feature extraction module of the first feature extractor is the skeleton data, and the input of the remaining layer feature extraction modules in the first feature extractor is the second feature data output by the previous layer feature extraction module of the layer feature extraction module. Then, a preset number of second feature data is obtained, and the preset number of second feature data is vectorized by the capsule module of the trained gait recognition network to obtain first feature data, wherein the first feature data is vector data. According to the first feature data, a gait recognition result of the to-be-recognized object is obtained. In the present disclosure, the skeleton data is processed into the first feature data of the vector data by the capsule module. The vector data can represent the spatial relationship and posture information between different body parts, and can more accurately capture and express the gait pattern, thereby improving the accuracy of gait recognition.

[0075] Optionally, the preset number is 2, and the preset number of second feature data is second feature data output by last two layers of the multi-layer feature extraction module. For example, refer to Figure 4 The normalization module 110 can be a Batch Norm module. The first feature extractor 120 includes seven layers of feature extraction modules connected in sequence. The feature extraction module can be a BottleNeck GCN (BottleNeck Graph Convolutional Network) module in Figure 4 The capsule module 130 is a Graph Capsules module in Figure 4 Refer to Figure 4 The skeleton data sequence 200 is input into the Batch Norm module 110, and the Batch Norm module is used to normalize the skeleton data sequence to obtain a normalized skeleton data sequence. The skeleton data sequence is sent to the first layer of the seven-layer feature extraction module, and the first layer of the feature extraction module is used to extract features, changing the original three channels to 64 channels. The feature data output by the first layer of the feature extraction module is input into the second layer of the feature extraction module for feature extraction. In this way, the features extracted by the previous layer of the feature extraction module are input into the next layer, until the seventh layer of the feature extraction module extracts features. The second feature data output by the sixth layer of the feature extraction module and the seventh layer of the feature extraction module is input into the Graph Capsules module 130, and the Graph Capsules module 130 is used to process the second feature data output by the sixth layer of the feature extraction module to obtain the first feature data corresponding to the sixth layer of the feature extraction module. The Graph Capsules module 130 processes the second feature data output by the seventh layer of the feature extraction module in a similar manner, which will not be described here.

[0076] Optionally, as shown in Figure 5 The BottleNeck GCN module includes a Graph Convolution (Graph Convolution) and a Temporal Convolution (two-dimensional time domain convolution), and both of the two convolution blocks have a BottleNeck structure and a Skip Connection.

[0077] The BottleNeck GCN module processes the second feature data output by the seventh layer of the feature extraction module in a similar manner, which will not be described here.

[0078]

[0079] where H(X) represents the output feature matrix (feature map), which is the result after spatial graph convolution processing, X is the input feature matrix, which represents the spatial information at the current moment, and is usually a node feature matrix. That is, X is actually a tensor containing time and space features, as well as an adjacency matrix, which are provided as input to the model at the same time. D is a predefined maximum graph distance, and d represents the graph distance between each pair of nodes from 0 to D. GCN extracts features for each pair of nodes, so it can capture the spatial relationship between different positions (nodes). represents element-wise multiplication, A d represents the D-order adjacency matrix of the joint pair marked by the graph distance. Λ d is used to normalize A d . W d and M d are weight matrices.

[0080] Optionally, on the basis of Figure 4 , the trained gait recognition network 100 further comprises Figure 6 a second feature extractor 140 in . Based on this, the method further comprises: obtaining contour data 300 of the object to be identified, wherein the contour data and the skeleton data can be two different types of data obtained from the same image. It can be understood that the contour data and the skeleton data both express the same object to be identified, the posture and action at the same moment. Then, the second feature extractor 140 is used to extract features from the contour data 300 to obtain fourth feature data. In an embodiment, step S130 comprises: obtaining the gait recognition result for the object to be identified according to the first feature data and the fourth feature data.

[0081] In this embodiment, the skeleton data is processed by the first feature extractor and the capsule module, and the contour data is processed by the second feature extractor 140. The double-flow processing of the skeleton data and the contour data by the trained gait recognition network can improve the processing speed of the data, and the gait recognition combined with the two types of data can improve the accuracy of the gait recognition.

[0082] Figure 6 Please continue to refer to , the trained gait recognition network 100 further comprises a feature splicing module 150, and the gait recognition result for the object to be identified according to the first feature data and the fourth feature data can be obtained as follows: the first feature data and the fourth feature data are spliced by the feature splicing module to obtain spliced feature data; and the gait recognition result for the object to be identified is obtained by using the spliced feature data.

[0083] As one approach, the trained gait recognition network 100 also includes a feature fusion module (in... Figure 6 (Not shown in the diagram) After feature stitching by feature stitching module 150, stitched feature data is obtained. Then, feature fusion module performs feature fusion on the stitched feature data to obtain fused feature data. Based on the fused feature data, gait recognition results are obtained. In this embodiment, feature fusion provides a richer and more comprehensive feature representation, facilitating subsequent gait recognition.

[0084] The trained gait recognition network is trained in the following manner:

[0085] Obtain multiple training samples, wherein the multiple training samples include multiple input samples x1, x2, ... x N And the corresponding output samples x1', x2', ..., x for each input sample. N Then input the target sample x. i As the output of the gait recognition network, and the target input sample x is used as the output of the gait recognition network. i The corresponding target output sample x' i As the output of the gait recognition network, the gait recognition network is trained, wherein the target input sample is any one of the plurality of input samples. Then, based on the target input sample and the target output sample, a supervised contrastive loss function is obtained, which can be understood as the loss function corresponding to the gait recognition network. And based on the input sample and the capsule class output by the capsule module of the gait recognition network, a graph reconstruction loss function is obtained, which can be understood as the loss function corresponding to the capsule module. Based on the supervised contrastive loss function and the graph reconstruction loss function, a target loss function is obtained, that is, the sum of the supervised contrastive loss function and the graph reconstruction loss function is calculated as the target loss function. The gait recognition network is trained using the target loss function to obtain the trained gait recognition network.

[0086] The supervised contrastive loss function is obtained through formulas (5-1) and (5-2):

[0087]

[0088]

[0089] Among them, L SupCon To supervise the contrastive loss function, For the loss function of the target sample, v i The target input sample can be understood as sample i, v out of 2N samples. j(i)is the target output sample, τ is a temperature parameter, 2N is the total number of training samples, v k is an indicator function, M is 1 when k is not equal to i, and M is 0 when k is equal to i, for excluding positive sample pairs.

[0090] where v j(i) is a sample representation derived from sample i, which can be understood as sample j in the 2N samples, because they are derived from the same sample, the purpose of training is to make the features of sample i and sample j closer. In addition to sample i and sample j, for the remaining 2(N-1) samples, because they are derived from different samples than sample i, their features are better the farther away from the features of i.

[0091] τ is a positive number used to scale the scale of inner product, when training neural network, temperature parameter is sometimes used to adjust the scale of loss function, to balance the contribution of different components or make training more stable. In the learning process of contrast such as Supervised Contrastive Loss, temperature parameter τ can affect the similarity measure between positive and negative samples. Higher temperature value can make the similarity between sample pairs smoother, thus reducing the sensitivity of the model. Temperature parameter τ is usually a hyperparameter, which needs to be adjusted according to the task and data.

[0092] The supervised contrastive loss function in the embodiment considers all positive and negative samples in the batch, contains more positive samples in a batch, unlike the three tuple loss which is limited to one positive sample corresponding to one negative sample, the supervised contrastive loss function in the embodiment considers more samples, so when used for model training, better results can be obtained.

[0093] The graph reconstruction loss function is represented by formula (6):

[0094]

[0095] where L Recon is the graph reconstruction loss function, N is the total number of training samples, x i is the target input sample, W1 and W2 are both weight matrices, y i is the capsule.

[0096] The disclosure first introduces the feature extraction method of non-traditional convolution, i.e., the capsule network (i.e., the aforementioned capsule module) and the dynamic routing mechanism into the gait recognition method, which is not only an improvement idea for improving the connection between human joints in gait recognition, but also explores the feasibility of dynamic routing in the gait recognition task, and belongs to the pioneering application. Since the disclosure adopts the dynamic routing technology of the capsule network, it can overcome the shortcomings of the translation invariance of the traditional convolution. Here, the translation invariance refers to that although the sub-features of the object to be recognized do not change, the traditional convolution cannot learn the difference if the directionality deviates. However, the capsule network of the disclosure can accurately capture the weak dynamic features of the human gait sequence, thereby overcoming the problem of translation invariance. In addition, the neurons in the capsule network are a whole, containing various important information of the feature state, such as length, angle, direction, etc., while each neuron in the traditional convolutional neural network is an independent individual and cannot depict position, angle, etc. Therefore, adopting the capsule network to replace the traditional graph convolution can more effectively cope with the multi-view transformation scene. Compared with the over-smoothing risk faced by the construction of the depth graph convolution network, the embedding method of the capsule network can obtain better recognition effect. The aforementioned gait recognition network relies on matrix transformation to process the four-dimensional tensor [Batchsize, Bhannel, Frame, Node] into the node capsule vector [Batchsize, Frame*Node, 32, 8], which is the first capsule storage form of the gait feature. The disclosure improves the spatio-temporal graph convolution feature extraction network as a branch, together with the convolutional gait recognition network based on the human contour (i.e., the aforementioned second feature extractor) to build a double-branch structure. And using a feature fusion method, the skeleton joint feature mapping and the image mapping extracted by the two branch networks are complementary, the joint feature supervises the learning of the silhouette feature, so that the model has more stable feature capturing ability in the sub-set with larger covariates, further improving the accuracy of gait recognition.

[0097] Based on the same inventive concept, the disclosure provides a double-flow gait recognition device based on graph capsules, which is described in detail in Figure 7 The double-flow gait recognition device based on graph capsules 400 comprises:

[0098] The acquisition module 410 is configured to acquire skeleton data of an object to be recognized.

[0099] The extraction module 420 is configured to perform feature extraction on the skeleton data by using a trained gait recognition network to obtain first feature data.

[0100] The recognition module 430 is configured to obtain a gait recognition result for the object to be recognized according to the first feature data.

[0101] The trained gait recognition network comprises a first feature extractor and a capsule module, the first feature extractor comprises a plurality of feature extraction modules connected in sequence, and the trained gait recognition network performs feature extraction in the following manner:

[0102] The skeleton data is subjected to feature extraction through the plurality of feature extraction modules in the first feature extractor, and second feature data output by each layer of the plurality of feature extraction modules is obtained, wherein the input of a first layer feature extraction module of the first feature extractor is the skeleton data, and the input of each layer feature extraction module other than the first layer feature extraction module of the first feature extractor is second feature data output by a previous layer feature extraction module of the layer feature extraction module;

[0103] A preset number of second feature data are obtained, and the preset number of second feature data are subjected to vectorization processing through the capsule module to obtain the first feature data, wherein the first feature data is vector data.

[0104] Optionally, the preset number is 2, and the preset number of second feature data is second feature data output by the last two layers of the plurality of feature extraction modules.

[0105] Optionally, the extraction module 420 comprises:

[0106] The vectorization module is configured to perform vectorization processing on each of the preset number of second feature data through the capsule module to obtain third feature data.

[0107] The routing calculation module is configured to perform dynamic routing calculation on the third feature data through the capsule module to obtain the first feature data.

[0108] Optionally, the trained gait recognition network further comprises a second feature extractor, and the dual-flow gait recognition device 400 based on a graph capsule further comprises:

[0109] The contour data acquisition module is configured to acquire contour data of the to-be-identified object.

[0110] The feature extraction module is configured to perform feature extraction on the contour data through the second feature extractor to obtain fourth feature data.

[0111] The recognition module 430 is specifically configured to obtain the gait recognition result for the to-be-identified object according to the first feature data and the fourth feature data.

[0112] Optionally, the trained gait recognition network further comprises a feature splicing module, and the recognition module 430 comprises:

[0113] The splicing module is configured to perform feature splicing on the first feature data and the fourth feature data by the feature splicing module to obtain spliced feature data.

[0114] The splicing identification module is configured to obtain the gait recognition result for the to-be-recognized object based on the spliced feature data.

[0115] Optionally, the graph capsule-based dual-flow gait recognition apparatus 400 further comprises a training module.

[0116] The training module is configured to obtain a plurality of training samples, wherein the plurality of training samples comprise a plurality of input samples and an output sample corresponding to each input sample.

[0117] The target input sample is taken as the output of the gait recognition network, and the target output sample corresponding to the target input sample is taken as the output of the gait recognition network, and the gait recognition network is trained, wherein the target input sample is any input sample in the plurality of input samples.

[0118] A supervised contrast loss function is obtained according to the target input sample and the target output sample.

[0119] A graph reconstruction loss function is obtained according to the input sample and a class capsule output by a capsule module of the gait recognition network.

[0120] A target loss function is obtained according to the supervised contrast loss function and the graph reconstruction loss function.

[0121] The gait recognition network is trained by the target loss function to obtain the trained gait recognition network.

[0122] Optionally, the supervised contrast loss function is represented by the following formula:

[0123]

[0124]

[0125] wherein L SupCon is a supervised contrast loss function, is a loss function for a target sample, v i is a target input sample, v j(i) is a target output sample, τ is a temperature parameter, and 2N is a total number of training samples, v k is any one of the 2N samples, and M is an indicator function, which is 0 when k is equal to i, and is used to exclude positive sample pairs, and which is 1 when k is not equal to i.

[0126] Optionally, the graph reconstruction loss function is represented by the following formula:

[0127]

[0128] wherein L Recon is a graph reconstruction loss function, N is the total number of training samples, x i is a target input sample, W1 and W2 are weight matrices, y i is a class capsule.

[0129] As to the gait recognition device 400 of the graph capsule in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment related to the method, and will not be described in detail here.

[0130] Figure 8 is a block diagram of an electronic device according to an exemplary embodiment. As Figure 8 shown, the electronic device 700 can include a processor 701, a memory 702. The electronic device 700 can also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0131] The processor 701 is configured to control overall operations of the electronic device 700 to complete all or part of the steps of the gait recognition method based on the graph capsule described above. The memory 702 is configured to store various types of data to support operations of the electronic device 700, which can include, for example, instructions for any application or method operating on the electronic device 700, and application-related data, such as contact data, transmitted and received messages, pictures, audio, video, and the like. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk. The multimedia component 703 can include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 702 or transmitted through the communication component 705. The audio component further includes at least one speaker configured to output audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 705 is configured to perform wired or wireless communication between the electronic device 700 and other devices. The wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, and the like, or a combination of one or more of them, is not limited herein. Therefore, the communication component 705 can include, for example, a Wi-Fi module, a Bluetooth module, an NFC module, and the like.

[0132] In an exemplary embodiment, the electronic device 700 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for performing the above-described gait recognition method based on graph capsules.

[0133] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described gait recognition method based on graph capsules. For example, the computer-readable storage medium can be the above-described memory 702 including program instructions, which can be executed by the processor 701 of the electronic device 700 to complete the above-described gait recognition method based on graph capsules.

[0134] In another exemplary embodiment, a computer program product is also provided, which contains a computer program capable of being executed by a programmable device, the computer program having code portions for performing the above-described gait recognition method based on graph capsules when executed by the programmable device.

[0135] In summary, the dual-flow gait recognition method based on graph capsules, the device and the medium provided by the present disclosure are provided. Skeleton data of an object to be recognized is obtained, and then the multi-layer feature extraction module in the first feature extractor of the trained gait recognition network is used to extract features of the skeleton data to obtain second feature data output by each layer of the multi-layer feature extraction module. The input of the first layer feature extraction module of the first feature extractor is the skeleton data, and the input of the remaining layer feature extraction modules in the first feature extractor is the second feature data output by the previous layer feature extraction module of the layer feature extraction module. Then, a preset number of second feature data is obtained, and the capsule module of the trained gait recognition network is used to perform vectorization processing on the preset number of second feature data to obtain first feature data, wherein the first feature data is vector data. According to the first feature data, a gait recognition result for the object to be recognized is obtained. In the present disclosure, the skeleton data is processed into the first feature data of the vector data by the capsule module. The vector data can represent the spatial relationship and posture information between different body parts, and can more accurately capture and express the gait pattern, thereby improving the accuracy of gait recognition.

[0136] The preferred embodiments of the present disclosure are described in detail above in combination with the drawings, but the present disclosure is not limited to the specific details in the above-described embodiments. Within the technical concept range of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all belong to the protection range of the present disclosure.

[0137] In addition, it should be noted that each specific technical feature described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, various possible combinations are not described again in the present disclosure.

[0138] In addition, any combination of various different embodiments of the present disclosure can also be made, as long as it does not deviate from the idea of the present disclosure, and it should also be considered as disclosed by the present disclosure.

Claims

1. A two-stream gait recognition method based on graph capsules, characterized in that, The method comprises: obtaining skeleton data of an object to be identified; extracting features of the skeleton data through a trained gait recognition network to obtain first feature data; obtaining a gait recognition result for the object to be identified according to the first feature data; wherein the trained gait recognition network comprises a first feature extractor and a capsule module, the first feature extractor comprises a plurality of feature extraction modules connected in sequence, and the trained gait recognition network extracts features in the following manner: extracting features of the skeleton data through the plurality of feature extraction modules in the first feature extractor to obtain second feature data output by each layer of the plurality of feature extraction modules, wherein the input of a first layer feature extraction module of the first feature extractor is the skeleton data, and the input of each layer feature extraction module other than the first layer feature extraction module is the second feature data output by a previous layer feature extraction module of the layer feature extraction module; obtaining a preset number of second feature data and performing vectorization processing on the preset number of second feature data through the capsule module to obtain the first feature data, wherein the first feature data is vector data; the trained gait recognition network further comprises a second feature extractor and a feature splicing module, and the method further comprises: obtaining contour data of the object to be identified; extracting features of the contour data through the second feature extractor to obtain fourth feature data; wherein the gait recognition result for the object to be identified is obtained according to the first feature data, comprising: splicing the first feature data and the fourth feature data through the feature splicing module to obtain spliced feature data; obtaining the gait recognition result for the object to be identified through the spliced feature data.

2. The method of claim 1, wherein, The preset number is 2, and the preset number of second feature data is the second feature data output by the last two layers of the plurality of feature extraction modules.

3. The method of claim 1, wherein, The vectorization processing on the preset number of second feature data through the capsule module to obtain the first feature data comprises: performing vectorization processing on each of the preset number of second feature data through the capsule module to obtain third feature data; performing dynamic routing calculation on the third feature data through the capsule module to obtain the first feature data.

4. The method of claim 1, wherein, The trained gait recognition network is trained in the following manner: obtaining a plurality of training samples, wherein the plurality of training samples comprise a plurality of input samples and an output sample corresponding to each input sample; taking a target input sample as the input of the gait recognition network and taking a target output sample corresponding to the target input sample as the output of the gait recognition network to train the gait recognition network, wherein the target input sample is any input sample in the plurality of input samples; obtaining a supervised contrast loss function according to the target input sample and the target output sample; According to the input sample and a capsule output by a capsule module of the gait recognition network, a graph reconstruction loss function is obtained; According to the supervised contrast loss function and the graph reconstruction loss function, a target loss function is obtained; The gait recognition network is trained through the target loss function, and the trained gait recognition network is obtained.

5. The method of claim 4, wherein, The supervised contrast loss function is represented by the following formula: wherein, is a supervision contrast loss function, is a loss function for a target sample, is a target input sample, is a target output sample, is a temperature parameter, N is a total number of training samples, is any one of the N samples, M is an indicator function, M is 0 when k is equal to i, used to exclude positive samples, M is 1 when k is not equal to i.

6. The method of claim 4, wherein, The graph reconstruction loss function is represented by the following formula: wherein, is a graph reconstruction loss function, N is the total number of training samples, is a target input sample, and are weight matrices, is a capsule.

7. A dual-stream gait recognition apparatus based on graph capsules, characterized in that, The device comprises: An acquisition module configured to acquire skeleton data of a to-be-identified object; An extraction module configured to perform feature extraction on the skeleton data through a trained gait recognition network, and obtain first feature data; An identification module configured to obtain a gait recognition result for the to-be-identified object according to the first feature data. The trained gait recognition network comprises a first feature extractor and a capsule module, the first feature extractor comprises a plurality of layers of feature extraction modules connected in sequence, and the trained gait recognition network further comprises a second feature extractor and a feature splicing module. The trained gait recognition network performs feature extraction in the following manner: The skeleton data is subjected to feature extraction through the plurality of layers of feature extraction modules in the first feature extractor, and second feature data output by each layer of the plurality of layers of feature extraction modules is obtained, wherein the input of a first layer of feature extraction module of the first feature extractor is the skeleton data, and the input of each layer of feature extraction module other than the first layer of feature extraction module in the first feature extractor is second feature data output by a previous layer of feature extraction module of the layer of feature extraction module; A preset number of second feature data are acquired, and the preset number of second feature data are subjected to vectorization processing through the capsule module, and first feature data are obtained, wherein the first feature data are vector data; A contour data acquisition module configured to acquire contour data of the to-be-identified object; A feature extraction module configured to perform feature extraction on the contour data through the second feature extractor, and obtain fourth feature data; The identification module is specifically configured to perform feature splicing on the first feature data and the fourth feature data through the feature splicing module, and obtain spliced feature data; and obtain the gait recognition result for the to-be-identified object through the spliced feature data.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method in any one of claims 1-6.

Images