Cross-view gait recognition method based on feature fusion and view triplet loss

Abstract

The present invention discloses a cross-view gait recognition method based on feature fusion and perspective triplet loss. The present invention collects walking image sequences under different perspectives, extracts global gait energy graphs and local spatiotemporal sequence gait representations; realizes nonlinear mapping of gait representations to deep discriminative features; defines the perspective hierarchical Euclidean distance between the deep features of gait representations as the metric distance between gait representations; uses the perspective triplet loss function to supervise the training process of the feature learning network; fuses two different types of deep features; uses the differences between different gaits to calculate the Euclidean distance between the gait fusion features to be identified and the gait fusion features in the database, and returns the gait sample with the smallest Euclidean distance to the gait sample to be identified in the database as the identity recognition result of the gait to be identified. The present invention is simple and effective, with high recognition accuracy, can effectively cope with the challenge of gait recognition with perspective changes, and realizes unlimited gait recognition under cross-perspectives.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a cross-view gait recognition method based on feature fusion and view triplet loss. Background Art

[0002] Gait recognition is a novel biometric recognition technology that aims to use cutting-edge digital technologies such as computer vision and pattern recognition to mine the intrinsic walking motion characteristics of individuals and convert them into unique biometrics, thereby realizing identity recognition and verification tasks based on human gait. Compared with other biometric recognition technologies, gait recognition has great development prospects due to its advantages such as gait uniqueness, non-physical contact, individual passivity, and long-distance observability. Therefore, gait recognition has always been an important topic in the field of pattern recognition. In actual monitoring application scenarios, most gait recognition methods can normally achieve identity recognition under simple conditions and achieve a high accuracy recognition rate. However, these methods have low robustness to changes in gait appearance caused by changes in walking conditions (such as changes in walking speed, perspective, clothing, carrying status, distance, etc.). That is, when walking conditions change, the appearance of pedestrians' gait usually deforms, so the correct recognition rate will drop sharply, which seriously hinders the advancement of the actual application of gait recognition. Among these walking conditions, one of the biggest difficulties lies in the impact of changes in perspective. Since the walking route and camera angle of the identified object in the monitoring scene cannot remain unchanged, and cross-camera gait recognition is required in actual monitoring, the problem of perspective change is an unavoidable problem in gait recognition.

[0003] At present, most of the solutions to the problem of cross-view gait recognition are to find view-independent gait features and use deep neural networks for metric learning. Although many effective cross-view gait recognition studies have emerged, these methods still face the following challenges:

[0004] (1) Many existing methods based on manually extracted view-independent features have low robustness to view changes. When the gait angle changes greatly, the recognition accuracy will be greatly reduced or even fail. In addition, these methods usually rely on accurate gait profile images and a large amount of computing resources.

[0005] (2) Relying on a single type of global feature or local feature. Human gait is an extremely complex nonlinear motion. It is difficult to accurately describe human gait using a single type of feature. Describing human gait from multiple perspectives and extracting multiple types of features as clues to describe human gait can enhance the expressive power of features and reduce the sensitivity of features to viewing angles, which will obviously be of great help to the realization of cross-view gait recognition.

[0006] (3) Most deep learning-based and metric learning-based methods only consider the differences between the extracted deep feature vectors, ignoring the fact that changes in perspective bring about changes in gait appearance. In fact, changes in perspective can cause dramatic changes in the appearance of human gait. When the perspective changes greatly, the intra-class differences may even be greater than the inter-class differences. If the fact that changes in perspective bring about changes in the appearance of gait contours can be considered when training the gait recognition model, it will obviously enhance the robustness to changes in perspective. Summary of the invention

[0007] The purpose of the present invention is to overcome the limitations and shortcomings of existing methods and provide a cross-view gait recognition method based on feature fusion and view triplet loss. The method is simple and effective, has high recognition accuracy, and can adapt to gait recognition solutions based on global local features and deep metric learning with large changes in view angle.

[0008] The present invention extracts two different types of gait representations, namely, a conventional global gait energy graph and a local spatiotemporal sequence, from a binary contour image sequence of human walking; based on the global and local gait representations, feature learning networks are respectively designed to realize nonlinear mapping from gait representation to deep features; the perspective hierarchical Euclidean distance between the deep features of gait representation is defined as the metric distance between gait representations, and the metric distance function uses the perspective difference amplitude between gait representations to weight the difference between the deep feature vectors of gait representations; a perspective triplet loss function is used to supervise the training process of the feature learning network, and the loss function considers that the gait contour image is jointly affected by the individual gait motion characteristics and the perspective, and uses the deep features, identity and perspective information of the training samples in combination with the perspective hierarchical Euclidean distance to calculate the loss value, and the loss value is calculated in the gait training data. The online most difficult samples and viewpoint difficult samples are used to mine to increase the triplets available for each training. Finally, the viewpoint triplet loss function is minimized so that the learned nonlinear mapping can map the gait representation to a specific space, in which the metric distance between the gait representations from the same pedestrian is smaller than the metric distance between the gait representations from different pedestrians, wherein the metric distance between the gait representations is related to their deep feature differences and viewpoint differences; two different types of deep features are fused as the final human gait feature representation; by utilizing the differences between different gait representations, the Euclidean distance between the fused features of the gait to be identified and the fused features of the gait in the database is calculated, and the gait sample in the database with the smallest Euclidean distance to the gait sample to be identified is returned as the identity recognition result of the gait to be identified, thereby realizing cross-viewpoint gait recognition.

[0009] The specific technical solution of the present invention is achieved through the following steps:

[0010] Step 1: Collect walking image sequences from different perspectives and extract global and local gait representations;

[0011] A walking image sequence under different viewing angles is collected, and the binary walking profile image of each frame is extracted; from all the binary gait profile images within a gait cycle, two different types of gait representations, the global gait energy map G and the local spatiotemporal sequence L, are extracted to represent the global human gait motion and the gait motion of each local part of the human body. The global gait energy map represents the overall motion characteristics of the gait, that is, the spatiotemporal changes of the overall gait profile shape. The local spatiotemporal sequence describes the spatiotemporal characteristics of each local part of the human body, that is, the spatial changes of the gait profile shapes of each local part of the human body over time.

[0012] The conventional method for extracting the global gait energy map is as follows: in a walking image sequence, all binary walking contour images in T consecutive frames are averaged point by point in the time dimension to obtain the global gait energy map G, where T represents the number of binary walking contour images contained in a gait cycle.

[0013] The extraction method of the local space-time sequence is as follows: in a gait sequence, all binary walking profile images in consecutive T frames are vertically and evenly divided into m sub-profile images in the spatial dimension, and the sub-profile images at the same position are vertically spliced ​​in time sequence to obtain m local space-time images, that is, the local space-time sequence L is obtained. The mathematical expression of the extraction method is:

[0014] (1-1) Get all binary walking contour images in frame T:

[0015] S={t i |i=1,2,...,T} (1)

[0016] (1-2) Divide the binary walking contour image vertically and evenly into m sub-contour images:

[0017] t i,j ,i=1,2,...,T; j=1,2,...,m; (2)

[0018] where t ij is the jth sub-contour image of the i-th frame;

[0019] (1-3) Vertically splice the sub-contour graphs at the same position in time order to obtain the local spatiotemporal graph of the j-th sub-contour graph in a gait cycle:

[0020] L j =[t ·,j ] VC (3)

[0021] in[·] VC represents the splicing operation along the vertical dimension, t ·,jrepresents the jth sub-contour image of all binary walking contour images in a gait cycle, L j is the local spatiotemporal graph of the jth sub-contour graph in a gait cycle;

[0022] (1-4) Obtain local space-time sequence:

[0023] L={L j |j=1,2,...,m} (4).

[0024] The extraction process of local spatiotemporal series is as follows: Figure 2 shown.

[0025] Step 2: Implement nonlinear mapping from gait representation to deep discriminative features;

[0026] According to the global gait energy map extracted in step 1, a global feature learning network is designed based on dense connection layers and attention transition layers, and the global nonlinear mapping is defined and parameterized using the global feature learning network. A nonlinear mapping from global gait representation to global deep features is realized. Among them, the dense connection layer is responsible for the main feature extraction work, realizing the mutual and direct connection of each network layer within the layer. The spatiotemporal attention module is used in the transition layer of the global feature learning network, emphasizing useful features while reducing the influence of useless information, and is used to adjust the size of the feature map.

[0027] According to the local spatiotemporal sequence gait representation extracted in step 1, a local feature learning network is built using a convolutional layer, and the local feature learning network is used to define and parameterize the local nonlinear mapping Implement nonlinear mapping from local gait representation to local deep features.

[0028] Step 3: Network training;

[0029] The perspective hierarchical Euclidean distance between the deep features of gait representation is defined as the metric distance between gait representations. The metric distance function uses the amplitude of the perspective difference between gait representations to weight the difference between the deep feature vectors of gait representations. The perspective triplet loss function is used to supervise the training process of the feature learning network. The loss function considers that the gait profile image is jointly affected by the individual gait motion characteristics and the perspective. The deep features, identity and perspective information of the training samples are used in combination with the perspective hierarchical Euclidean distance to calculate the loss value. In the gait training data, the online most difficult samples and perspective difficult samples are used for mining to increase the triplets available for each training. Finally, the perspective triplet loss function is minimized so that the learned nonlinear mapping can map the gait representation to a specific space. The metric distance between the gait representations from the same pedestrian in this specific space is smaller than the metric distance between the gait representations from different pedestrians, and the metric distance between the gait representations is related to their deep feature differences and perspective differences.

[0030] The view-level Euclidean distance is designed based on the difference in the deep feature vectors between gait representations and the magnitude of the view-level difference, so that the similarity between gait representations does not only depend on the difference in deep features, but also takes into account the view-level difference between gaits by weighting the difference between their deep feature vectors by the magnitude of the view-level difference between gait representations. The view-level Euclidean distance is calculated as follows:

[0031] (3-1) Calculate the squared Euclidean distance between the depth features of two gaits:

[0032]

[0033] Where D is the squared Euclidean distance, F is the global or local nonlinear mapping, x A and x B are two different gait representation inputs of the same type;

[0034] (3-2) Based on the fact that human gait is a symmetrical movement pattern, the difference in viewing angle is limited to the range of [0,90°]; the limitation is achieved by keeping v A or v B No change, modify v B or v A , limiting the difference in viewing angle to 0°≤|v A -v B |≤90°, which can be expressed by the following formula:

[0035] v A =v A (6)

[0036]

[0037] Among them, v A and v B are the gait representation input x A and x B The viewing angle;

[0038] (3-3) According to the perspective difference between the gait representations, the weighted coefficient v(v A ,v B ), weighting coefficient ω(v A ,v B ) follows the criterion: the greater the difference in perspective between the gait representations, the greater the weighting coefficient of the metric distance between the gait representations, and the criterion is expressed by the following formula:

[0039] ω(v A ,v B )∝|vA -v B | (8)

[0040] (3-4) Calculate the Euclidean distance of the perspective hierarchy:

[0041]

[0042] Where D * is the Euclidean distance of the perspective hierarchy; I A =(F(x A ),v A ) and I B =(F(x B ),v B ) indicates that a gait sample is composed of the depth features and perspective information of the gait representation; Indicates v A With v B The number of viewing angle intervals between them; μ>0 indicates the number of defined viewing angles; is a weighted term, which uses the difference in perspective between gait representations to weight the distance between their depth feature vectors; in addition to being affected by the intrinsic gait motion characteristics of the human body, the gait appearance of the human body also changes with the change of perspective, and this change is hierarchical, that is, the greater the difference in perspective between gaits, the greater the difference in gait appearance. Therefore, when the parameter μ remains unchanged, the weighted term is proportional to the weighting coefficient and the difference in perspective:

[0043]

[0044] The metric learning goal is that the metric distance between gait representations from the same pedestrian is smaller than the metric distance between gait representations from different pedestrians, where the metric distance between gait representations is related to their deep feature differences and viewpoint differences. The mathematical expression of the metric learning goal is as follows:

[0045] D * ((F(x a ), v a ), (F(x p ), v p ))<D * ((F(x a ), v a ), (F(x n ), v n )) (11)

[0046] D * ((F(x a ), v a ), (F(x p ), v p ))∝D(F(xa ), F(x p )) (12)

[0047] D * ((F(x a ), v a ), (F(x n ), v n ))∝D(F(x a ), F(x n )) (13)

[0048]

[0049]

[0050] Among them, I a =(F(x a ), v a ) is the anchor sample, I p =(F(x p ), v p ) is related to I a Positive samples of the same identity, I n =(F(x n ), v n ) is related to I a Negative samples of different identity categories.

[0051] The loss function used in the training process of the supervised global and local feature learning network is a perspective triplet loss function; the perspective triplet loss considers that the gait profile image is jointly affected by the individual gait motion characteristics and the gait perspective, and the calculated loss value is related to the depth feature and the perspective angle of the gait representation; the perspective triplet loss uses the identity and perspective information of the gait training samples at the same time to mine the most difficult samples and perspective difficult samples online, and increase the triplet available for each training; the perspective difficult samples are samples of the same category as the most difficult samples and the number of perspective intervals is less than or equal to 1; the perspective triplet loss uses the depth feature, identity and perspective information of the training samples and combines the perspective hierarchical Euclidean distance to calculate the loss value; the method for calculating the loss value using the perspective triplet loss function is as follows:

[0052] (31) Randomly extract P × K gait training samples online, where P represents the number of people and K represents the number of samples for each person;

[0053] (32) Select any sample as anchor sample I a , perform online most difficult sample mining to obtain the most difficult positive sample I p and the most difficult negative sample I n ;

[0054] (33) Perform online perspective difficulty sample mining and select perspective difficulty positive samples and perspective difficulty positive sample sets:

[0055]

[0056]

[0057] in is a positive sample with difficult viewing angles, I vp Positive sample set with difficult perspective, Indicates that v p and The number of viewing angle intervals between them, R is the number of positive samples with viewing angle difficulty;

[0058] (34) Perform online perspective difficulty sample mining and select perspective difficulty negative samples and perspective difficulty negative sample sets:

[0059]

[0060]

[0061] in is a negative sample with difficult viewing angle, I vn View difficulty negative sample set, Indicates that v n and The number of viewing angle intervals between them, Q is the number of negative samples with viewing angle difficulties;

[0062] (35) The loss value is calculated using the sample’s deep features, identity, and perspective information combined with the perspective hierarchical Euclidean distance:

[0063]

[0064] in[·] + =max(0,·), α1 is the balanced positive sample pair And negative sample pairs The threshold of α2 is the positive sample pair of balanced perspective And perspective negative sample pairs The threshold value.

[0065] Step 4: Feature fusion and gait recognition;

[0066] The global deep discriminative features and the local deep discriminative features are fused as the final human gait feature representation; the fusion process is represented by the following formula:

[0067]

[0068] Where [·,·] is the concatenation operation of one-dimensional feature vectors, k1 is the weight of the global feature, and k2 is the weight of the local feature.

[0069] The above fusion features are used to characterize the human gait under different viewing angles. Since it is unrealistic to predict the viewing angle of pedestrian gait in an actual monitoring environment, it is assumed that the viewing angle of the gait to be identified is unknown in the recognition stage. By calculating the Euclidean distance between the fusion features of the gait to be identified and the fusion features of the gait in the database, the gait sample with the smallest Euclidean distance to the gait sample to be identified in the database is returned as the identity recognition result of the gait to be identified, thereby realizing cross-view gait recognition.

[0070] Compared with other existing methods, the present invention has the following advantages:

[0071] 1. Compared with the traditional method based on manually extracting view-independent features, the present invention uses deep neural networks to parameterize global and local nonlinear mappings, maps global and local gait representations into global and local deep discriminative features and represents them as human gait feature representations, realizes automatic feature learning to simplify the gait recognition process.

[0072] 2. Extract different types of deep features as clues to describe human gait, enhance the expressiveness of features and reduce their sensitivity to viewing angles, and further integrate global and local deep features. Multi-faceted and sufficient feature information will greatly help the realization of cross-view gait recognition.

[0073] 3. Compared with other deep learning and metric learning methods, the present invention uses the hierarchical Euclidean distance of perspective, and uses the amplitude of perspective difference to weight the differences between gaits, thereby enhancing the robustness of the gait recognition model to cope with perspective changes. In addition, the present invention uses the perspective triplet loss function to supervise the training process of the feature learning network, taking into account the intrinsic motion characteristics and perspective information of the gait, and using the deep features, identity and perspective information of the training samples to calculate the loss value, which helps to balance the gait differences caused by differences in individual gait motion characteristics and perspective differences. At the same time, using perspective difficult samples, the number of available triplets for each training is increased, which helps to train a network with higher recognition accuracy and more reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a flow chart of the cross-view gait recognition method based on deep feature fusion and view triplet loss proposed in the present invention.

[0075] Figure 2 This is a schematic diagram of the extraction process of local spatiotemporal sequences, where m=8.

[0076] Figure 3 It is a schematic diagram of the structure of the global feature learning network proposed in the present invention.

[0077] Figure 4 Schematic diagram of the structure of the local feature learning network proposed in the present invention, in which r1=3.

[0078] Figure 5 2 is a schematic diagram of a two-dimensional visualization of global depth discriminative features in an embodiment.

[0079] Figure 6 3 is a two-dimensional visualization diagram of local depth discriminative features in an embodiment. DETAILED DESCRIPTION

[0080] The following content further describes the present invention in detail with reference to the embodiments and drawings, but the implementation manner of the present invention is not limited to this specific situation.

[0081] Example

[0082] like Figure 1 As shown, a cross-view gait recognition method based on feature fusion and view triplet loss includes the following steps:

[0083] Step 1: Collect walking image sequences from different perspectives and extract global and local gait representations;

[0084] The gait database used in this embodiment is the CASIA-B gait database. The database contains 124 people in total, each of whom has 11 different viewing angles (0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 144°, 162°, 180°), and each person has 10 gait sequences at each angle, of which 6 are gait sequences under normal conditions (referred to as #NM01-06), 2 are gait sequences under the condition of carrying a backpack (referred to as #BG01-02), and the remaining 2 are gait sequences under the condition of wearing a coat (referred to as #CL01-02), so there are 124×11×10=13640 gait sequences in total. This embodiment selects all gait sequences under normal conditions as research objects to study gait recognition under changes in viewing angles. All gait sequences of the first 74 pedestrians under normal conditions are used as training sets, and all gait sequences of the remaining 50 pedestrians under normal conditions are used for testing. In the test phase, the first 4 gait sequences under normal conditions (#NM01-04) are used to build a database, and the remaining 2 gait sequences under normal conditions (#NM05-06) are included in the test set as gaits to be identified. The present invention sequentially performs background subtraction and morphological processing on the gait video used, and intercepts the circumscribed rectangular partial image of the human gait contour in each frame image, and performs size normalization and alignment processing based on the contour center to obtain a binary walking contour image.

[0085] The present invention extracts two different types of gait representations, namely, a global gait energy graph G and a local spatiotemporal sequence L, from all binary walking profile image sequences used.

[0086] In a walking image sequence, all binary walking profile images in T consecutive frames are averaged point by point in the time dimension to obtain a global gait energy graph G, where T represents the number of binary walking profile images contained in a gait cycle. In this embodiment, T=26.

[0087] like Figure 2 As shown in FIG. 1 , in a gait sequence, all binary walking profile images in consecutive T frames are vertically and evenly divided into m sub-profile images (in this embodiment, m=8) in the spatial dimension, and the sub-profile images at the same position are vertically spliced ​​in time sequence to obtain m local space-time images, that is, the local space-time sequence L is obtained. The mathematical expression of the extraction method is:

[0088] (1-1) Get all binary walking contour images in frame T:

[0089] S={t i |i=1,2,...,T} (1)

[0090] (1-2) Divide the binary walking contour image vertically and evenly into m sub-contour images:

[0091] t i，j ,i=1,2,...,T; j=1,2,...,m; (2)

[0092] where t ij is the jth sub-contour image of the i-th frame;

[0093] (1-3) Vertically splice the sub-contour graphs at the same position in time order to obtain the local spatiotemporal graph of the j-th sub-contour graph in a gait cycle:

[0094] L j =[t. ，j ] VC (3)

[0095] in[·] VC represents the splicing operation along the vertical dimension, t ，j represents the jth sub-contour image of all binary walking contour images in a gait cycle, L j is the local spatiotemporal graph of the jth sub-contour graph in a gait cycle;

[0096] (1-4) Obtain local space-time sequence:

[0097] L={L j|j=1,2,...,m} (4).

[0098] Step 2: Implement nonlinear mapping from gait representation to deep discriminative features;

[0099] Based on the global gait energy map extracted in step 1, a global feature learning network is used to define and parameterize the global nonlinear mapping Implement nonlinear mapping from global gait representation to global deep features.

[0100] The structural diagram of the global feature learning network is as follows Figure 3 As shown in the figure, the deep feature embedding module is the backbone network of the global feature learning network, which consists of several densely connected layers and attention transition layers, and is responsible for the main feature extraction work. The densely connected layers realize the mutual and direct connection of each layer. The channel attention layer and the spatial attention layer are used in the attention transition layer of the global feature learning network to emphasize useful features while reducing the influence of useless information, and to adjust the size of the feature map.

[0101] The global feature learning network includes an input module, a deep feature embedding module and a downsampling module connected in sequence;

[0102] The input module includes a 7×7 convolution layer, a batch normalization layer, a linear rectification function and a 3×3 pooling layer connected in sequence;

[0103] The deep feature embedding module includes a dense connection layer I, an attention transition layer I, a dense connection layer II, an attention transition layer II and a dense connection layer III connected in sequence; the dense connection layer I, the dense connection layer II and the dense connection layer III are respectively composed of 1, 3 and 6 composite function layers; the composite function layer includes a batch normalization layer, a linear rectification function, a 1×1 convolution layer, a batch normalization layer, a linear rectification function and a 3×3 convolution layer connected in sequence, and its output is k0 feature maps, k0 is a pre-set positive integer; the attention transition layer I and the attention transition layer II have the same structure, including a transition layer, a channel attention layer, a spatial attention layer and a 2×2 average pooling layer connected in sequence, wherein the input of the channel attention layer is the output of the transition layer, and the output of the spatial attention layer is The input is the dot product result of the input and output of the channel attention layer, and the input of the 2×2 average pooling layer is the dot product result of the input and output of the channel attention layer; the transition layer includes a batch normalization layer, a linear rectification function and a 1×1 convolution layer connected in sequence; the channel attention layer includes a global maximum pooling layer, a one-dimensional convolution layer with a convolution kernel dimension of 3 and a Sigmoid function connected in sequence, wherein the global maximum pooling layer is performed in the spatial dimension; the spatial attention module includes a global pooling layer, a 3×3 convolution layer and a Sigmoid function connected in sequence, wherein the global pooling layer is performed in the channel dimension; the global pooling layer includes a global maximum pooling layer and a global average pooling layer connected in parallel, and the concatenation of the outputs of the two in the channel dimension is used as the output;

[0104] The downsampling module includes a 3×3 maximum pooling layer;

[0105] The output of the global feature learning network is the output of the downsampling module and is expanded into a one-dimensional feature vector as a global deep feature.

[0106] Based on the local spatiotemporal sequence gait representation extracted in step 1, a local feature learning network is used to define and parameterize the local nonlinear mapping Implement nonlinear mapping from local gait representation to local deep features.

[0107] The local feature learning network includes m parameter-sharing local feature learning branch networks, wherein the input of the local feature learning network is a local spatiotemporal sequence, and the inputs of the m local feature learning branch networks correspond to m local spatiotemporal graphs of the local spatiotemporal sequence respectively; the local feature learning branch network includes a local embedding module I, a local embedding module II, a local embedding module III and an output module connected in sequence;

[0108] The local embedding module I, local embedding module II, and local embedding module III all include a local convolution layer I, a local convolution layer II, and a local maximum pooling layer connected in sequence; the local convolution module I of the local embedding module I, local embedding module II, and local embedding module III is respectively composed of r1, r2, and r3 parameter-shared convolution layers I; the local convolution module II of the local embedding module I, local embedding module II, and local embedding module III is respectively composed of r1, r2, and r3 parameter-shared convolution layers II; the local maximum pooling layer of the local embedding module I, local embedding module II, and local embedding module III is respectively composed of r1, r2, and r3 3×3 maximum pooling layers; the local convolution layer I includes a batch normalization layer, a linear rectification function, and a 1×1 convolution layer connected in sequence; the local convolution layer II includes a batch normalization layer, a linear rectification function, and a 3×3 convolution layer connected in sequence;

[0109] The output module includes a batch normalization layer, a linear rectification function, a 3×3 convolution layer, a batch normalization layer, a linear rectification function, a 3×3 convolution layer and a 3×3 maximum pooling layer connected in sequence;

[0110] The output of the local feature learning branch network is the output of the output module, and is expanded into a one-dimensional feature vector; the output of the local feature learning network is the concatenation of the outputs of m local feature learning branch networks, and is used as a local deep feature.

[0111] In this embodiment, r1=26, r2=13, and r3=4.

[0112] Step 3: Network training;

[0113] The perspective hierarchical Euclidean distance between the deep features of gait representation is defined as the metric distance between gait representations. The metric distance function uses the amplitude of the perspective difference between gait representations to weight the difference between the deep feature vectors of gait representations. The perspective triplet loss function is used to supervise the training process of the feature learning network. The loss function considers that the gait profile image is jointly affected by the individual gait motion characteristics and the perspective. The deep features, identity and perspective information of the training samples are used in combination with the perspective hierarchical Euclidean distance to calculate the loss value. In the gait training data, the online most difficult samples and perspective difficult samples are used for mining to increase the triplets available for each training. Finally, the perspective triplet loss function is minimized so that the learned nonlinear mapping can map the gait representation to a specific space. The metric distance between the gait representations from the same pedestrian in this specific space is smaller than the metric distance between the gait representations from different pedestrians, and the metric distance between the gait representations is related to their deep feature differences and perspective differences.

[0114] The gait data used for training are all the gait sequences under normal conditions of the first 74 individuals in the CASIA-B gait database.

[0115] The calculation method of the perspective hierarchical Euclidean distance is as follows:

[0116] (3-1) Calculate the squared Euclidean distance between the deep features of two gait representations:

[0117]

[0118] Where D is the squared Euclidean distance, F is the global or local nonlinear mapping, x A and x B are two different gait representation inputs of the same type;

[0119] (3-2) Based on the fact that human gait is a symmetrical movement pattern, the difference in viewing angle is limited to the range of [0,90°]; the limitation is achieved by keeping v A or v B No change, modify v B or v A , limiting the difference in viewing angle to 0°≤|v A -v B |≤90°, which can be expressed by the following formula:

[0120] v A =v A (6)

[0121]

[0122] Among them, v A and v B are the gait representation input x A and x B The viewing angle;

[0123] (3-3) According to the perspective difference between the gait representations, the weighted coefficient ω(v A ,v B ), weighting coefficient ω(v A ,v B ) follows the criterion: the greater the difference in perspective between the gait representations, the greater the weighting coefficient of the metric distance between the gait representations, and the criterion is expressed by the following formula:

[0124] ω(v A ,v B )∝|v A -v B | (8)

[0125] (3-4) Calculate the Euclidean distance of the perspective hierarchy:

[0126]

[0127] Where D * is the Euclidean distance of the perspective hierarchy; I A =(F(x A ),v A ) and I B =(F(x B ),v B ) indicates that a gait sample is composed of the depth features and perspective information of the gait representation; Indicates v A With v B The number of viewing angle intervals between them; μ>0 indicates the number of defined viewing angles; is a weighted term, which uses the difference in perspective between gait representations to weight the distance between their depth feature vectors; in addition to being affected by the intrinsic gait motion characteristics of the human body, the gait appearance of the human body also changes with the change of perspective, and this change is hierarchical, that is, the greater the difference in perspective between gaits, the greater the difference in gait appearance. Therefore, when the parameter μ remains unchanged, the weighted term is proportional to the weighting coefficient and the difference in perspective:

[0128]

[0129] Furthermore, the metric learning goal is: the metric distance between gait representations from the same pedestrian in this specific space is smaller than the metric distance between gait representations from different pedestrians, where the metric distance between gait representations is related to their depth feature differences and perspective differences, that is, the greater the depth feature difference between gait representations, the greater the metric distance; at the same time, the greater the perspective angle difference between gait representations, the greater the metric distance. The mathematical expression of the metric learning goal is as follows:

[0130] D * ((F(x a ),v a ),(F(x p ),v p )) <D * ((F(x a ),v a ),(F(x n ),v n )) (11)

[0131] D * ((F(x a ),v a ),(F(x p ),v p ))∝D(F(xa ),F(x p )) (12)

[0132] D * ((F(x a ), v a ), (F(x n ), v n ))∝D(F(x a ), F(x n )) (13)

[0133]

[0134]

[0135] Among them, I a =(F(x a ), v a ) is the anchor sample, I p =(F(x p ), v p ) is related to I a Positive samples of the same identity, I n =(F(x n ), v n ) is related to I a Negative samples of different identity categories.

[0136] The network is trained by minimizing the view triplet loss function so that the learned nonlinear mapping can meet the metric learning objective. The method for calculating the loss value using the view triplet loss function is as follows:

[0137] (31) Randomly extract P×K gait training samples online, where P represents the number of people and K represents the number of samples for each person. In this embodiment, for the training of the global feature learning network, P×K=4×32; for the training of the local feature learning network, P×K=4×8.

[0138] (32) Select any sample as anchor sample I a , perform online most difficult sample mining to obtain the most difficult positive sample I p and the most difficult negative sample I n .

[0139] (33) Perform online perspective difficulty sample mining and select perspective difficulty positive samples and perspective difficulty positive sample sets:

[0140]

[0141]

[0142] in is a positive sample with difficult viewing angles, I vp Positive sample set with difficult perspective, Indicates that v p and The number of viewing angle intervals between them, R is the number of positive samples with viewing angle difficulty;

[0143] (34) Perform online perspective difficulty sample mining and select perspective difficulty negative samples and perspective difficulty negative sample sets:

[0144]

[0145]

[0146] in is a negative sample with difficult viewing angle, I vn View difficulty negative sample set, Indicates that v n and The number of viewing angle intervals between them, Q is the number of negative samples with viewing angle difficulties;

[0147] (35) The loss value is calculated using the sample’s deep features, identity, and perspective information combined with the perspective hierarchical Euclidean distance:

[0148]

[0149] in[·] + =max(0,·), α1 is the balanced positive sample pair And negative sample pairs The threshold of α2 is the positive sample pair of balanced perspective And perspective negative sample pairs The threshold value.

[0150] Step 4: Feature fusion and gait recognition;

[0151] The global deep discriminative features and local deep discriminative features are fused as the final human gait feature representation, and the fusion process is expressed by the following formula:

[0152]

[0153] Where [·,·] is the concatenation operation of the one-dimensional feature vector, k1 is the weight of the global feature, and k2 is the weight of the local feature. In this embodiment, because the dimension ratio of the global feature to the local feature is about 1:2, for the sake of fairness, the feature weights are set to: k1 = 1.0, k2 = 0.5.

[0154] The above-mentioned fusion features are used to characterize the human gait under different viewing angles. Since it is unrealistic to predict the viewing angle of pedestrian gait in an actual monitoring environment, the method of the present invention considers that the viewing angle of the gait to be identified in the recognition stage is unknown. The method of the present invention calculates the Euclidean distance between the fusion features of the gait to be identified and the fusion features of the gait in the database, and returns the gait sample with the smallest Euclidean distance to the gait sample to be identified in the database as the identity recognition result of the gait to be identified, thereby realizing cross-view gait recognition. Among them, all gait sequences under normal conditions of the last 50 pedestrians in the CASIA-B gait database are used as tests. In the test stage, the first 4 gait sequences under normal conditions (#NM01-04) are used to build the registration set database, and the remaining 2 gait sequences under normal conditions (#NM05-06) are included in the test set as gaits to be identified.

[0155] As shown in Table 1, the recognition accuracy (Rank-1) results of the cross-view gait recognition experiment in the present invention are shown. The experimental data illustrate the feasibility and superiority of the present invention for cross-view recognition. From the results in Table 1, it can be observed that the method of the present invention can achieve high recognition accuracy under multiple experimental viewing angles and at the average level. Even when the viewing angle changes greatly, it still maintains a high accurate recognition rate and has high robustness to viewing angle changes.

[0156] Table 1

[0157]

[0158] In this example, three pedestrians are randomly selected from the CASIA-B gait database, and their gait sequences under normal conditions at viewing angles of 18°, 54°, 90°, and 126° are randomly selected. Their global depth discriminant features and local depth discriminant features are extracted and visualized. Figure 5 and Figure 6 As shown, both the global depth discriminative features and the local depth discriminative features have strong feature expression and discrimination capabilities, and are robust to identity changes and perspective changes.

[0159] The above embodiments are relatively preferred implementations of the present invention, but the implementations of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A cross-view gait recognition method based on feature fusion and view triplet loss, characterized in that: The following steps are included: Step 1: Collect walking image sequences from different perspectives and extract global and local gait representations; Collect walking image sequences from different perspectives and extract the binary walking profile image of each frame; based on all binary walking profile images in a gait cycle, extract two different types of gait representations, the global gait energy graph G and the local spatiotemporal sequence L, to represent the global human gait motion and the gait motion of each local part of the human body; Step 2: Implement nonlinear mapping from gait representation to deep discriminative features; Based on the global gait representation extracted in step 1, a global feature learning network is used to define and parameterize the global nonlinear mapping Realize the nonlinear mapping from global gait representation to global deep features; at the same time, based on the local gait representation extracted in step 1, use the local feature learning network to define and parameterize the local nonlinear mapping Implement nonlinear mapping from local gait representation to local deep features; Step 3: Network training; The perspective hierarchical Euclidean distance between the deep features of gait representation is defined as the metric distance between gait representations. The metric distance function uses the amplitude of the perspective difference between gait representations to weight the difference between the deep feature vectors of gait representations. The perspective triplet loss function is used to supervise the training process of the global feature learning network and the local feature learning network. The loss function considers that the gait profile image is jointly affected by the individual gait motion characteristics and the perspective. The deep features, identity and perspective information of the training samples are used in combination with the perspective hierarchical Euclidean distance to calculate the loss value. In the gait training data, the most difficult samples and perspective difficult samples are mined online to increase the triplets available for each training. Finally, the perspective triplet loss function is minimized so that the learned nonlinear mapping can map the gait representation to a specific space. The metric distance between the gait representations from the same pedestrian in the specific space is smaller than the metric distance between the gait representations from different pedestrians, wherein the metric distance between the gait representations is related to their deep feature differences and perspective differences. Step 4: Feature fusion and gait recognition; Fusion of two different types of deep discriminative features, namely global deep discriminative features and local deep discriminative features As the final human gait feature representation; The Euclidean distance between the gait fusion feature to be identified and the gait fusion feature in the database is calculated, and the gait sample in the database with the smallest Euclidean distance to the gait sample to be identified is returned as the identity recognition result of the gait to be identified, thereby realizing cross-view gait recognition.

2. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 1, characterized in that: In step 1, a local space-time sequence is extracted as a local gait representation; the method for extracting the local space-time sequence is as follows: in a gait sequence, all binary walking profile images in continuous T frames are vertically and evenly divided into m sub-profile images in the spatial dimension, and the sub-profile images at the same position are vertically spliced ​​in time sequence to obtain m local space-time images, that is, the local space-time sequence L is obtained; the mathematical expression of the method for extracting the local space-time sequence is: (1-1) Get all binary walking contour images in T consecutive frames: S={t i |i=1,2,...,T} (1) Where T is the number of binary walking profile images contained in a gait cycle; (1-2) Divide the binary walking contour image vertically and evenly into m sub-contour images: t i,j ,i=1,2,...,T;j=1,2,...,m; (2) where t ij is the jth sub-contour image of the i-th frame; (1-3) Vertically splice the sub-contour graphs at the same position in time order to obtain the local spatiotemporal graph of the j-th sub-contour graph in a gait cycle: L j =[t ·,j ] VC (3) in[·] VC represents the splicing operation along the vertical dimension, t ·,j represents the jth sub-contour image of all binary walking contour images in a gait cycle, L j is the local spatiotemporal graph of the jth sub-contour graph in a gait cycle; (1-4) Obtain local space-time sequence: L={L j |j=1,2,...,m} (4)。 3. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 1 or 2, characterized in that: In step 2, the global nonlinear mapping Able to realize global gait representation G to global deep features A nonlinear mapping, wherein the mapping is described by the network parameter θ1 of the global feature learning network; the local nonlinear mapping Able to realize local spatiotemporal sequence gait representation L to local deep features A nonlinear mapping of θ to θ, where the mapping is described by the network parameters θ2 of the local feature learning network.

4. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 3 is characterized in that: In step 2, the global feature learning network includes an input module, a deep feature embedding module and a downsampling module connected in sequence; The input module includes a 7×7 convolution layer, a batch normalization layer, a linear rectification function and a 3×3 pooling layer connected in sequence; The deep feature embedding module includes a dense connection layer I, an attention transition layer I, a dense connection layer II, an attention transition layer II and a dense connection layer III which are connected in sequence; the dense connection layer I, the dense connection layer II and the dense connection layer III are respectively composed of 1, 3 and 6 composite function layers; the composite function layer includes a batch normalization layer, a linear rectification function, a 1×1 convolution layer, a batch normalization layer, a linear rectification function and a 3×3 convolution layer which are connected in sequence, and its output is k0 feature maps, where k0 is a pre-set positive integer; the attention transition layer I and the attention transition layer II have the same structure, and both include a transition layer, a channel attention layer, a spatial attention layer and a 2×2 average pooling layer which are connected in sequence, wherein the input of the channel attention layer is the output of the transition layer, and the input of the spatial attention layer is is the dot product result of the input and output of the channel attention layer, and the input of the 2×2 average pooling layer is the dot product result of the input and output of the spatial attention layer; the transition layer includes a batch normalization layer, a linear rectification function and a 1×1 convolution layer connected in sequence; the channel attention layer includes a global maximum pooling layer, a one-dimensional convolution layer with a convolution kernel of 3 and a Sigmoid function connected in sequence, wherein the global maximum pooling layer is performed in the spatial dimension; the spatial attention layer includes a global pooling layer, a 3×3 convolution layer and a Sigmoid function connected in sequence, wherein the global pooling layer is performed in the channel dimension; the global pooling layer includes a global maximum pooling layer and a global average pooling layer connected in parallel, and the concatenation of the outputs of the two in the channel dimension is used as the output; The downsampling module includes a 3×3 maximum pooling layer; The output of the global feature learning network is the output of the downsampling module and is expanded into a one-dimensional feature vector as a global deep feature.

5. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 1, 2 or 4, characterized in that: In step 2, the local feature learning network includes m parameter-sharing local feature learning branch networks, wherein the input of the local feature learning network is a local spatiotemporal sequence, and the inputs of the m local feature learning branch networks correspond to m local spatiotemporal graphs of the local spatiotemporal sequence respectively; the local feature learning branch network includes a local embedding module I, a local embedding module II, a local embedding module III and an output module connected in sequence; The local embedding module I, local embedding module II, and local embedding module III all include a local convolution layer I, a local convolution layer II, and a local maximum pooling layer connected in sequence; the local convolution module I of the local embedding module I, local embedding module II, and local embedding module III is respectively composed of r1, r2, and r3 parameter-shared convolution layers I; the local convolution module II of the local embedding module I, local embedding module II, and local embedding module III is respectively composed of r1, r2, and r3 parameter-shared convolution layers II; the local maximum pooling layer of the local embedding module I, local embedding module II, and local embedding module III is respectively composed of r1, r2, and r3 3×3 maximum pooling layers; the local convolution layer I includes a batch normalization layer, a linear rectification function, and a 1×1 convolution layer connected in sequence; the local convolution layer II includes a batch normalization layer, a linear rectification function, and a 3×3 convolution layer connected in sequence; The output module includes a batch normalization layer, a linear rectification function, a 3×3 convolution layer, a batch normalization layer, a linear rectification function, a 3×3 convolution layer and a 3×3 maximum pooling layer connected in sequence; The output of the local feature learning branch network is the output of the output module, and is expanded into a one-dimensional feature vector; the output of the local feature learning network is the concatenation of the outputs of m local feature learning branch networks, and is used as a local deep feature.

6. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 5, characterized in that: In step 3, the metric distance between the human gait representations refers to the perspective hierarchical Euclidean distance between the depth features of the gait representations; the perspective hierarchical Euclidean distance is designed according to the depth feature vector difference and the perspective difference amplitude between the gait representations, so that the similarity between the gaits does not only depend on the depth feature difference, but also considers the perspective difference between the gaits by weighting the difference between the depth feature vectors by the perspective difference amplitude between the gait representations; the perspective hierarchical Euclidean distance is calculated as follows: (3-1) Calculate the squared Euclidean distance between the deep features of two gait representations: where D is the squared Euclidean distance, F(·) is the global or local nonlinear mapping, and x A and x B are two different gait representation inputs of the same type; (3-2) Based on the fact that human gait is a symmetrical movement pattern, the difference in viewing angle is limited to the range of [0,90°]; the limitation is achieved by keeping v A or v B No change, modify v B or v A , limiting the difference in viewing angle to 0°≤|v A -v B |≤90° can be expressed by the following formula: v A =v A (6) Among them, v A and v B are the gait representation input x A and x B The viewing angle; (3-3) According to the perspective difference between gaits, the weighted coefficient ω(v A ,v B ), weighting coefficient ω(v A ,v B ) follows the criterion: the greater the difference in perspective between the gait representations, the greater the weighting coefficient of the metric distance between the gait representations, and the criterion is expressed by the following formula: ω(v A ,v B )∝|v A -v B | (8) (3-4) Calculate the Euclidean distance of the perspective hierarchy: Where D * is the Euclidean distance of the perspective hierarchy; I A =(F(x A ),v A ) and I B =(F(x B ),v B ) indicates that a gait sample is composed of the depth features and perspective information of the gait representation; Indicates v A With v B The number of viewing angle intervals between them; μ>0 indicates the number of defined viewing angles; is a weighted term, which uses the difference in perspective between gait representations to weight the distance between their depth feature vectors; in addition to being affected by the intrinsic gait motion characteristics of the human body, the gait appearance of the human body also changes with the change of perspective, and this change is hierarchical, that is, the greater the difference in perspective between gaits, the greater the difference in gait appearance. Therefore, when the parameter μ remains unchanged, the weighted term is proportional to the weighting coefficient and the difference in perspective:

7. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 6, characterized in that: In step 3, the metric learning goal is that the metric distance between gait representations from the same pedestrian is smaller than the metric distance between gait representations from different pedestrians, where the metric distance between gait representations is related to their deep feature differences and perspective differences. The mathematical expression of the metric learning goal is as follows: D * ((F(x a ),v a ),(F(x p ),v p ))<D * ((F(x a ),v a ),(F(x n ),v n )) (11) D * ((F(x a ),v a ),(F(x p ),v p ))∝D(F(x a ),F(x p )) (12) D * ((F(x a ),v a ),(F(x n ),v n ))∝D(F(x n ),F(x n )) (13) Among them, I a =(F(x a ),v a ) is the anchor sample, I p =(F(x p ),v p ) is related to I a Positive samples of the same identity, I n =(F(x n ),v n ) is related to I a Negative samples of different identity categories.

8. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 1 or 7, characterized in that: In step three, the loss function used in the supervised global and local feature learning network training process is a perspective triplet loss function; the perspective triplet loss considers that the gait profile image is jointly affected by the individual gait motion characteristics and the gait perspective, and the calculated loss value is related to the depth feature and the perspective angle represented by the gait; the perspective triplet loss uses the identity and perspective information of the gait training sample at the same time to mine the most difficult sample and the perspective difficult sample online, and increase the triplet available for each training; the perspective difficult sample is a sample of the same category as the most difficult sample and the number of perspective intervals is less than or equal to 1; the perspective triplet loss uses the depth feature, identity and perspective information of the training sample and combines the perspective hierarchical Euclidean distance to calculate the loss value; the method for calculating the loss value using the perspective triplet loss function is as follows: (31) Randomly extract P × K gait training samples online, where P represents the number of people and K represents the number of samples for each person; (32) Select any sample as anchor sample I a , perform online most difficult sample mining to obtain the most difficult positive sample I p and the most difficult negative sample I n ; (33) Perform online perspective difficulty sample mining and select perspective difficulty positive samples and perspective difficulty positive sample sets: in is a positive sample with difficult viewing angles, I vp Positive sample set with difficult perspective, Indicates that v p and The number of viewing angle intervals between them, R is the number of positive samples with viewing angle difficulty; (34) Perform online perspective difficulty sample mining and select perspective difficulty negative samples and perspective difficulty negative sample sets: in is a negative sample with difficult viewing angle, I vn View difficulty negative sample set, Indicates that v n and The number of viewing angle intervals between them, q is the number of negative samples with viewing angle difficulty; (35) The loss value is calculated using the sample’s deep features, identity, and perspective information combined with the perspective hierarchical Euclidean distance: in[·] + =max(0,·), α1 is the balanced positive sample pair And negative sample pairs The threshold of α2 is the positive sample pair of balanced perspective And perspective negative sample pairs The threshold value.

9. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 1, characterized in that: In step 4, the global depth feature and the local depth feature are fused as the final human gait feature representation; the fusion of the global depth feature and the local depth feature is represented by the following formula: Where [·,·] is the concatenation operation of one-dimensional feature vectors, k1 is the weight of the global feature, and k2 is the weight of the local feature.

10. The cross-view gait recognition method based on feature fusion and view triplet loss according to claim 1 or 9, characterized in that In step 4, the Euclidean distance between the fusion features of the gait to be identified and the fusion features of the gait in the database is calculated, and the gait sample in the database with the smallest Euclidean distance to the gait sample to be identified is returned as the identity recognition result of the gait to be identified, thereby realizing cross-view gait recognition.

Images