A pedestrian re-identification and gait recognition method based on spatiotemporal feature complementary fusion

Abstract

The application relates to a pedestrian re-identification and gait recognition method based on spatiotemporal feature complementary fusion, which comprises the following steps: acquiring a to-be-detected optical image and a to-be-detected pedestrian gait image in a to-be-detected video; inputting the to-be-detected optical image and the to-be-detected pedestrian gait image into a pedestrian re-identification network and a gait recognition network respectively, and outputting first pedestrian features and first pedestrian gait features; inputting the first pedestrian features output by the pedestrian re-identification network and the first pedestrian gait features output by the gait recognition network into a spatiotemporal feature complementary fusion module of a trained gait feature auxiliary pedestrian re-identification network for feature fusion, so as to obtain an initial pedestrian feature vector; connecting and normalizing the initial pedestrian feature vector and the first pedestrian features according to dimensions, so as to obtain a final pedestrian feature vector. The gait feature auxiliary pedestrian re-identification method improves the accuracy of a pedestrian re-identification algorithm model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a person re-identification and gait recognition method based on spatiotemporal feature complementary fusion. BACKGROUND

[0002] Person re-identification (Person ReID) is a technology that uses computer vision techniques to extract features from cross-camera personnel images, thereby matching personnel. Due to the important role of person re-identification in security applications, the research of person re-identification tasks is the core of security event-related research. In recent years, with the successful application of machine learning and deep learning in many fields, researchers have begun to try to apply them to person re-identification. The development of machine learning and later deep learning methods have opened up a wide range of advanced possibilities for person re-identification, which will bring safer environments for society and individuals in homes, offices, communities, bus stops, airports, etc., and benefit all mankind.

[0003] The person re-identification framework is similar to most deep learning-based image processing problems. First, a convolutional neural network (CNN) is used to extract features using representation learning to obtain a feature vector, and then a metric learning method is used for class judgment. Existing person re-identification algorithms can be mainly divided into two categories: one is image-based person re-identification algorithm, and the other is video-based person re-identification algorithm. The current image-based person re-identification algorithm extracts person features through a convolutional neural network, and then uses metric learning to re-identify the person. The video-based person re-identification algorithm uses deep neural networks to mine person appearance features and establish person spatiotemporal motion relationships. It relies on richer person video features and uses metric learning for person re-identification, which outperforms image person re-identification. Since multiple frames of images are introduced for person re-identification, noise is also introduced. Due to the problems of large image noise, complex background, and frame misalignment, the current video-based person re-identification algorithm still has defects. SUMMARY

[0004] In order to solve the above problems in the prior art, the present application provides a person re-identification and gait recognition method based on spatiotemporal feature complementary fusion. The technical problem to be solved by the present application is solved by the following technical scheme:

[0005] One embodiment of the present application provides a person re-identification and gait recognition method based on spatiotemporal feature complementary fusion, which comprises:

[0006] Step 1, obtaining a to-be-detected optical image in a to-be-detected video and a to-be-detected pedestrian gait image, wherein the to-be-detected optical image is an original image extracted from the to-be-detected video, and the to-be-detected pedestrian gait image is a contour image of a pedestrian obtained by using the to-be-detected optical image;

[0007] Step 2, inputting the to-be-detected optical image and the to-be-detected pedestrian gait image into a pedestrian re-identification network and a gait recognition network of a trained gait feature assisted pedestrian re-identification network respectively, and outputting the first pedestrian feature and the first pedestrian gait feature by the pedestrian re-identification network and the gait recognition network respectively;

[0008] Step 3, inputting the first pedestrian feature output by the pedestrian re-identification network and the first pedestrian gait feature output by the gait recognition network into a spatio-temporal feature complementary fusion module of the trained gait feature assisted pedestrian re-identification network for feature fusion, and obtaining an initial pedestrian feature vector;

[0009] Step 4, connecting and normalizing the initial pedestrian feature vector and the first pedestrian feature according to dimensions to obtain a final pedestrian feature vector.

[0010] In an embodiment of the present application, the step 1 comprises:

[0011] Step 1.1, sampling and frame extracting the to-be-detected video to obtain the to-be-detected optical image;

[0012] Step 1.2, processing the to-be-detected optical image by using a trained Mask RCNN network to obtain a pedestrian mask, and taking the pedestrian mask as the to-be-detected pedestrian gait image.

[0013] In an embodiment of the present application, the step 3 comprises:

[0014] Step 3.1, performing average pooling on the first pedestrian feature to obtain a second pedestrian feature, performing normalization on the second pedestrian feature to obtain a first pedestrian feature vector, performing time pooling and generalized average pooling on the first pedestrian gait feature to obtain a second pedestrian gait feature, and performing full connection layer and normalization processing on the second pedestrian gait feature to obtain a first pedestrian gait feature vector;

[0015] Step 3.2, mapping the first pedestrian feature vector and the first pedestrian gait feature vector to the same dimension by two first linear layers respectively to obtain a second pedestrian feature vector and a second pedestrian gait feature vector in the same dimension;

[0016] Step 3.3, connecting the second pedestrian feature vector and the second pedestrian gait feature vector according to dimensions to obtain a connected first dimension vector;

[0017] Step 3.4, the first dimension vector is processed by two second linear layers respectively, and two second dimension vectors are obtained after being activated by a RELU activation function;

[0018] Step 3.5, one of the second dimension vectors is mapped to a third dimension vector with N dimensions by using a third linear layer, and the other second dimension vector is mapped to an N*N dimension vector by using a fourth linear layer, and the size of the N*N dimension vector is rearranged to (N, N) as a dynamic weight;

[0019] Step 3.6, the third dimension vector is multiplied by the dynamic weight to obtain an N-dimensional fusion feature vector;

[0020] Step 3.7, the fifth linear layer is used to process the N-dimensional fusion feature vector, and the initial pedestrian feature vector is obtained by residual connection with the first pedestrian feature.

[0021] In an embodiment of the present application, the training method of the gait feature assisted pedestrian re-identification network comprises:

[0022] S1, a data set comprising a plurality of optical images and a plurality of gait images is obtained, the data set comprising a training set, a test set and a search set;

[0023] S2, the optical images and the gait images in the training set are input into the pedestrian re-identification network and the gait recognition network respectively, and a third pedestrian feature and a third pedestrian gait feature are obtained;

[0024] S3, the third pedestrian feature and the third pedestrian gait feature are fused by using the spatio-temporal feature complementary fusion module to obtain a pedestrian feature fusion vector;

[0025] S4, based on the cross-entropy loss function and the triplet loss function corresponding to the pedestrian feature fusion vector, an adaptive weighting method is used to obtain a total loss value;

[0026] S5, the parameters of the gait feature assisted pedestrian re-identification network are optimized by gradient back propagation of the total loss value until a preset condition is reached, and a trained gait feature assisted pedestrian re-identification network is obtained.

[0027] In an embodiment of the present application, before step S2, it further comprises:

[0028] The optical images in the training set are subjected to data enhancement by random flipping and / or random erasing, and the gait images in the training set are subjected to data enhancement by random flipping.

[0029] In an embodiment of the present application, the pedestrian re-identification network is a Resnet50-I3D network obtained by optimizing a ResNet50-2D network, specifically:

[0030] On the basis of the ResNet50-2D network, all 2D convolutional layers in the ResNet50-2D network are changed to 3D convolutional layers, all two-dimensional normalization layers are changed to three-dimensional normalization layers, and a one-dimensional time dimension is added.

[0031] In an embodiment of the present application, the gait recognition network includes a GaitGL network, and the gait recognition network divides pedestrian gait features into 4 blocks and flattens them into 512-dimensional outputs.

[0032] In an embodiment of the present application, after step S5, further comprising:

[0033] S6, processing the test set and the search set by using the trained gait feature assisted pedestrian re-identification network, inputting the search set into the trained gait feature assisted pedestrian re-identification network to match images of the same identity in the test set, to complete the test.

[0034] In an embodiment of the present application, step S6 includes:

[0035] S6.1, extracting pedestrian features of the test set and pedestrian features of the search set by using the trained gait feature assisted pedestrian re-identification network

[0036] S6.2, calculating the cosine similarity between the pedestrian features of the test set and the pedestrian features of the search set to obtain the identity of the most matched pedestrian image of the test set.

[0037] Compared with the prior art, the present application has the beneficial effects that:

[0038] The present application is based on a pedestrian re-identification algorithm and a gait recognition algorithm, and proposes a pedestrian re-identification and gait recognition method based on complementary fusion of space-time features. The present application designs a pedestrian re-identification and gait recognition double-branch network, constructs a space-time feature complementary fusion module to fuse pedestrian features output by a pedestrian re-identification network and pedestrian gait features output by a gait recognition network, thereby overcoming the problems of large image noise, complex background and frame misalignment faced by the pedestrian re-identification algorithm, and the problem of difficult large scene application caused by the lack of appearance information in gait recognition, and improving the accuracy of the pedestrian re-identification algorithm model.

[0039] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1is a flowchart of a pedestrian re-identification and gait recognition method based on spatiotemporal feature complementary fusion provided by an embodiment of the present application.

[0041] Figure 2 is a flowchart of another pedestrian re-identification and gait recognition method based on spatiotemporal feature complementary fusion provided by an embodiment of the present application.

[0042] Figure 3 is a video pedestrian re-identification network flowchart provided by an embodiment of the present application.

[0043] Figure 4 is a gait recognition network flowchart provided by an embodiment of the present application.

[0044] Figure 5 is a video pedestrian data preprocessing flowchart provided by an embodiment of the present application.

[0045] Figure 6 is a video pedestrian data preprocessing result diagram provided by an embodiment of the present application.

[0046] Figure 7 is a flowchart of a spatiotemporal feature complementary fusion module provided by an embodiment of the present application.

[0047] Figure 8 is a pedestrian re-identification test diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0049] Please refer to Figure 1 , Figure 1 is a pedestrian re-identification and gait recognition method based on spatiotemporal feature complementary fusion provided by an embodiment of the present application, Figure 2 is another pedestrian re-identification and gait recognition method based on spatiotemporal feature complementary fusion provided by an embodiment of the present application. The present application proposes a pedestrian re-identification and gait recognition method based on spatiotemporal feature complementary fusion, which comprises:

[0050] Step 1, obtaining a to-be-detected optical image and a to-be-detected pedestrian gait image in a to-be-detected video, wherein the to-be-detected optical image is an original image extracted from the to-be-detected video, and the to-be-detected pedestrian gait image is a contour image of a pedestrian obtained by using the to-be-detected optical image.

[0051] Gait recognition is a kind of identity recognition by using the behavior characteristics of people walking, which can extract gait features through walking posture to identify the specific identity of the object. Gait recognition is a kind of whole body recognition technology, which is determined by body shape, muscle strength, motor nerve sensitivity, walking posture, etc. Local changes will not affect the recognition result; environmental adaptation is stronger because gait recognition will not be disturbed by light, light source, ground environment, etc. As a task with image sequence as input, gait recognition is suitable for modeling by using deep learning method. At present, the mainstream method is to construct features that maintain the time and space information in the gait sequence by gait contour, and to learn discriminative feature representation by machine learning method. However, gait recognition does not have appearance information, and it is difficult to expand to complex world and large database for rapid recognition. Therefore, the embodiment will complement the advantages of video pedestrian re-identification and gait recognition, so as to obtain a method with higher pedestrian re-identification accuracy.

[0052] In one specific embodiment, step 1 can include steps 1.1-1.2, wherein:

[0053] Step 1.1, sampling and frame extraction are performed on the to-be-detected video to obtain a to-be-detected optical image.

[0054] Step 1.2, using the trained Mask RCNN network, the to-be-detected optical image is processed to obtain a pedestrian mask, and the pedestrian mask is used as a to-be-detected pedestrian gait image.

[0055] Specifically, a MASKRCNN network trained on the COCO dataset is initialized to detect characters and extract masks. Specifically, the MASKRCNN network is: replacing the RoI Pooling layer in the Faster RCNN network with the RoIAlign layer, and adding a parallel FCN layer (i.e. mask layer for instance segmentation) after the last layer of the Faster RCNN network, so as to obtain the Mask RCNN network. The Mask RCNN network is trained using the COCO dataset, which can effectively find the target and complete high-quality semantic segmentation.

[0056] Then, the optical image is input into the trained Mask RCNN network, the target detection layer (object detection) in the trained Mask RCNN network is used to extract the target information in all optical images, and the instance segmentation layer (instance segment) in the trained Mask RCNN network is used to extract the contour information of the pedestrians in all pictures in the original picture set, that is, the pedestrian gait image.

[0057] Step 2, please refer toFigure 3 and Figure 4 inputting the to-be-detected optical image and the to-be-detected pedestrian gait image into a pedestrian re-identification network and a gait recognition network of a trained gait feature assisted pedestrian re-identification network respectively, and outputting the first pedestrian feature and the first pedestrian gait feature by the pedestrian re-identification network and the gait recognition network respectively.

[0058] Specifically, the embodiment first constructs a gait feature assisted pedestrian re-identification network including a pedestrian re-identification network, a gait recognition network and a space-time feature complementary fusion module, the pedestrian re-identification network is used for extracting a pedestrian feature, the gait recognition network is used for extracting a pedestrian gait feature, and the space-time feature complementary fusion module is used for feature fusion of the pedestrian feature and the pedestrian gait feature.

[0059] In an optional embodiment, the pedestrian re-identification network is a Resnet50-I3D network obtained by optimizing a ResNet50-2D network, and specifically is:

[0060] On the basis of the ResNet50-2D network, all 2D convolution layers in the ResNet50-2D network are changed into 3D convolution layers, and all two-dimensional normalization layers are changed into three-dimensional normalization layers.

[0061] Specifically, in the feature extraction part of the video pedestrian image, a video pedestrian re-identification baseline method of Resnet50+average pooling (AveragePooling) is adopted. The ResNet50-2D network structure is divided into stem+4stage (stages), wherein the stem structure is simple and mainly performs preprocessing on the input data. The last 4 stages are composed of Bottleneck, which adopts a residual structure to embed different convolutions to extract features. Since the object of video pedestrian re-identification is a pedestrian image sequence, the embodiment is upgraded from Resnet50-2D to Resnet50-I3D based on the former, so as to extract the space-time and appearance features of the pedestrian image sequence. The changes are as follows: all 2D convolutions are changed to 3D convolutions, adding one-dimensional time dimension, all two-dimensional normalization (2D BN) is changed to three-dimensional normalization (3D BN), and the kernel size (kernel size) and stride of 3D convolution and max pooling (Max Pooling) are also reset. Specifically, the kernel size and stride of 3D convolution are changed from the original 3x3 / stride (2, 2) to 5x3x3 / stride (2, 2, 2), and max pooling is changed from the original 3x3 / stride (2, 2) to 1x3x3 / stride (2, 2, 2). In the specific implementation process of the video pedestrian re-identification baseline method, data preprocessing is performed through the convolution-normalization-activation function-max pooling (Conv-BN-RELU+Max Pooling) structure of the stem stage. Then, the extracted pedestrian image sequence feature map is obtained through the four stages of Resnet50-I3D.

[0062] In an optional embodiment, the gait recognition network includes a GaitGL network, and the gait recognition network divides pedestrian gait features into 4 blocks and flattens them into 512-dimensional outputs. Because only the gait information is input, when the GeM (generalized-mean) pooling is adopted, the feature map is subdivided into 64 parts, each part is used as an independent feature to calculate the cross-entropy and triplet loss, and more refined features can be learned. However, the embodiment combines the information of the original frame, so it is not necessary to subdivide so many, but only needs to divide the pedestrian gait features into 4 blocks and flatten them to obtain the overall features, which is convenient for combining with the pedestrian re-identification features.

[0063] Specifically, the gait recognition network selects the GaitGL network. The GaitGL network utilizes global visual information and local region details, develops a global and local feature extractor (GLFE) to aggregate and integrate global and local features. Local temporal aggregation (LTA) is adopted to further preserve spatial information by reducing the temporal resolution to obtain higher spatial resolution. The network first extracts shallow gait features from the original input pedestrian gait sequence using 3D convolution. Then, the local temporal aggregation operation is performed on the shallow gait features to aggregate temporal information and preserve more gait spatial information. Further, the global and local feature extractor is used to combine and integrate global and local information to obtain a pedestrian gait feature map.

[0064] In an optional embodiment, the pre-training weight initialization is performed on the pedestrian re-identification network and the gait recognition network respectively.

[0065] Specifically, for the pedestrian re-identification network, the ResNet50 network model pre-trained on the large image dataset ImageNet dataset is used for weight initialization. For the gait recognition network, the GaitGL network model pre-trained on the CASIA-B dataset is used for weight initialization.

[0066] Based on the pedestrian re-identification network and the gait recognition network constructed above, a training method of a gait feature assisted pedestrian re-identification network is provided, which specifically includes:

[0067] S1, obtaining a dataset including a plurality of optical images and a plurality of gait images, the dataset including a training set, a test set and a lookup set.

[0068] Specifically, referring to Figure 5 and Figure 6 , the original pedestrian video data for training is obtained, the original pedestrian video data is sampled and framed to obtain the required optical images, the obtained optical images are extracted by using the trained MASKRCNN network to obtain the required gait images, and the obtained optical image and gait image data are divided to obtain the training set, the test set and the lookup set. According to the rules, the number of persons in the training set, the test set and the lookup set is equal or similar. The number of person images of the training set and the test set is more, while the lookup set only needs to provide a small number of person images for lookup, so the number of images of the lookup set can be set to be small. The identities of the person images of the training set and the latter two can be different, while the identities of the person images of the test set and the lookup set must be the same, or the number of person images of the test set is more than that of the lookup set, otherwise there may be an empty matching problem.

[0069] In the training process, in order to improve the robustness of the trained model, data augmentation methods can be used on the training set data. However, any data augmentation cannot be used in the test reasoning stage. The size of the optical image can be set to 256*128, and the size of the gait image can be set to 64*44. At the same time, during training, the optical image adopts data augmentation methods such as random flipping and / or random erasing, and the gait image only adopts random flipping due to the simplicity of the data.

[0070] That is, the pictures in the training set are preprocessed according to the preset data augmentation method, and the size of all pictures in the training set is set to a predetermined size, i.e., 256*128*3 for the pedestrian re-identification network and 64*44*1 for the gait network.

[0071] S2, the optical image and the gait image in the training set are input into the pedestrian re-identification network and the gait recognition network respectively to obtain the third pedestrian feature and the third pedestrian gait feature.

[0072] S3, the third pedestrian feature and the third pedestrian gait feature are fused by using the space-time feature complementary fusion module to obtain a pedestrian feature fusion vector.

[0073] Specifically, the image data in the training set is read by using a data loading function (dataloader), and the read image data is input into the gait feature assisted pedestrian re-identification network in batches. The output of the network is the final pedestrian feature (i.e., the pedestrian feature fusion vector) of the batch data. Under the existing computing resources, the batch size (batch_size) of the pictures for training the gait feature assisted pedestrian re-identification network is preferably set to 32. If the computing resources are sufficient, it can also be set to 64 or 128.

[0074] S4, based on the cross-entropy loss function and the triplet loss function corresponding to the pedestrian feature fusion vector, an adaptive weighting method is used to obtain a total loss value.

[0075] Specifically, the cross-entropy loss function and the triplet loss function are used to calculate the loss of the obtained pedestrian feature. Then, the two loss values are adaptively weighted to obtain a total loss value.

[0076] The cross-entropy loss function is used to measure the gap between the predicted value and the true value, and the loss function is represented as:

[0077] H y′ (y)=-∑ i y i′ log(y t )

[0078] Where H y′ (y) is the loss function, yi′ yi is the i-th real value in the actual label, y i yi is the i-th prediction value corresponding to the pedestrian feature fusion vector in the softmax function. Obviously, when y i The more accurate, the smaller the result, and finally the average value is obtained, that is, the loss value of the embodiment.

[0079] Triplet loss is used to reduce intra-class distance and increase inter-class distance. The loss function is represented as:

[0080] L(a, p, n) = max(D(a, p) - D(a, n) + margin, 0)

[0081] In the above formula, the input is a triplet, including anchor sample (a), positive sample (p) and negative sample (n). The positive sample p and the anchor sample a are samples of the same class, and the negative sample n and the anchor sample a are samples of different classes. Margin is a constant greater than 0. D is a distance metric function, usually Euclidean distance or cosine distance. Obviously, the optimization goal of L is to reduce the distance between a and p, and to increase the distance between a and n.

[0082] For the two loss function values, an adaptive weighting method is used to obtain the total loss value. The adaptive method can be represented as:

[0083] L = aL ce + bL tri

[0084] where L ce and L tri are the cross-entropy loss function value and the triplet loss function value, respectively, and a and b are learnable parameters. Through the learnable parameters, the adaptive weighted loss function value is obtained.

[0085] S5, the parameters of the gait feature assisted pedestrian re-identification network are optimized through gradient back propagation of the total loss value until the preset condition is reached, and the trained gait feature assisted pedestrian re-identification network is obtained.

[0086] The network parameters are learned through gradient back propagation of the total loss value, and the Adam optimizer is used to optimize the training process, and multi-step learning rate, i.e. learning rate decay, is used to ensure the stability of the training process; wherein the Adam optimizer is used to set the training gradient descent function of the gait feature assisted pedestrian re-identification network.

[0087] The preset condition is, for example, that the network is trained for a total of 250 rounds. During the training process, a test is performed after 40 rounds, and a test is performed after 10 rounds when the number of training rounds reaches 100. The cumulative matching characteristic curve (CMC curve) and the mean average precision (mAP) are used as indexes to measure the performance of the model during the test.

[0088] S6, using the trained gait feature assisted pedestrian re-identification network to process the test set and the search set, inputting the search set into the trained gait feature assisted pedestrian re-identification network to match the images of the same identity in the test set, so as to complete the test.

[0089] In an optional embodiment, step S6 can specifically include S6.1-S6.2, wherein:

[0090] S6.1, using the trained gait feature assisted pedestrian re-identification network to extract the pedestrian features of the test set and the pedestrian features of the search set.

[0091] Specifically, the trained gait feature assisted pedestrian re-identification network is used to process the test set and the search set. The test set and the search set are input into the network, and the pedestrian features of all images in the test set and the pedestrian features of all images in the search set are extracted.

[0092] S6.2, calculating the cosine similarity between the pedestrian features of the test set and the pedestrian features of the search set to obtain the identity of the most matched pedestrian image of the test set.

[0093] Specifically, the cosine similarity between the pedestrian features of the test set and the pedestrian features of the search set is calculated to re-identify the target pedestrian and obtain the most matched test set pedestrian image identity. The cosine similarity between the pedestrian features of each image in the test set and the pedestrian features of each image in the search set is calculated using the cosine similarity formula, and for each image in the search set, the image with the cosine value closest to 1 is selected from the test set.

[0094]

[0095] wherein X is the search set pedestrian feature vector, Y is the test set pedestrian feature vector, and θ is the angle between the two vectors. The cosine similarity value is in the range of [-1, 1], and the greater the similarity value, the higher the similarity between the two vectors. In particular, a cosine similarity of 1 represents that the angle between the two feature vectors is 0, i.e., complete similarity. For example, please refer to Figure 8 Pedestrian re-identification test results.

[0096] Step 3, input the first pedestrian feature output by the pedestrian re-identification network and the first pedestrian gait feature output by the gait recognition network into the spatio-temporal feature complementary fusion module of the trained gait feature auxiliary pedestrian re-identification network for feature fusion to obtain an initial pedestrian feature vector.

[0097] In an optional embodiment, referring to Figure 7 , step 3 can specifically include steps 3.1-3.7, wherein:

[0098] Step 3.1, average pooling is performed on the first pedestrian feature to obtain a second pedestrian feature, normalization is performed on the second pedestrian feature to obtain a first pedestrian feature vector, time pooling (TP) and generalized average pooling (GeM) are performed on the first pedestrian gait feature to obtain a second pedestrian gait feature, and the second pedestrian gait feature is processed through a fully connected layer (FC) and normalization (BN) to obtain a first pedestrian gait feature vector;

[0099] Specifically, for the pedestrian feature map (i.e., the first pedestrian feature), average pooling and normalization (AveragePooling+BN) are performed to obtain a 2048-dimensional first pedestrian feature vector. For the gait feature map (i.e., the first pedestrian gait feature), time pooling and generalized average pooling are first used to realize feature mapping, and then a fully connected layer and normalization are used to flatten the features to obtain a 512-dimensional first pedestrian gait feature vector. Finally, two linear layers are used to map the two feature vectors to 512-dimensional vectors, respectively.

[0100] Step 3.2, map the first pedestrian feature vector and the first pedestrian gait feature vector to the same dimension through two first linear layers to obtain a second pedestrian feature vector and a second pedestrian gait feature vector of the same dimension, for example, 512-dimensional vectors.

[0101] Step 3.3, connect the second pedestrian feature vector and the second pedestrian gait feature vector according to the dimension to obtain a connected first dimension vector.

[0102] Specifically, the 512-dimensional second pedestrian feature vector and the second pedestrian gait feature vector are connected according to the dimension to obtain a 1024-dimensional first dimension vector.

[0103] Step 3.4, use two second linear layers to respectively perform dimension reduction processing on the first dimension vector, and activate through a RELU (Linear rectification function) activation function to obtain two second dimension vectors.

[0104] Specifically, two second linear layers are used to reduce the first dimension vector to 512 dimensions and activated by a RELU activation function to obtain two second dimension vectors of 512 dimensions.

[0105] Step 3.5, one of the second dimension vectors is mapped to a third dimension vector of N dimensions using a third linear layer, and the other second dimension vector is mapped to an N*N dimension vector using a fourth linear layer, and the size of the N*N dimension vector is rearranged to (N, N) as a dynamic weight, N is for example 512 dimensions.

[0106] Specifically, one of the second dimension vectors is mapped to a 512 dimension vector as a feature vector using a third linear layer, and the other second dimension vector is mapped to a 512*512 dimension vector using a fourth linear layer, and the size of the 512*512 dimension vector is rearranged to (512, 512) as a dynamic weight.

[0107] Step 3.6, matrix multiplication is performed between the third dimension vector and the dynamic weight to obtain an N dimension fusion feature vector.

[0108] Specifically, matrix multiplication is performed between the third dimension vector and the dynamic weight to obtain a 512 dimension fusion feature vector.

[0109] Step 3.7, the fifth linear layer is used to perform dimension increasing processing on the N dimension fusion feature vector, and residual connection is performed with the first pedestrian feature to obtain an initial pedestrian feature vector.

[0110] Specifically, the fifth linear layer is used to increase the dimension of the fusion feature vector to 2048 dimensions, and residual connection is performed with the input pedestrian feature to obtain a 2048 dimension initial pedestrian feature vector.

[0111] Step 4, the initial pedestrian feature vector and the first pedestrian feature are connected according to the dimension and normalized to obtain a final pedestrian feature vector.

[0112] Specifically, the 2048 dimension initial pedestrian feature vector and the 2048 dimension first pedestrian feature are spliced according to the dimension to obtain a 4096 dimension feature, and finally normalized to obtain a final pedestrian feature vector, so that the final pedestrian feature vector obtained can be used to realize pedestrian re-identification. In particular, the pedestrian feature vector needs to be mapped to the dimension of the class number by a linear layer, so as to perform class probability operation by a softmax function.

[0113] Experiment

[0114] The experiment involves the MARS database. The MARS database is a public database of pedestrian re-identification based on videos, which is an expansion of the Market-1501 database, and the MARS database is photographed by 6 almost synchronous cameras, of which 5 are 1080*1920 HD cameras and one is a 640*480 SD camera. In the MARS database, there are 1261 different pedestrians, and the 1261 pedestrians all appear in at least 2 different cameras, the amount of data in the MARS database is large, which can provide enough data for the training of the convolutional neural network, thereby improving the robustness. The picture in the MARS database is 256*128.

[0115] Evaluation criteria: The evaluation criteria used in the experiment are the cumulative matching curve (CMC curve) and the average accuracy (mAP). Rank1, Rank5, Rank10 and Rank20 respectively represent the accuracy of the target contained in the first 1, the first 5, the first 10 and the first 20 returned results of the model prediction.

[0116] Table 1 video pedestrian re-identification accuracy

[0117] Rank 1 Rank 5 Rank 10 Rank 20 mAP Pedestrian re-identification baseline method 89.6 96.7 97.5 98.1 84.3 Gait recognition method 11.3 21.7 27.7 34.6 3.8 Method of the invention 90.6 97.0 98.0 98.4 85.5

[0118] From the comparison of the experimental results in the table, it can be seen that the gait feature assisted pedestrian re-identification method based on spatio-temporal feature complementary fusion proposed in the application has good performance improvement in Rank 1 and mAP of the MARS database. The video pedestrian re-identification baseline method can achieve 89.6% Rank1 precision and 84.5% mAP precision on the MARS dataset; the gait recognition method can only achieve 31.5% Rank1 precision and 17.5% mAP precision on the MARS dataset. The application combines the advantages of the two, and proposes a gait feature assisted pedestrian re-identification method based on spatio-temporal feature complementary fusion, which achieves 90.5% Rank1 precision and 85.5% mAP precision on the MARS dataset.

[0119] The pedestrian re-identification method of the application is a gait feature assisted pedestrian re-identification method based on spatio-temporal feature complementary fusion, which can use optical images and gait images to train the convolutional neural network, use gait features to assist pedestrian re-identification, and improve the accuracy of the pedestrian re-identification method of the application.

[0120] The gait feature assisted pedestrian re-identification method based on space-time feature complementary fusion of the embodiment of the application can not only be applied to a single camera, but also can be applied to pedestrian re-identification of multiple cameras, and can be applied to finding lost people in public places, and in the case that face information is missing and face recognition cannot be performed, the gait feature assisted pedestrian re-identification method of the embodiment of the application is used to assist in finding relevant personnel.

[0121] In the description of the application, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0122] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0123] Although the present application is described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and implemented by those skilled in the art with reference to the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. Some measures are described in mutually different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0124] The above is a further detailed description of the application in conjunction with specific preferred embodiments, and the specific implementation of the application cannot be limited to these descriptions. Modifications made by those skilled in the art without departing from the concept of the application should be considered within the scope of protection of the application.

Claims

1. A pedestrian re-identification and gait recognition method based on complementary fusion of spatiotemporal features, characterized by: The pedestrian re-identification and gait recognition method includes: Step 1: Acquire an optical image to be detected and a gait image of a pedestrian to be detected in a video to be detected, wherein the optical image to be detected is an original image extracted from the video to be detected, and the gait image of the pedestrian to be detected is a profile image of the pedestrian obtained using the optical image to be detected; Step 2: Inputting the optical image to be detected and the gait image of the pedestrian to be detected into a pedestrian re-identification network and a gait recognition network of a trained gait feature-assisted pedestrian re-identification network, respectively, and the pedestrian re-identification network and the gait recognition network output the first pedestrian feature and the first pedestrian gait feature, respectively; Step 3: Input the first pedestrian feature output by the pedestrian re-identification network and the first pedestrian gait feature output by the gait recognition network into the spatiotemporal feature complementary fusion module of the trained gait feature-assisted pedestrian re-identification network for feature fusion to obtain an initial pedestrian feature vector; Step 4: Concatenate the initial pedestrian feature vector and the first pedestrian feature according to dimension and normalize them to obtain a final pedestrian feature vector; The step 3 includes: Step 3.1: Perform average pooling on the first pedestrian feature to obtain a second pedestrian feature, normalize the second pedestrian feature to obtain a first pedestrian feature vector, perform temporal pooling and generalized average pooling on the first pedestrian gait feature to obtain a second pedestrian gait feature, pass the second pedestrian gait feature through a fully connected layer and normalize it to obtain the first pedestrian gait feature vector; Step 3.2: Map the first pedestrian feature vector and the first pedestrian gait feature vector to the same dimension through two first linear layers to obtain a second pedestrian feature vector and a second pedestrian gait feature vector of the same dimension; Step 3.3: Connect the second pedestrian feature vector and the second pedestrian gait feature vector according to the dimension to obtain a connected first-dimensional vector; Step 3.4: Use two second linear layers to perform dimensionality reduction processing on the first dimensional vector respectively, and activate it through the RELU activation function to obtain two second dimensional vectors; Step 3.5: Use a third linear layer to map one of the second-dimensional vectors to a third-dimensional vector of dimension N. Use a fourth linear layer to map the other second-dimensional vector to an N×N-dimensional vector, and rearrange the dimensions of the N×N-dimensional vector to (N, N) as a dynamic weight. Step 3.6: Perform matrix multiplication on the third-dimensional vector and the dynamic weight to obtain an N-dimensional fused feature vector; Step 3.7: Use the fifth linear layer to increase the dimension of the N-dimensional fusion feature vector, and perform a residual connection with the first pedestrian feature to obtain the initial pedestrian feature vector.

2. The method for pedestrian re-identification and gait recognition based on complementary fusion of spatiotemporal features according to claim 1, characterized in that: The step 1 comprises: Step 1.1, sampling the video to be detected to obtain the optical image to be detected; Step 1.2: Use the trained Mask RCNN network to process the optical image to be detected to obtain a pedestrian mask, and use the pedestrian mask as the gait image of the pedestrian to be detected.

3. The method for pedestrian re-identification and gait recognition based on complementary fusion of spatiotemporal features according to claim 1, characterized in that: The training method of the gait feature-assisted pedestrian re-identification network includes: S1. Acquire a data set including a plurality of optical images and a plurality of gait images, wherein the data set includes a training set, a test set, and a search set; S2. Inputting the optical image and gait image in the training set into the pedestrian re-identification network and the gait recognition network respectively to obtain a third pedestrian feature and a third pedestrian gait feature; S3. Using the spatiotemporal feature complementary fusion module to perform feature fusion on the third pedestrian feature and the third pedestrian gait feature to obtain a pedestrian feature fusion vector; S4. Based on the cross entropy loss function and the triplet loss function corresponding to the pedestrian feature fusion vector, an adaptive weighted method is used to obtain a total loss value; S5. Optimize the parameters of the gait feature-assisted person re-identification network by using the total loss value through gradient back propagation until a preset condition is met, thereby obtaining a trained gait feature-assisted person re-identification network.

4. The method for pedestrian re-identification and gait recognition based on complementary fusion of spatiotemporal features according to claim 3, characterized in that: Before step S2, the method further includes: The optical images in the training set are subjected to data enhancement by adopting random flipping and / or random erasing methods, and the gait images in the training set are subjected to data enhancement by adopting random flipping methods.

5. The method for pedestrian re-identification and gait recognition based on complementary fusion of spatiotemporal features according to claim 1 or 3, characterized in that: The pedestrian re-identification network is a Resnet50-I3D network obtained by optimizing the ResNet50-2D network, specifically: Based on the ResNet50-2D network, all 2D convolutional layers in the ResNet50-2D network are changed to 3D convolutional layers, all two-dimensional normalization layers are changed to three-dimensional normalization layers, and a one-dimensional time dimension is added.

6. The method for pedestrian re-identification and gait recognition based on complementary fusion of spatiotemporal features according to claim 1 or 3, characterized in that: The gait recognition network includes a GaitGL network, and the gait recognition network divides pedestrian gait features into 4 blocks and flattens them into 512-dimensional outputs.

7. The method for pedestrian re-identification and gait recognition based on complementary fusion of spatiotemporal features according to claim 3, characterized in that: After step S5, the method further includes: S6. Use the trained gait feature-assisted person re-identification network to process the test set and the search set, input the search set into the trained gait feature-assisted person re-identification network to match images of the same identity in the test set to complete the test.

8. The method for pedestrian re-identification and gait recognition based on complementary fusion of spatiotemporal features according to claim 7, characterized in that: Step S6 includes: S6.

1. Using the trained gait features to assist the pedestrian re-identification network to extract the pedestrian features of the test set and the pedestrian features of the search set S6.

2. Calculate the cosine similarity between the pedestrian features of the test set and the pedestrian features of the search set to obtain the identity of the pedestrian image that best matches the test set.

Images