A pedestrian re-identification method and system based on local features and graph convolution

Abstract

The present invention relates to a method and system for pedestrian re-identification based on local features and graph convolution, the method comprising the following steps: obtaining N frames of a video sequence to be identified; dividing each frame of the image into blocks, and performing average pooling processing on the features of each image block to obtain a first feature matrix; extracting spatial features from the first feature matrix to obtain spatial features; using graph convolution to extract temporal features from the first feature matrix to obtain temporal features; splicing the temporal features and spatial features of the video sequence to be identified to obtain the splicing features of the video sequence to be identified; identifying pedestrians in the video sequence based on the splicing features. The present invention achieves the distinction of local features by performing block processing, uses graph convolution to focus on the features that change in each image block, combines the temporal change features with each local spatial feature for identification, and improves the discrimination of pedestrian re-identification.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-identification, and in particular to a pedestrian re-identification method and system based on local features and graph convolution. Background Art

[0002] Person re-identification, also known as pedestrian re-recognition, is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is widely considered to be a sub-problem of image retrieval. Given a surveillance pedestrian video image, retrieve the video image of the pedestrian across devices. It aims to make up for the visual limitations of fixed cameras.

[0003] The research on person re-identification began in the mid-1990s. Researchers borrowed and introduced some mature methods in the fields of image processing and pattern recognition, focusing on the available features and simple classification algorithms of pedestrians. Since 2014, the training library of pedestrian re-identification technology has tended to be large-scale, and deep learning frameworks have been widely used. With the continuous in-depth research of universities, research institutes and some manufacturers, pedestrian re-identification technology has developed rapidly.

[0004] With the increasing number of video surveillance in society, various deep learning technologies have been widely used in criminal investigation, danger warning, unmanned supermarkets, lost rescue and other fields. For example, face recognition technology has become relatively mature and commercialized very successfully. Pedestrian re-identification is mainly used in the security field. Compared with face recognition, it is more difficult and has more environmental interference factors. The images obtained by pedestrian re-identification are often far away, and the pixels of people are very low, making it difficult to identify. At the same time, during the movement of pedestrians, there will be occlusion, posture changes, lighting changes, perspective changes, etc., which add more difficulties to identification.

[0005] How to improve the discrimination of pedestrian re-identification and overcome the technical difficulties in recognition caused by occlusion, posture changes, lighting changes, and perspective changes has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of this, the present invention provides a pedestrian re-identification method and system based on local features and graph convolution to improve the discrimination of pedestrian re-identification and overcome the technical difficulties of recognition difficulties caused by occlusion, posture changes, illumination changes, and perspective changes.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A pedestrian re-identification method based on local features and graph convolution, the method comprising the following steps:

[0009] Obtain N frames of the video sequence to be identified;

[0010] Divide each frame of the image into blocks, and perform average pooling processing on the features of each image block obtained by the block division to obtain a feature vector of each image block to form a first feature matrix;

[0011] Performing spatial feature extraction on the first feature matrix to obtain spatial features of the video sequence to be identified;

[0012] Using graph convolution to extract time features from the first feature matrix to obtain time features of the video sequence to be identified;

[0013] splicing the temporal features and the spatial features of the video sequence to be identified to obtain a splicing feature of the video sequence to be identified;

[0014] The splicing features of the video sequence to be identified are compared with the splicing features of the video sequence with the pedestrian ID tag in the database to determine the ID of the pedestrian in the video sequence to be identified. The splicing features of the video sequence with the pedestrian ID tag in the database are obtained in the same way as the splicing features of the video sequence to be identified.

[0015] Optionally, the image of each frame is divided into blocks, and the features of each image block obtained by the block division are respectively averaged pooled to obtain a feature vector of each image block to form a first feature matrix, which specifically includes:

[0016] Input the N frames of the image into the Resnet50 convolutional network for convolution and pooling operations to obtain the preliminary features of each frame of the image;

[0017] Divide the preliminary features of each frame of the picture into blocks to obtain the preliminary features of each image block;

[0018] An average pooling operation is performed on the preliminary features of each image block to obtain the feature vector of each image block to form a first feature matrix.

[0019] Optionally, the extracting spatial features from the first feature matrix to obtain the spatial features of the to-be-identified video sequence specifically includes:

[0020] A maximum pooling operation is performed on the first feature matrix in a first dimension to obtain spatial features of the video sequence to be identified.

[0021] Optionally, a maximum pooling operation is performed on the first feature matrix in a first dimension to obtain spatial features of the to-be-identified video sequence, and then the method further includes:

[0022] Expand the spatial features of the video sequence to be identified to a preset dimension.

[0023] Optionally, the extracting time features of the first feature matrix by graph convolution to obtain the time features of the video sequence to be identified specifically includes:

[0024] Dividing the first feature matrix into S second feature matrices according to the time sequence of N frames of pictures;

[0025] Performing a maximum pooling operation on each of the second feature matrices respectively, obtaining S feature vectors after maximum pooling, and forming a third feature matrix;

[0026] The third feature matrix is ​​divided according to the image blocks to obtain P fourth feature matrices; wherein P represents the number of image blocks, and each of the fourth feature matrices contains S maximum pooled features;

[0027] Perform multiple graph convolution operations on each of the fourth characteristic matrices to obtain P graph convolution results;

[0028] Perform a maximum pooling operation on each of the graph convolution results to obtain the temporal features of the video sequence to be identified.

[0029] Optionally, performing multiple graph convolution operations on each of the fourth feature matrices to obtain P graph convolution results may also include:

[0030] Construct the adjacency matrix:

[0031] Among them, A represents the adjacency matrix, D represents the diagonal characteristic matrix, D (i,i) represents the (i, i)th element in D, A2 represents the self-circulating adjacency matrix, A2=A1+E, A1 represents the characteristic relationship matrix, E represents the unit matrix, represents the (i, j)th element in A1, represents the (i, j)th element in A2, x i and x j represent the i-th and j-th elements in the fourth characteristic matrix respectively, Represents x i and x j , S is the number of features involved in graph convolution.

[0032] Optionally, the formula for performing graph convolution operation on each of the third feature matrices is:

[0033] in, represents the result of performing k graph convolution operations on the p-th fourth characteristic matrix; represents the result of performing k-1 graph convolution operations on the pth fourth feature matrix, W k represents the parameter matrix of the k-th graph convolution, and A represents the adjacency matrix.

[0034] Optionally, the method further includes performing a maximum pooling operation on each of the graph convolution results to obtain a temporal feature of a video sequence to be identified, and then:

[0035] The temporal features of the video sequence to be identified are expanded to a preset dimension.

[0036] A pedestrian re-identification system based on local features and graph convolution, the system comprising:

[0037] An image acquisition module is used to acquire N frames of images of a video sequence to be identified;

[0038] The image segmentation module is used to segment each frame of the image into blocks, and perform average pooling processing on the features of each image block obtained by the segmentation to obtain a feature vector of each image block to form a first feature matrix;

[0039] A spatial feature extraction module, used to extract spatial features from the first feature matrix to obtain spatial features of the video sequence to be identified;

[0040] A time feature extraction module, configured to extract time features from the first feature matrix by using graph convolution to obtain time features of the video sequence to be identified;

[0041] A feature splicing module, used for splicing the temporal features and spatial features of the video sequence to be identified to obtain the splicing features of the video sequence to be identified;

[0042] The recognition module is used to compare the splicing features of the video sequence to be recognized with the splicing features of the video sequence with the pedestrian ID tag in the database to determine the ID of the pedestrian in the video sequence to be recognized.

[0043] Optionally, the image segmentation module specifically includes:

[0044] The convolution and pooling unit is used to input the N frames of the image into the Resnet50 convolutional network for convolution and pooling operations to obtain preliminary features of each frame of the image;

[0045] A block division unit is used to divide the preliminary features of each frame of the picture into blocks to obtain preliminary features of each image block;

[0046] The average pooling unit is used to perform an average pooling operation on the preliminary features of each image block to obtain a feature vector of each image block to form a first feature matrix.

[0047] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] The present invention discloses a pedestrian re-identification method based on local features and graph convolution, the method comprising the following steps: obtaining N frames of a video sequence to be identified; dividing each frame of the picture into blocks, and performing average pooling processing on the features of each image block obtained by the block division, obtaining the feature vector of each image block, and forming a first feature matrix; extracting spatial features from the first feature matrix to obtain the spatial features of the video sequence to be identified; using graph convolution, extracting temporal features from the first feature matrix to obtain the temporal features of the video sequence to be identified; splicing the temporal features and spatial features of the video sequence to be identified to obtain the splicing features of the video sequence to be identified; comparing the splicing features of the video sequence to be identified with the splicing features of the video sequence with pedestrian ID tags in the database, and determining the ID of the pedestrian in the video sequence to be identified. The present invention realizes the distinction of local features by performing block processing, uses graph convolution to focus on the features that change in each image block, combines the temporal change features with each local spatial feature for recognition, improves the discrimination of pedestrian re-identification, and overcomes the technical difficulties of recognition difficulties caused by occlusion, posture changes, illumination changes, and perspective changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A flowchart of a pedestrian re-identification method based on local features and graph convolution provided by the present invention;

[0051] Figure 2 A schematic diagram of a pedestrian re-identification method based on local features and graph convolution provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The purpose of the present invention is to provide a pedestrian re-identification method and system based on local features and graph convolution, so as to improve the discrimination of pedestrian re-identification and overcome the technical difficulties of recognition difficulties caused by occlusion, posture change, illumination change, and perspective change.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] A complete person re-identification system includes several links, such as person detection, feature extraction, and similarity measurement. At present, the academic community usually studies person detection separately, while our person re-identification system mainly studies the feature extraction link. In the entire re-identification system, feature extraction requires learning features that can cope with changes in pedestrians under different cameras, and requires feature extraction to be robust. Then the learned features are mapped to a new space so that the same people are closer in the feature space and different people are farther away. Finally, the image features are sorted according to the distance between them and the retrieval results are returned.

[0056] In order to obtain more discriminative features and at the same time obtain the temporal flow features of each block during the pedestrian movement, this paper proposes a pedestrian re-identification method based on local features and graph convolution.

[0057] like Figure 1 As shown, the present invention provides a pedestrian re-identification method based on local features and graph convolution, and the method comprises the following steps:

[0058] Step 101, obtaining N frames of images of a video sequence to be identified;

[0059] Step 102, dividing each frame of the picture into blocks, and performing average pooling processing on the features of each image block obtained by the block division, to obtain a feature vector of each image block to form a first feature matrix;

[0060] Step 103, extracting spatial features from the first feature matrix to obtain spatial features of the video sequence to be identified;

[0061] Step 104, extracting time features from the first feature matrix by graph convolution to obtain time features of the video sequence to be identified;

[0062] Step 105, splicing the temporal features and spatial features of the video sequence to be identified to obtain splicing features of the video sequence to be identified;

[0063] Step 106 : Compare the splicing features of the video sequence to be identified with the splicing features of video sequences with pedestrian ID tags in a database to determine the ID of the pedestrian in the video sequence to be identified.

[0064] Specific as Figure 2 The specific implementation of the method of the present invention is as follows:

[0065] Step A: Data processing. The person re-identification dataset is randomly divided into two subsets: training set and test set, and the image data is standardized and labeled before being input into the network. The specific implementation of data processing is as follows:

[0066] Step A01: Randomly select 50% of the images in the person re-identification dataset as the training set and 50% of the images as the test set.

[0067] Step A02. Obtain the label information of each picture according to the name of each picture, and merge the picture itself and the label information into a data array, where data[0] is the picture itself and data[1] is the label information.

[0068] Step B: Design a preliminary feature extraction network. For a video sequence, pass each frame of the image through the convolutional neural network's convolutional layer, normalization layer, and final activation function to output the preliminary features of each frame. The preliminary feature extraction method is as follows:

[0069] Step B01. For an input video sequence, assume that there are N frames of pictures, and the size of each picture is 3×256×128. The total matrix X is [N, 3, 256, 128].

[0070] Step B02. Input X into the Resnet50 convolutional network, and obtain a preliminary feature F through feature extraction and pooling operations of multiple convolutional layers. out , the dimension is [N, 2048, 16, 8], where N is the number of image frames, 2048 is the feature dimension, 16 and 8 are height and width respectively.

[0071] Step C: Design and extract the spatial structural features of pedestrians. Divide the features of each frame of the image extracted initially into blocks, and then reduce the dimension through pooling in the time series dimension. Make the features of each block representative. The specific implementation of video spatial feature extraction is as follows:

[0072] Step C01. Convert the F output in step B to out As the input of step C, the features of each frame of the image are divided into P blocks from top to bottom to obtain F part .

[0073] Step C02. Perform average pooling on the features of each block to obtain F' part At this time F' partThe dimension is [N, 2048, P]. That is, N×P feature vectors.

[0074] Step C03. For F' part , perform the maximum pooling operation on the first dimension to obtain the most representative spatial structural features of each block in the time series, and output F space The feature dimension is [P, 2048].

[0075] Step C04. Expand the features to P×2048 dimensions.

[0076] Step D: Design and extract the temporal structural features of pedestrians. Divide the video frames into S groups, each group has x frames, and perform the maximum pooling operation in each group of pictures, and use the result as the feature of this group of images. Then input the features of the S images into the graph convolutional network, and perform the maximum pooling operation on the last output S features to obtain a representative final feature. The specific implementation of video temporal feature extraction is as follows:

[0077] Step D01. F' output in step C part As the input of step D. Divide the N frames into S groups in chronological order, with each group containing N / S pictures. The dimension of feature F is [S, N / S, P, 2048].

[0078] Step D02: Perform maximum pooling on the image features in each group in F to extract the most representative features of each group. t The dimension is [S,P,2048].

[0079] Step D03. Set feature F t Divide by blocks, denoted as F (1) ,F (2) ,F (3) ……F (P) , each group has S features, recorded as x1,x2,x3...x S .

[0080] Step D04. Construct the adjacency matrix A for graph convolution. For the relationship between each two features Normalizing the matrix yields Construct a self-circular adjacency matrix A2=A1+E, where E is the identity matrix. Finally, we get the adjacency matrix A= in

[0081] Step D05. Perform graph convolution operation, the formula is Where k represents the number of layers of graph convolution. For example, when k = 1, it represents the result of the first layer of graph convolution. The feature F obtained in step D03 (1), W is the parameter matrix to be learned in each layer.

[0082] Step D06. (2) ,F (3) ……F (P) Repeat step D06 to obtain the graph convolution result of each block.

[0083] Step D07. Perform the maximum pooling operation on the S features in each block to obtain the final F time , at this time F time The feature dimension is [P, 2048].

[0084] Step D08. Expand the features to P×2048 dimensions.

[0085] Step E: Concatenate the features from step C and step D to get the final features, calculate the feature distance, and find the video ID of the same pedestrian. The final feature processing is as follows:

[0086] Step E01. Concatenate the features obtained in step C and step D to obtain the final features, calculate the pedestrian feature distance based on the final features, and find the video ID of the same pedestrian.

[0087] The present invention also provides a pedestrian re-identification system based on local features and graph convolution, the system comprising:

[0088] The image acquisition module is used to acquire N frames of images of the video sequence to be identified.

[0089] The image segmentation module is used to segment each frame of the image into blocks, and perform average pooling processing on the features of each image block obtained by segmentation to obtain a feature vector of each image block to form a first feature matrix.

[0090] The image segmentation module specifically includes: a convolution and pooling unit, which is used to input N frames of the image into the Resnet50 convolutional network for convolution and pooling operations to obtain preliminary features of each frame of the image; a segmentation unit, which is used to segment the preliminary features of each frame of the image respectively to obtain preliminary features of each image block; and an average pooling unit, which is used to perform an average pooling operation on the preliminary features of each image block to obtain a feature vector of each image block to form a first feature matrix.

[0091] The spatial feature extraction module is used to extract spatial features from the first feature matrix to obtain the spatial features of the video sequence to be identified.

[0092] The time feature extraction module is used to extract the time features of the first feature matrix by using graph convolution to obtain the time features of the video sequence to be identified.

[0093] The feature splicing module is used to splice the temporal features and spatial features of the video sequence to be identified to obtain the splicing features of the video sequence to be identified.

[0094] The recognition module is used to compare the splicing features of the video sequence to be recognized with the splicing features of the video sequence with the pedestrian ID tag in the database to determine the ID of the pedestrian in the video sequence to be recognized.

[0095] Compared with the prior art, the present invention has the following beneficial effects:

[0096] The present invention uses block graph convolution to focus on the features that change in the time series within each block, and combines the spatial features to make the extracted features more discriminative.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A pedestrian re-identification method based on local features and graph convolution, characterized in that: The method comprises the following steps: Obtain N frames of the video sequence to be identified; Divide each frame of the image into blocks, and perform average pooling processing on the features of each image block obtained by the block division to obtain a feature vector of each image block to form a first feature matrix; Performing spatial feature extraction on the first feature matrix to obtain spatial features of the video sequence to be identified; Using graph convolution to extract time features from the first feature matrix to obtain time features of the video sequence to be identified; splicing the temporal features and the spatial features of the video sequence to be identified to obtain a splicing feature of the video sequence to be identified; Compare the splicing features of the video sequence to be identified with the splicing features of video sequences with pedestrian ID tags in the database to determine the ID of the pedestrian in the video sequence to be identified; The extracting spatial features from the first feature matrix to obtain the spatial features of the to-be-identified video sequence specifically includes: Performing a maximum pooling operation on the first feature matrix in a first dimension to obtain spatial features of the video sequence to be identified; The step of extracting time features from the first feature matrix by graph convolution to obtain time features of the video sequence to be identified specifically includes: Dividing the first feature matrix into S second feature matrices according to the time sequence of N frames of pictures; Performing a maximum pooling operation on each of the second feature matrices respectively, obtaining S feature vectors after maximum pooling, and forming a third feature matrix; The third feature matrix is ​​divided according to the image blocks to obtain P fourth feature matrices; wherein P represents the number of image blocks, and each of the fourth feature matrices contains S maximum pooled features; Perform multiple graph convolution operations on each of the fourth characteristic matrices to obtain P graph convolution results; Performing a maximum pooling operation on each of the graph convolution results to obtain the temporal features of the video sequence to be identified; The method of performing multiple graph convolution operations on each of the fourth feature matrices to obtain P graph convolution results also includes: Construct the adjacency matrix: Among them, A represents the adjacency matrix, D represents the diagonal characteristic matrix, D (i,i) represents the (i, i)th element in D, A2 represents the self-circulating adjacency matrix, A2=A1+E, A1 represents the characteristic relationship matrix, E represents the unit matrix, represents the (i, j)th element in A1, represents the (i, j)th element in A2, x i and x j represent the i-th and j-th elements in the fourth characteristic matrix respectively, Represents x i and x j , S is the number of features involved in graph convolution; The formula for performing graph convolution operation on each of the third feature matrices is: in, represents the result of performing k graph convolution operations on the p-th fourth feature matrix; represents the result of performing k-1 graph convolution operations on the pth fourth feature matrix, W k represents the parameter matrix of the k-th graph convolution, and A represents the adjacency matrix.

2. The pedestrian re-identification method based on local features and graph convolution according to claim 1, characterized in that: The image of each frame is divided into blocks, and the features of each image block obtained by the block division are average pooled to obtain the feature vector of each image block to form a first feature matrix, which specifically includes: Input the N frames of the image into the Resnet50 convolutional network for convolution and pooling operations to obtain the preliminary features of each frame of the image; Divide the preliminary features of each frame of the picture into blocks to obtain the preliminary features of each image block; An average pooling operation is performed on the preliminary features of each image block to obtain the feature vector of each image block to form a first feature matrix.

3. The pedestrian re-identification method based on local features and graph convolution according to claim 1, characterized in that: The step of performing a maximum pooling operation on the first feature matrix in a first dimension to obtain the spatial features of the video sequence to be identified further includes: Expand the spatial features of the video sequence to be identified to a preset dimension.

4. The pedestrian re-identification method based on local features and graph convolution according to claim 1, characterized in that: The method further comprises performing a maximum pooling operation on each of the graph convolution results to obtain the temporal features of the video sequence to be identified, and then: The temporal features of the video sequence to be identified are expanded to a preset dimension.

5. A person re-identification system based on local features and graph convolution, characterized in that: The system applies the method described in any one of claims 1 to 4, and the system comprises: An image acquisition module is used to acquire N frames of images of a video sequence to be identified; The image segmentation module is used to segment each frame of the image into blocks, and perform average pooling processing on the features of each image block obtained by the segmentation to obtain a feature vector of each image block to form a first feature matrix; A spatial feature extraction module, used to extract spatial features from the first feature matrix to obtain spatial features of the video sequence to be identified; A time feature extraction module, configured to extract time features from the first feature matrix by using graph convolution to obtain time features of the video sequence to be identified; A feature splicing module, used for splicing the temporal features and spatial features of the video sequence to be identified to obtain the splicing features of the video sequence to be identified; The recognition module is used to compare the splicing features of the video sequence to be recognized with the splicing features of the video sequence with the pedestrian ID tag in the database to determine the ID of the pedestrian in the video sequence to be recognized.

6. The pedestrian re-identification system based on local features and graph convolution according to claim 5, characterized in that: The image segmentation module specifically includes: The convolution and pooling unit is used to input the N frames of the image into the Resnet50 convolutional network for convolution and pooling operations to obtain preliminary features of each frame of the image; A block division unit is used to divide the preliminary features of each frame of the picture into blocks to obtain preliminary features of each image block; The average pooling unit is used to perform an average pooling operation on the preliminary features of each image block to obtain a feature vector of each image block to form a first feature matrix.

Images