Video pedestrian re-identification method based on graph convolution and graph adaptive pooling

Abstract

The invention provides a video pedestrian re-identification method based on graph convolution and graph adaptive pooling, which comprises the following steps: extracting features of each frame of image in a video sequence to obtain a frame-level feature graph; aggregating the video features subjected to average pooling from a time dimension by using a TAM module to obtain global feature representation; constructing a block relation graph, and associating the video frames; processing the block relation graph by using a graph convolutional neural network, and modeling a space-time relation of different inter-frame areas; optimizing the topological structure of the graph model by using a graph adaptive pooling technology, and removing nodes with low contribution rate to pedestrian features; gathering node features after graph convolution and graph adaptive pooling processing into video feature representation through average pooling operation; and the global feature representation and the video feature representation are subjected to feature fusion to obtain final video pedestrian feature representation. According to the invention, through optimizing local feature extraction and suppressing interference information, the discrimination and robustness of pedestrian identity characterization are significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling. Background Art

[0002] With the popularization of surveillance technology and the development of intelligent video analysis technology, video-based person re-identification (V-ReID) has shown increasingly important application value in the fields of public safety and intelligent surveillance. The core of video-based person re-identification lies in how to accurately extract and match pedestrian representations from video sequences to achieve cross-camera tracking and recognition of specific pedestrians. However, this task faces many challenges, among which the extraction of pedestrian representations is particularly critical.

[0003] As the mainstream research direction of V-ReID, video pedestrian representation learning aims to extract discriminative pedestrian features from video sequences through deep learning and other technical means. Early solutions mainly focused on selecting neural network architectures that are easy to extract time domain features, such as 3D convolutional neural networks (3D-CNN) or recurrent neural networks (RNN), to capture dynamic information in video sequences. These methods have improved the timeliness of pedestrian representation to a certain extent, but are still insufficient when dealing with complex scenes.

[0004] In order to further improve the robustness and accuracy of video pedestrian representation, researchers began to explore potential auxiliary features, such as pedestrian attributes, posture, gait, etc., to enhance the ability of representation learning. Especially in the field of image pedestrian re-identification, human attributes have been proven to effectively improve recognition performance, and this idea has gradually been introduced into video pedestrian re-identification.

[0005] Although significant progress has been made in the study of video person re-identification, there are still some problems and difficulties in existing methods. Among them, how to extract discriminative video-level visual features is a key issue that needs to be solved urgently. Most current methods follow the "space-time" sequence to extract features of the entire video sequence and then obtain video-level feature representation. However, in practical applications, the target pedestrian is often affected by various interference factors, such as occlusion in the video frame, background clutter, etc.

[0006] Specifically, when a pedestrian is occluded by an obstacle, if only relying on pixel information in a single frame, the occluded part may be regarded as missing information, thus affecting the extraction of the complete pedestrian representation by the network. This occlusion problem is particularly common in video sequences and is difficult to effectively solve through simple spatio-temporal feature fusion. In addition, background clutter may also be confused with pedestrian features, interfering with the feature extraction and recognition process of the model and reducing the robustness and accuracy of the model.

[0007] Therefore, to overcome the above challenges, a new video pedestrian re-identification method is needed. This method can make full use of spatio-temporal information in the video sequence, effectively cope with interference factors such as occlusion and background clutter, and extract more discriminative pedestrian features. In particular, a technology that can intelligently complete the occluded area and filter out information interference unrelated to pedestrians is required to improve the accuracy and robustness of video pedestrian re-identification. The present invention is proposed based on this background technology, aiming to provide an innovative solution to meet the urgent needs in practical applications. Summary of the Invention

[0008] Aiming at the problems existing in the prior art, the purpose of the present invention is to provide a video pedestrian re-identification method based on graph convolution and graph adaptive pooling. This method first uses a graph convolutional network to model the spatio-temporal relationship of the inter-frame difference regions in the video sequence, optimizes the regional features of each frame image through inter-frame complementary information, alleviates occlusion and spatial misalignment, and then uses graph adaptive pooling to optimize the topological structure of the graph model, which can finely remove nodes with low contribution rate to pedestrian features and reduce the interference of background clutter.

[0009] The present invention realizes the above purpose through the following technical solutions:

[0010] A video pedestrian re-identification method based on graph convolution and graph adaptive pooling, comprising the following steps:

[0011] Extract the features of each frame image in the video sequence to obtain a frame-level feature map;

[0012] Perform average pooling on the extracted video features, and use the TAM module to aggregate the video features after average pooling from the time dimension to obtain a global feature representation;

[0013] Construct a block relationship graph, divide the frame-level feature map in the video sequence into multiple blocks, and convert each block into a feature vector, and associate the video frames by constructing a block relationship graph;

[0014] Use a graph convolutional neural network to process the block relationship graph and model the spatio-temporal relationship of different regions between frames;

[0015] Optimize the topological structure of the graph model using graph adaptive pooling technology to remove nodes with low contribution rates to pedestrian features;

[0016] Aggregate the node features after graph convolution and graph adaptive pooling into video feature representations through average pooling operations;

[0017] Among them, the global feature representation and the video feature representation are used to obtain the final video pedestrian feature representation through feature fusion for the video pedestrian re-identification task.

[0018] According to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, use the RMNet network structure to extract the features of each frame image in the video sequence to obtain frame-level feature maps; among them, the RMNet network is a backbone network obtained by embedding a video pedestrian re-identification network with motion attention and long-range temporal aggregation into ResNet50;

[0019] Then, send the extracted frame-level feature maps into a two-stream network, and the two-stream network includes a global branch and a graph branch; among them, the global branch is used to extract the global features of each frame image, and the graph branch is used to extract cross-frame correlation features.

[0020] According to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, the construction of the block relationship graph includes:

[0021] Use V = {I1, I2, …, I T} to represent the given video, where T is the number of frames sampled from the video. For each frame of the video, define the frame-level feature map extracted by the backbone network as the following formula:

[0022] F = {F1, F2, …, F T}

[0023] Among them, F i is the feature map of the i-th frame in the video, where H and W respectively represent the length and width of the spatial shape, and C represents the number of feature channels; divide each feature map F i horizontally into S blocks with W×C as the section, and convert all blocks into feature vectors through average pooling operations. The feature vector of block i is denoted as x i ; Therefore, for a video with T frames, the total number of blocks is N = T·S, and the block feature vector of the video

[0024] According to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, let the block relationship graph composed of N nodes be G(V, E), and the node v i ∈V, edge (v i , vj ) ∈ E, each local block is regarded as a node, and the relationship between local block features is defined by the edges in E; the association between nodes is reflected by the adjacency matrix A ∈ R N×N For any two node feature vectors x i and x j in the block relationship graph, the relationship can be shown as the following formula:

[0025]

[0026] According to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, during the construction of the adjacency matrix, in the affinity matrix, the sum of the edge values in each row should be 1; all elements of the adjacency matrix are non-negative, and the element values representing the mutual relationship between nodes range from (0, 1); the elements in the adjacency matrix A are normalized through the Softmax function, expressed as the following formula:

[0027]

[0028] where, represents the self-loop adjacency matrix, and I ∈ R N×N is the identity matrix.

[0029] According to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, a re-normalization method is used to approximate the graph-Laplacian matrix to facilitate performing graph convolution operations, expressed as the following formula:

[0030]

[0031] where, Finally, the adjacency matrix corresponding to the block relationship graph G(V, E) is obtained Thereby further modeling the spatio-temporal relationship between blocks.

[0032] According to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, in the graph branch, for a given adjacency matrix the temporal relationship between different blocks in the frames of the GCN video sequence is modeled, and the graph convolutional neural network is implemented through the following formula:

[0033]

[0034] where, is the original block feature, is the learned parameter matrix, σ(·) represents the activation function, and the graph convolutional neural network outputs the updated block features for each video segment

[0035] A video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, the graph adaptive pooling technology includes pooling topology learning and node feature fusion. In the structure-based pooling topology learning, GCN is used to calculate the structure information score S1 of the node, which is expressed as the following formula:

[0036]

[0037] where is the normalized graph-Laplacian matrix, H is the block feature optimized by the graph convolution network, W is the learned weight vector, and σ is the activation function;

[0038] Feature-based topology learning FBTL is used to calculate the feature information score. A multi-layer perceptron MLP is used to extract the node features, which is expressed as the following formula:

[0039] S2 = σ(MLP(H))

[0040] where H is the block feature optimized by GCN, σ is the activation function, and S2 is the aggregation of node features, which can be regarded as the node evaluation value;

[0041] The structure feature topology learning SFTL component is used to fuse the structure information score and the feature information score to generate the final pooling topology, and the combination is expressed as the following formula:

[0042] S = αS1+(1 - α)S2

[0043] where the weight α is a user-defined hyperparameter. The nodes are sorted according to the scores of the nodes, and the top k nodes with the highest scores are used as the pooling result, that is, the top-k selection pooling method.

[0044] A video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention. In the node feature fusion, a graph attention network GAT is selected as the node feature fusion method, which is expressed as the following formula:

[0045]

[0046] where h i is the feature vector of node i, h j represents the neighbor nodes of node i, N i is the number of adjacent nodes of node i, is the attention value between h i and h j and W is the weight matrix.

[0047] According to a video pedestrian re-identification method based on graph convolution and graph adaptive pooling provided by the present invention, when the input is the hidden layer feature H, the execution process of graph adaptive pooling is represented by the following formula:

[0048] H out = H idx,: ⊙ S idx

[0049] where represents the block feature read through the node index and after feature fusion, ⊙ represents element-wise multiplication, · idx represents the indexing operation, and the output feature

[0050] performs an average pooling operation on H out For each video segment, through graph convolution and graph adaptive pooling, the video feature can be obtained

[0051] It can be seen that the present invention proposes a video pedestrian re-identification method based on graph convolution and graph adaptive pooling, which has the following remarkable beneficial effects:

[0052] 1. Improve the discriminability and robustness of pedestrian identity representation: By introducing a graph convolutional neural network and constructing a block relationship graph based on the mutual relationship between different local features in video frames, the present invention can dynamically capture the spatio-temporal relationship between different blocks in the frames, so as to more comprehensively extract the identity features of pedestrians. Compared with traditional methods, the pedestrian identity representation extracted by the present invention not only has higher discriminability and can accurately distinguish different pedestrians, but also has stronger robustness and can maintain stable performance in complex and changeable environments.

[0053] 2. Effectively suppress occlusion information and background clutter: Aiming at the common occlusion problems and background clutter interference in video pedestrian re-identification, the present invention proposes a graph adaptive pooling technology, which can dynamically adjust the pooling strategy according to the block relationship graph and effectively suppress the interference of occlusion information and background clutter on the extraction of pedestrian identity representation. Through this mechanism, the present invention can still maintain a high recognition accuracy when the pedestrian is partially occluded or the background is complex.

[0054] 3. Enhance the adaptability and generalization ability of the method: Since the present invention is a method based on graph convolution and graph adaptive pooling, it can flexibly adapt to video pedestrian re-identification tasks in different scenarios. Whether it is an indoor or outdoor environment, whether it is a single-person or multi-person scenario, the present invention can dynamically capture and analyze the spatio-temporal relationship between blocks in the frames and extract discriminative pedestrian identity representations, making the present invention have strong generalization ability and can be widely applied to various monitoring and tracking systems.

[0055] 4. Experimental verification of the method's effectiveness: The present invention has verified its effectiveness through a large number of experiments. The experimental results show that the present invention has achieved excellent performance on multiple video pedestrian re-identification datasets, outperforming traditional methods in terms of both accuracy and robustness, providing strong support for the practical application of the present invention and further demonstrating the potential and value of the present invention in the field of video pedestrian re-identification.

[0056] In summary, the present invention proposes a video pedestrian re-identification method based on graph convolution and graph adaptive pooling. By optimizing local feature extraction and suppressing interference information, this method significantly improves the discriminability and robustness of pedestrian identity representation. At the same time, the present invention also has strong adaptability and generalization ability, and can be widely applied to various monitoring and tracking systems, providing strong technical support for fields such as public safety and intelligent transportation.

[0057] The following further elaborates on the present invention in detail in conjunction with the accompanying drawings and specific embodiments. Description of the Drawings

[0058] Figure 1 It is a flowchart of an embodiment of the video pedestrian re-identification method based on graph convolution and graph adaptive pooling of the present invention.

[0059] Figure 2 It is a structural diagram of the ResNet50 backbone network in an embodiment of the video pedestrian re-identification method based on graph convolution and graph adaptive pooling of the present invention.

[0060] Figure 3 It is a network structural diagram of the TAM module in an embodiment of the video pedestrian re-identification method based on graph convolution and graph adaptive pooling of the present invention.

[0061] Figure 4 It is a construction process diagram of the block relationship graph in an embodiment of the video pedestrian re-identification method based on graph convolution and graph adaptive pooling of the present invention.

[0062] Figure 5 It is a network structural diagram of graph adaptive pooling in an embodiment of the video pedestrian re-identification method based on graph convolution and graph adaptive pooling of the present invention.

[0063] Figure 6 It is an algorithm flowchart of applying a graph neural network to the video pedestrian re-identification method in an embodiment of the video pedestrian re-identification method based on graph convolution and graph adaptive pooling of the present invention. Detailed Embodiments

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts belong to the scope of protection of the present invention.

[0065] As used herein, the mention of "embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0066] See Figures 1 to 6 , this embodiment provides a video pedestrian re-identification method based on graph convolution and graph adaptive pooling, and the method includes the following steps:

[0067] Step S1, extract the features of each frame image in the video sequence to obtain a frame-level feature map;

[0068] Step S2, perform average pooling on the extracted video features, and use the TAM module to aggregate the video features after average pooling from the time dimension to obtain a global feature representation;

[0069] Step S3, construct a block relationship graph, divide the frame-level feature map in the video sequence into multiple blocks, and convert each block into a feature vector, and associate the video frames by constructing a block relationship graph;

[0070] Step S4, use a graph convolutional neural network to process the block relationship graph and model the spatio-temporal relationship between different regions among frames;

[0071] Step S5, use graph adaptive pooling technology to optimize the topological structure of the graph model and remove nodes with low contribution rate to pedestrian features;

[0072] Step S6, through average pooling operation, converge the node features after graph convolution and graph adaptive pooling processing into a video feature representation;

[0073] Step S7, the global feature representation and the video feature representation are fused by feature fusion to obtain the final video pedestrian feature representation for video pedestrian re-identification tasks.

[0074] In the above step S1, the RMNet network structure is adopted to extract the features of each frame image in the video sequence, and a frame-level feature map is obtained; among them, the RMNet network is a backbone network after embedding a video pedestrian re-identification network integrating motion attention and remote time aggregation into ResNet50; then, the extracted frame-level feature map is sent into a two-stream network, and the two-stream network includes a global branch and a graph branch; among them, the global branch is used to extract the global features of each frame image, and the graph branch is used to extract cross-frame correlation features.

[0075] Specifically, the video pedestrian re-identification method based on Graph Convolution and Graph Self-adaptive Pooling (GCGSP) in this embodiment mainly includes:

[0076] 1. Introduce a graph convolutional neural network to optimize the regional features of frame-by-frame images according to the complementary information between adjacent frames, and model the spatio-temporal relationship of different regions between frames in the video sequence, so as to further improve the robustness of the video pedestrian re-identification network;

[0077] 2. In order to establish a stronger and more stable pedestrian overall representation with stronger identity information expression ability under the condition of background clutter in video frames, a method of graph self-adaptive pooling is introduced to optimize the topological structure of the graph model, finely remove the nodes with low contribution rate to pedestrian features, and reduce the interference of background clutter.

[0078] As Figure 2 shown, the overall framework of the method in this embodiment is shown in Figure 2 and mainly includes the following parts:

[0079] 1. Feature extraction: Use RMNet (the backbone network after embedding a video pedestrian re-identification network integrating motion attention and remote time aggregation into ResNet50 is called RMNet) to extract image features, and then send the extracted features into a two-stream network. This two-stream network is divided into a global branch and a graph branch, where the global branch is used to extract the global features of each frame image, and the graph branch is used to extract cross-frame correlation features; the structure of the ResNet50 backbone network is shown in Table 1, and Table 1 represents the original structure of ResNet50, with a default input size of 224×224×3 (height × width × number of channels).

[0080] 2. Global branch: Perform average pooling on the video features extracted by RMNet, and then use the TAM module to aggregate the video features from the time dimension; the TAM structure is shown in Figure 3 as follows.

[0081] 3. Graph Branch. First, to obtain the correlation information of the inter-frame difference regions, the video frames are associated by constructing a block relationship graph. Then, graph convolution and graph adaptive pooling are successively used to process the block relationship graph. Among them, the role of the graph convolutional neural network is to model the spatio-temporal correlation of the inter-frame difference regions, mine the hidden spatio-temporal information in the video sequence to improve the local representation; the role of graph adaptive pooling is to improve the graph topology structure, finely remove the nodes that contribute less to the pedestrian representation, so as to reduce the interference of background clutter; finally, the improved node features are aggregated into video features through average pooling operation.

[0082] (1)

[0084] In the above step S3, constructing the block relationship graph specifically includes:

[0085] Let V = {I1, I2, …, I T} represent the given video, where T is the number of frames sampled from the video. For each frame of the video, the frame-level feature map extracted by the backbone network is defined by the following formula:

[0086] F = {F1, F2, …, F T}

[0087] where F i is the feature map of the i-th frame in the video, where H and W respectively represent the length and width of the spatial shape, and C represents the number of feature channels; each feature map F i is horizontally divided into S blocks with W×C as the section, and all blocks are converted into feature vectors through average pooling operation. The feature vector of block i is denoted as x i ; therefore, for a video with T frames, the total number of blocks is N = T·S, and the block feature vector of the video

[0088] The difference between pedestrian re-identification based on video and image-based methods lies in that there are multiple image frames with a sequence length in the video. These image frames can not only strengthen the already distinct pedestrian representation, but also supplement the pedestrian representation that is disturbed in a certain frame. Therefore, the pedestrian postures contained in the video sequence are more plump, the shooting angles are more diverse, and the structural information of the pedestrians is also more substantial. To finely mine this hidden information, in this embodiment, the video frames are decomposed into local blocks, and the correlation between these local blocks is examined from the perspective of a graph, that is, the local blocks are used as the nodes of the graph, and the connections between the local blocks naturally become the edges of the graph. Considering that the local features contained in the nodes need to be optimized by the spatio-temporal information of the video sequence, in this embodiment, the relationship between the nodes is modeled through a graph convolutional network.

[0089] Let the block relationship graph composed of N nodes be G(V, E), and the node v i ∈V, and the edge (v i , v j ) ∈E. Treat each local block as a node, and the relationship between local block features is defined by the edges in E; the association between nodes is reflected by the adjacency matrix A ∈ R N×N . For any two node feature vectors x i and x j in the block relationship graph, the relationship can be shown as follows:

[0090]

[0091] However, in the process of constructing the adjacency matrix, the following two points need to be taken into consideration: (1) In the affinity matrix, the sum of the edge values in each row should be 1; (2) All elements of the adjacency matrix are non - negative, and the element values representing the mutual relationship between nodes range from (0, 1). Normalize the elements in the adjacency matrix A through the Softmax function, which is expressed as the following formula:

[0092]

[0093] where, represents the self - loop adjacency matrix, and I ∈ R N×N is the identity matrix.

[0094] In order to be able to perform graph convolution operations, a re - normalization method is needed to approximate the graph - Laplacian matrix, which is expressed as the following formula:

[0095]

[0096] where, Finally, the adjacency matrix corresponding to the block relationship graph G(V, E) is obtained Thereby further modeling the spatio - temporal relationship between blocks. The construction process of the block relationship graph is as Figure 4 shown.

[0097] After the graph is constructed, the next step is to select a suitable GNN model to extract graph features. In this embodiment, a graph convolutional neural network GCN and graph self - adaptive pooling GSP are used to extract graph features.

[0098] In the above step S4, for problems such as occlusion and spatial misalignment in video frames, relevant interferences can be offset by mining the complementary information of local blocks between video frames. In order to dynamically mine the spatio - temporal information contained in different blocks between these frames, a graph convolutional network is used in this embodiment. Specifically, taking the N blocks of each video sequence as nodes to construct a block relationship graph G(V, E), and then calculating the adjacency matrix

[0099] In the graph branch, for a given adjacency matrix The temporal relationship between different blocks in the frames of the GCN video sequence is modeled, and the graph convolutional neural network is implemented by the following formula:

[0100]

[0101] where is the original block feature, is the learned parameter matrix, σ(·) represents the activation function, and the graph convolutional neural network outputs the updated block feature of each video segment

[0102] In the above step S5, graph adaptive pooling can optimize the topological structure of the graph model. In the video person re-identification task, this technology can be used to finely remove those low-quality graph nodes, thereby reducing the interference of problems such as background clutter existing in the video sequence on pedestrian representation. This can enable the model to pay more attention to high-quality graph nodes, thereby improving the quality of the generated video features and the generalization performance of the model. As Figure 5 shown Figure 5 shows the structure of graph adaptive pooling.

[0103] Specifically, the graph adaptive pooling technology includes two parts: pooling topology learning and node feature fusion. In the first part, structure-based topology learning (SBTL) and feature-based topology learning (FBTL) calculate the importance scores according to the structure and feature information respectively. Then, the structure feature topology learning (SFTL) component will fuse the two importance scores to generate the final pooling topology. In the second part, a node feature fusion method based on the graph attention network GAT is adopted to combine the features of the unselected nodes around the selected nodes.

[0104] In structure-based topology learning (SBTL), a graph contains many nodes and edges, and there is rich structure information between these nodes and edges. Therefore, it is an effective method to evaluate the importance of nodes according to the structure information of the nodes. Considering that the graph convolutional network GCN can well capture the structure information between nodes, GCN can be used to calculate the structure information score S1 of the nodes, which is expressed as the following formula:

[0105]

[0106] where is the normalized graph Laplacian matrix, H is the block feature optimized by the graph convolutional network, W is the learned weight vector, and σ is the activation function;

[0107] Feature-based Topological Learning (FBTL) is used to calculate the feature information score. In graph data, nodes usually contain feature information. A node is represented to a large extent by its features. Therefore, it is very necessary to use node feature information for evaluation, and the influence of node features cannot be ignored. In this embodiment, a multi-layer perceptron (MLP) is used to extract node features, which is expressed by the following formula:

[0108] S2 = σ(MLP(H))

[0109] where H is the block feature optimized by GCN, σ is the activation function, and S2 is the aggregation of node features, which can be regarded as the node evaluation value. According to the aggregation result, additional important nodes can be reserved.

[0110] GCN is very effective in refining the local structure information of the graph, while MLP pays more attention to the feature information of nodes. With the structure information score and the feature information score, the Structure Feature Topological Learning (SFTL) component is used to fuse the structure information score and the feature information score to generate the final pooling topology. Considering two different node evaluation methods comprehensively effectively increases the diversity of evaluation and enhances the objectivity of the final node selection. The combination is expressed by the following formula:

[0111] S = αS1+(1 - α)S2

[0112] where the weight α is a user-defined hyperparameter. The nodes are sorted according to their scores, and the top k nodes with the highest scores are used as the pooling result, that is, the top-k selection pooling method. For the entire video, the higher the node score, the more representative the pedestrian local features learned by this block are, and the greater the impact on the video features, so it needs to be retained; on the contrary, the lower the score, the more likely this block is a background clutter area, and the smaller the impact on the video features, so it needs to be removed. Therefore, screening through node scores can more accurately select the nodes that contribute to the video features.

[0113] Specifically, first set the graph pooling ratio r ∈ (0,1] to determine the number of nodes to be retained in the model, that is, the nodes in the graph model corresponding to the first values of the retained node scores S. The following formula defines the method of taking node indices:

[0114]

[0115] where returns the first An operator with a value index, where idx represents the index.

[0116] However, in the top-k selection pooling method, only some nodes are selected as the pooling result. For the video person re-identification task, there may be some pedestrian representation information in the discarded nodes, and global features are undoubtedly beneficial to good pedestrian representations. Therefore, in order to use the information of the unselected nodes without causing too much contamination to the overall features, we choose to aggregate the features of the nodes before discarding them, make full use of the feature information of the nodes, and make the final graph embedding vector more representative.

[0117] In node feature fusion, the graph attention network GAT is selected as the node feature fusion method, which is expressed by the following formula:

[0118]

[0119] where h i is the feature vector of node i, h j represents the neighbor nodes of node i, N i is the number of adjacent nodes of node i, is the attention value between h i and h j , and W is the weight matrix. With the help of the GAT fusion method, the selected nodes can carry the feature information of adjacent nodes, and the adjacent nodes of the selected nodes are most likely to carry some pedestrian representation information. In this way, the pooling result can well represent the entire pedestrian representation.

[0120] For the input hidden layer feature H, the execution process of graph adaptive pooling is expressed by the following formula:

[0121] H out = H idx,: ⊙ S idx

[0122] where represents the block feature read through the node index and after feature fusion, ⊙ represents element-wise multiplication, · idx represents the indexing operation, and the output feature

[0123] Finally, an average pooling operation is performed on H out . For each video segment, through graph convolution and graph adaptive pooling, the video feature can be obtained.

[0124] In practical applications, the improvement effect of the video person re-identification method based on graph convolution and graph self-adaptive pooling (GCGSP) in this embodiment is verified through experiments and analyzed, including adding a graph convolutional neural network and graph self-adaptive pooling respectively. The experiments will be carried out on the determined video person re-identification baseline model ResNet50.

[0125] The video person re-identification model based on GCGSP uses ResNet50 embedded with the MALTA module as the backbone network. The size of the input image is uniformly adjusted to 256×128. The data augmentation method used in the training process is the synchronous data augmentation method. The specific means include random cropping, random horizontal flipping, and random erasing. The size of each batch of training data is set to 32 (4×8), the number of training epochs is set to 150, and the Adam optimizer is used to optimize the model. The weight decay hyperparameter of the optimizer is set to 0.0005. The initial learning rate in the training process is set to 0.01, and a fixed-epoch decay learning rate adjustment scheme is adopted. The learning rate is reduced to 0.1 times that of the previous epoch every 40 epochs. In addition, a Warm-up learning rate adjustment scheme is adopted in the training process, and the specific settings are the same as those in the baseline model experiment. Among other adjustable hyperparameters, the number of split blocks S = 4, the graph pooling ratio r = 0.5, and the score weight α = 0.4. The influence of the values of hyperparameters on the model performance will be discussed in subsequent experiments.

[0126] First, the GCGSP method is compared with the method of applying graph neural networks in video person re-identification. Second, the effectiveness of the GCGSP method in solving the video person re-identification problem is verified, and an ablation experiment is carried out on the GCGSP model.

[0127] As shown in Table 2, Table 2 lists some methods of applying graph neural networks in video person re-identification. Since most of the listed methods have not been experimented on the DukeMTMC-VideoReID dataset, only the data on the MARS dataset is analyzed. The Baseline in the experimental comparison is the baseline model constructed based on the backbone network ResNet50. Among them, Table 2 shows the experimental comparison of the method in this embodiment on MARS.

[0128] (2)

[0130] As can be seen from Table 2, the method of this embodiment has higher performance than most mainstream video pedestrian re-identification methods that apply graph neural networks, which proves that the method of this embodiment is beneficial for the video pedestrian re-identification task. In fact, the essence of most video pedestrian re-identification methods that apply graph neural networks is to extract spatio-temporal features through graph neural networks. The advantage of this embodiment compared with other methods is that it filters occlusion information and background clutter in video frames through graph adaptive pooling, and at the same time does not simply discard the regions that may contain the partial representation of pedestrians, but aggregates the features of these regions through graph attention. It is worth noting that the MGH method has much higher performance than the method of this embodiment. It must be pointed out that the number of training times of the MGH model is 300 times, which is much higher than 150 times of the method of this embodiment, which undoubtedly adds to the performance of the model. At the same time, it should also be noted that MGH introduces a hypergraph neural network to replace the graph neural network for feature extraction. One advantage of the hypergraph neural network compared with the graph neural network is that the hypergraph neural network can extract higher-order node features, which is undoubtedly beneficial to the model. However, the graph structure in MGH is not optimized accordingly, and it will inevitably be interfered by background clutter in video frames.

[0131] Secondly, to verify the effectiveness of the method of this embodiment in solving the video pedestrian re-identification problem, an ablation experiment is carried out on the GCGSP model. The Baseline in the experimental comparison is a baseline model constructed based on the backbone network ResNet50.

[0132] (3)

[0134] As shown in Table 3, Table 3 shows the experimental results of the effectiveness evaluation of each module. Based on RMNet, the global features and the graph branch are fused. In the graph branch, GCN is used to extract features. On the MARS dataset and the DukeMTMC-VideoReID dataset, the mAP is increased by 1.3% and 0.8% respectively, and the Rank-1 is increased by 1.2% and 1.0% respectively, indicating that GCN optimizes the pedestrian representation at the video level by modeling the temporal and spatial relationships of different regions between frames and utilizing the complementary correlation information between different regions. Then, the GSP module is introduced into the graph branch, that is, GCN+GSP is used to extract features. On the MARS dataset and the DukeMTMC-VideoReID dataset, the mAP is increased by 1.7% and 1.2% respectively, and the Rank-1 is increased by 1.7% and 1.5% respectively, indicating that the introduction of GSP effectively alleviates the influence of background clutter in the graph model on the overall performance of the model, making the model pay more attention to the high-quality regions in the video frames, and the pedestrian re-identification effect is better using the pedestrian features extracted from this part of the regions. The experiment proves that each module can jointly improve the performance of the model, making the extracted pedestrian features more discriminative and robust.

[0135] Next, the influence of the number of segmentation blocks on the video pedestrian re-identification network based on GCGSP is tested. Three cases of S = 2, 4, and 8 are selected for experiments respectively. The experimental datasets are the MARS and DukeMTMC-VideoReID datasets. The experimental results are shown in Table 4, and Table 4 shows the experimental results of different numbers of segmentation blocks:

[0136] (4)

[0138] Table 4 experimentally studies the effect of the change in the number of feature segmentation blocks S on the model performance, and experiments are conducted on the cases where each frame image in the video sequence is segmented into 2, 4, and 8 blocks. The experimental results show that when S = 4, the model achieves the best results, with mAP / Rank-1 accuracy reaching 82.6% / 88.4% and 95.1% / 96.0% on the MARS dataset and DukeMTMC-VideoReID dataset, respectively. This shows that when S = 4, the number of nodes in the graph model reaches a good balance, which can integrate the correlation information of local regions between frames, more accurately express the spatiotemporal relationship contained in the video sequence, and extract those high-quality pedestrian representations in a fine-grained manner, while minimizing the impact of unselected nodes, amplifying the promotion effect of global features on the model, and the generated video features are robust enough to problems such as occlusion, spatial dislocation, and background clutter. When the number of blocks is 2, the graph model extracts local information of pedestrians' body parts very roughly and cannot reflect more detailed pedestrian information, which greatly reduces the role of the graph neural network; when the number of blocks is 8, more blocks with background clutter will be cut out, and the selected nodes will be contaminated by aggregation, which will affect the model's ability to achieve better performance.

[0139] Next, we test the effect of the graph pooling ratio on the GCGSP-based video pedestrian re-identification network. We select three cases of r = 0.25, 0.5, and 0.75 for the experiment. The experimental datasets are MARS and DukeMTMC-VideoReID datasets. The experimental results are shown in Table 5:

[0140] (5)

[0142] Table 5 explores the impact of the change of the graph pooling ratio r on the model performance through experimental research. The higher the graph pooling ratio is set, the more block nodes will be retained by the GCN graph model, and the final video-level pedestrian features will be generated through node aggregation. The experiment shows that the model achieves the best recognition effect when the graph pooling ratio r = 0.5, indicating that the retained block nodes and the removed block nodes have reached the most efficient balance in mapping the characteristics of the target pedestrians, that is, the retained block nodes contain sufficiently high-quality pedestrian features, while the discarded block nodes just contain more background clutter, and the pedestrian features in the discarded nodes can also be fully utilized. When the graph pooling ratio is greater than or less than 0.5, the mAP and Rank-1 accuracy of the model are not as good as when the graph pooling ratio r = 0.5, indicating that too many or too few nodes retained in the graph model have an adverse effect on the improvement of model performance, verifying the effectiveness of the parameter setting.

[0143] Next, experiments were conducted on the impact of the scoring weight on the video pedestrian re-identification network based on GCGSP. Six cases of α = 0, 0.2, 0.4, 0.6, 0.8, and 1.0 were selected for the experiments. The experimental datasets used were the MARS and DukeMTMC-VideoReID datasets, and the experimental results are shown in Table 6:

[0144] (6)

[0146] α is an empirical value. The method of this embodiment tested the recognition effect of the model when changing the weight score α to find the optimal value. Finally, the experiment showed that the optimal value of α was 0.4, and a similar experimental effect could also be achieved when the value of α was 0.6.

[0147] In this embodiment, the video pedestrian re-identification model based on GCGSP uses ResNet50 embedded with the MALTA module as the backbone network. The size of the input image is uniformly adjusted to 256×128. The data augmentation method used in the training process is the synchronous data augmentation method, and the specific means include random cropping, random horizontal flipping, and random erasing. The size of each batch of training data is set to 32 (4×8), the number of training epochs is set to 150, the Adam optimizer is used to optimize the model, the weight decay hyperparameter of the optimizer is set to 0.0005, the initial learning rate in the training process is set to 0.01, and a learning rate adjustment scheme with fixed-epoch decay is adopted. The learning rate is reduced to 0.1 times that of the previous epoch every 40 epochs. In addition, a Warm-up learning rate adjustment scheme is adopted during the training process, and the specific settings are the same as those of the baseline model experiment. Among other adjustable hyperparameters, the number of segmentation blocks S = 4, the graph pooling ratio r = 0.5, and the scoring weight α = 0.4

[0148] All the experiments of the method of this embodiment were completed on the NVIDIA RTX 2080Ti GPU hardware device using the Pytorch software framework. The experimental datasets used were the MARS and DukeMTMC-VideoReID datasets, as shown in Table 7:

[0149] (7)

[0151] In summary, the present invention proposes a video pedestrian re-identification method based on graph convolution and graph adaptive pooling. By optimizing local feature extraction and suppressing interference information, this method significantly improves the discriminability and robustness of pedestrian identity representation. At the same time, the present invention also has strong adaptability and generalization ability, and can be widely applied to various monitoring and tracking systems, providing strong technical support for fields such as public safety and intelligent transportation.

[0152] Furthermore, the present invention introduces a graph convolutional neural network and constructs a block relationship graph based on the mutual relationship between different local features in video frames, which can dynamically capture the spatio-temporal relationship between different blocks in frames, thereby more comprehensively extracting the identity features of pedestrians. Compared with traditional methods, the pedestrian identity representation extracted by the present invention not only has higher distinctiveness and can accurately distinguish different pedestrians, but also has stronger robustness and can maintain stable performance in complex and changeable environments.

[0153] Furthermore, aiming at the common occlusion problems and background clutter interference in video pedestrian re-identification, the present invention proposes a graph adaptive pooling technique, which can dynamically adjust the pooling strategy according to the block relationship graph, effectively suppressing the interference of occlusion information and background clutter on the extraction of pedestrian identity representation. Through this mechanism, the present invention can still maintain a high recognition accuracy when the pedestrian is partially occluded or the background is complex.

[0154] Furthermore, since the present invention is based on the methods of graph convolution and graph adaptive pooling, it can flexibly adapt to video pedestrian re-identification tasks in different scenarios. Whether it is an indoor or outdoor environment, whether it is a single-person or multi-person scenario, the present invention can extract discriminative pedestrian identity representations by dynamically capturing and analyzing the spatio-temporal relationship between frames, making the present invention have strong generalization ability and can be widely applied to various monitoring and tracking systems.

[0155] Furthermore, the present invention verifies its effectiveness through a large number of experiments. The experimental results show that the present invention has achieved excellent performance on multiple video pedestrian re-identification datasets, and is superior to traditional methods in terms of both accuracy and robustness, providing strong support for the practical application of the present invention, and further demonstrating the potential and value of the present invention in the field of video pedestrian re-identification.

[0156] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not conflict, it should be considered as the scope recorded in this specification.

[0157] The above embodiments are only the preferred embodiments of the present invention, and cannot be used to limit the scope of protection of the present invention. Any non-substantive changes and substitutions made by those skilled in the art based on the present invention belong to the scope required to be protected by the present invention.

Claims

1. A video pedestrian re-identification method based on graph convolution and graph adaptive pooling, characterized in that, It includes the following steps: Extract the features of each frame image in the video sequence to obtain a frame-level feature map; Perform average pooling on the extracted video features, and use the TAM module to aggregate the average-pooled video features from the temporal dimension to obtain a global feature representation; Construct a block relationship graph, divide the frame-level feature map in the video sequence into multiple blocks, and convert each block into a feature vector, and associate the video frames by constructing a block relationship graph; Use a graph convolutional neural network to process the block relationship graph and model the spatio-temporal relationships of different regions between frames; Use graph adaptive pooling technology to optimize the topological structure of the graph model and remove nodes with low contribution rates to pedestrian features; Aggregate the node features after graph convolution and graph adaptive pooling through an average pooling operation to obtain a video feature representation; Among them, the global feature representation and the video feature representation are used to obtain the final video pedestrian feature representation through feature fusion for the video pedestrian re-identification task.

2. The method according to claim 1, wherein: The RMNet network structure is used to extract the features of each frame image in the video sequence to obtain a frame-level feature map; among them, the RMNet network is a backbone network obtained by embedding a video pedestrian re-identification network with motion attention and remote time aggregation into ResNet50; Then, the extracted frame-level feature map is fed into a two-stream network, and the two-stream network includes a global branch and a graph branch; among them, the global branch is used to extract the global features of each frame image, and the graph branch is used to extract cross-frame association features.

3. The method according to claim 1, wherein The construction of the block relationship graph includes: Let \(V = \{I_1, I_2, \ldots, I\) T \} represent the given video, where \(T\) is the number of frames sampled from the video. For each frame of the video, the frame-level feature map extracted by the backbone network is defined by the following formula: F = {F1, F2, …, F T} Among them, F i is the feature map of the i-th frame in the video, where H and W respectively represent the length and width of the spatial shape, and C represents the number of feature channels; each feature map F i is horizontally divided into S blocks with W×C as the section, and all blocks are transformed into feature vectors through average pooling operation. The feature vector of block i is denoted as x i ; Therefore, for a video with T frames, the total number of blocks is N = T·S, and the block feature vectors of the video i = 1, 2, …, N.

4. The method according to claim 3, wherein: Let the block relationship graph composed of N nodes be G(V, E), where the node v i ∈V, and the edge (v i , v j ) ∈ E. Treat each local block as a node, and the relationship between local block features is defined by the edges in E; the association between nodes is reflected by the adjacency matrix A ∈ R N×N . The relationship between the feature vectors x i and x j of any two nodes in the block relationship graph can be shown as follows:

5. The method according to claim 4, wherein: In the construction process of the adjacency matrix, in the affinity matrix, the sum of the edge values in each row should be 1; all elements of the adjacency matrix are non-negative, and the element values representing the mutual relationship between nodes range from (0, 1); the elements in the adjacency matrix A are normalized through the Softmax function, expressed as the following formula: Among them, represents the self-loop adjacency matrix, I ∈ R N×N is the identity matrix.

6. The method according to claim 5, wherein: Use the re-normalization method to approximate the graph-Laplacian matrix to facilitate the execution of graph convolution operations, expressed as the following formula: Among them, finally obtain the adjacency matrix corresponding to the block relationship graph G(V, E) , thereby further modeling the spatio-temporal relationship between blocks.

7. The method according to claim 6, wherein: In the graph branch, for a given adjacency matrix The temporal relationship between different blocks in the frames of the GCN video sequence is modeled, and the graph convolutional neural network is implemented by the following formula: Among them, is the original block feature, is the learned parameter matrix, σ(·) represents the activation function, and the graph convolutional neural network outputs the updated block feature of each video segment 8. The method according to any one of claims 1 to 7, wherein: The graph adaptive pooling technology includes pooling topology learning and node feature fusion. In the structure-based pooling topology learning, a GCN is used to calculate the structure information score S1 of the nodes, expressed as the following formula: Among them, is the normalized graph Laplacian matrix, H is the block feature optimized by the graph convolutional network, W is the learned weight vector, and σ is the activation function; Feature-based topology learning FBTL is used to calculate the feature information score, and a multi-layer perceptron MLP is used to extract node features, expressed as the following formula: S2 = σ(MLP(H)) Among them, H is the block feature optimized by GCN, σ is the activation function, and S2 is the aggregation of node features, which can be regarded as the node evaluation value; Use the structure feature topology learning SFTL component to fuse the structure information score and the feature information score to generate the final pooling topology, and the combined representation is expressed as the following formula: S = αS1+(1-α)S2 Among them, the weight α is a custom hyperparameter. The nodes are sorted according to the scores of the nodes, and the top k nodes with the highest scores are used as the pooling result, that is, the top-k selection pooling method.

9. The method according to claim 8, wherein: In node feature fusion, the graph attention network GAT is selected as the node feature fusion method, which is expressed by the following formula: Among them, h i is the feature vector of node i, h j represents the neighbor nodes of node i, and N i is the number of adjacent nodes of node i. is the attention value between h i and h j , and W is the weight matrix.

10. The method according to claim 9, wherein: Given the input hidden layer feature H, the execution process of graph adaptive pooling is expressed by the following formula: H out = H idx,: ⊙ S idx Among them represents the block feature read through the node index and after feature fusion, ⊙ represents the element-wise product, · idx represents the indexing operation, and the output feature Perform average pooling on H out For each video segment, video features can be obtained through graph convolution and graph adaptive pooling

Images