A method for RGBT pedestrian multi-target tracking based on a graph neural network

Abstract

The present application belongs to the technical field of multi-modal multi-target tracking, and particularly relates to a kind of RGBT pedestrian multi-target tracking method based on graph neural network.The present application is based on graph neural network technology and multi-target tracking network technology, and the joint tracking of visible light image target and infrared image target is completed.The specific method is first to realize the fusion of visible light image and infrared image information through RGBT feature fusion module, then to realize the detection of pedestrian target through CenterNet detection network, and finally to realize the matching and association between detection targets through the feature association module based on graph neural network, to finally realize RGBT pedestrian multi-target tracking.Compared with the traditional single-modal multi-target tracking algorithm, the method proposed in the present application still has strong stability when tracking targets are in complex environments such as strong light, weak light and darkness.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal multi-target tracking, and particularly relates to an RGBT pedestrian multi-target tracking method based on a graph neural network. BACKGROUND

[0002] The main task of target tracking is to determine the position of a target in a continuous video sequence. In practical applications, target tracking technology has many application scenarios, such as video monitoring, automatic driving, auxiliary driving in the civil field, and air target tracking, related target reconnaissance, and early warning in the military field.

[0003] Currently, commonly used tracking data is visible light image data, which is received by a visible light camera through reflection of visible light from a target surface, and target and environmental information is saved through a grayscale image or an RGB image. Such an image depends on the external environment where the target is located, and when the target is in a weak light, strong light, or dark environment, the visible light camera will be difficult to capture the information of the target, thereby affecting the effect of target tracking. Therefore, using a single type of data source for tracking cannot meet the tracking needs of all scenarios.

[0004] To solve the above problems, infrared images can complement information. An infrared camera forms an infrared image of a target by capturing infrared radiation information of the target. Due to different imaging mechanisms, visible light images and infrared images can complement information in many scenarios: compared with visible light images, the imaging effect of infrared images is not affected by the light environment of the target, and even if the target is in a strong or weak light, dark environment, the infrared image can still capture target information. At the same time, the infrared image may also face the problem of thermal cross in some environments, and the visible light image can supplement the visible light information of the target; generally, the imaging resolution of the infrared image is low, and the signal-to-noise ratio is low, and the target lacks detailed information, while the visible light image has high resolution and can provide rich detailed texture information of the target; the infrared image represents the thermal radiation information of the target in grayscale, and therefore the visible light image can supplement the color information of the target that is lacking in the infrared image.

[0005] Currently, in the field of multi-target, there are few RGB multi-target tracking algorithms, which have not attracted enough attention from relevant researchers. In multi-modal target tracking, although different modal images introduce more target information, they may also introduce noise in different modal images. It is more efficient and reasonable to use different modal information and adopt a more reasonable information fusion method to solve the challenge in this field. At the same time, the similar appearance of pedestrian multi-targets, frequent ID switching of a single target, and occlusion bring certain difficulty to the determination of multi-target tracking trajectories. SUMMARY

[0006] In order to realize RGBT pedestrian multi-target tracking, the present application proposes a kind of RGBT pedestrian multi-target tracking method based on graph neural network for the above problems.The present application realizes the feature correlation matching between targets by fusing RGBT image information and using graph neural network technology, compared with traditional visible light multi-target tracking algorithm, the present application has more extensive application scene, when the tracking target is in strong light, weak light, darkness and other complex environments, still has stronger stability.

[0007] The technical scheme of the present application is:

[0008] A kind of RGBT pedestrian multi-target tracking algorithm based on graph neural network, comprising the following steps:

[0009] S1, training dataset construction: using data expansion method, the public pedestrian visible light multi-target tracking dataset is expanded using pseudo-infrared data generation method, and RGBT multi-target tracking training dataset is obtained;

[0010] S2, construct RGBT multi-target tracking model, specifically:

[0011] S21, construct RGBT feature fusion network:

[0012] This part adopts DenseNet network structure, two different modal images are directly spliced together in channel dimension when inputting network, in the process of fusion, the network will evaluate the effectiveness of information in the image for different modal images, and realizes the fusion of two kinds of information using adaptive information protection degree;

[0013] S22, construct detection network:

[0014] For the above fusion result, the CenterNet detection network is used to realize pedestrian target detection;

[0015] S23, construct feature correlation module based on graph neural network:

[0016] On the basis of the above detection result, the feature correlation module in the step is used to realize the feature correlation between pedestrian targets.The specific steps are as follows:

[0017] S231, initialize graph model:

[0018] The detection candidate boxes in the continuous frames f1, f2 are abstracted as node sets N1, N2 in a graph model, and the edges of the graph model are constructed by measuring the similarity between the nodes in N1 and N2. The nodes are generated by selecting the top K detection candidate boxes according to the scores as the nodes in the graph model, and the edges are generated by comprehensively measuring the similarity between the nodes in N1 and N2, specifically including the appearance similarity and spatial similarity between the detection boxes. In order to deal with the situation that some nodes cannot be connected during the edge construction process, the nodes are set as cache nodes and saved in N1, and a certain cache time is set to keep the information within a certain number of continuous frames, so as to deal with the situation that long-term association needs to be realized.

[0019] S232, graph model information propagation update:

[0020] Through the information propagation process in this step, the features of the nodes and edges in the graph model are continuously updated. Specifically, it can be divided into two steps of node feature update and edge feature update. In the node update process, the features of the nodes are updated by aggregating the features of the adjacent nodes and edges, and the calculation formula is as follows:

[0021]

[0022] wherein, v i n+1 and v i n represent the updated features of node i in the n+1th and nth rounds respectively, v j n represents the n-th round feature of the j-th neighbor node of the above node, e i,j n represents the n-th round feature of the edge between nodes i and j. The features of node i itself, its neighbor node j and the neighbor edge are concatenated and input into f node , which is composed of a fully connected layer, a LN (Layer Normalization) normalization layer and a ReLU activation function. After the features of all neighbor nodes of node i are calculated by f node , they are aggregated in an accumulative manner, and then divided by the number of neighbor nodes N i of node i. The obtained feature tensor is then input into a network f node with the same structure as f out , and the updated node feature is obtained.

[0023] In the edge update process, the features of the connected nodes and the initialized features of the edge itself are aggregated to update the features of the edge, and the calculation formula is as follows:

[0024] e i,j n+1 = f edge ([ei,j 0 e i,j n v i n+1 v j n+1 ])

[0025] where e i,j n+1 and e i,j n denote the edge e i,j the updated features of the n+1th round and the nth round, e i,j 0 denote the initialized features of the edge, v i n+1 and v j n+1 denote the features of the connected nodes i and j respectively. f edge The network structure is the same as f node in the above steps.

[0026] S233, updating the graph model output:

[0027] The edge features and node features in the last round of updated graph model will be output as prediction scores through a softmax layer respectively. The node prediction score evaluates whether the node corresponds to the target that exists in the matching relationship in the graph. If the score is lower than the threshold, it indicates that the node is a redundant node without a matching target. The edge prediction score is used to measure the degree of association between the two targets corresponding to the edge. The higher the score, the greater the probability of the existence of the matching relationship between the targets.

[0028] S24, target matching:

[0029] For the edge prediction score in the above calculation process, the following optimization method is used to calculate the optimal matching scheme:

[0030]

[0031]

[0032] where m represents the number of tracked targets, and n represents the number of targets in the current frame. a ij denotes the association cost between the ith tracked target and the jth detected target, that is, the coefficient of the edge predicted by the graph neural network. X represents the assignment matrix, x ijFor the elements in the matrix, the association state between the i th tracking target and the j th detection target is represented, and the value range is "0-1", "1" represents the association of the two, and "0" represents the disassociation of the two. The sum of each row of the distribution matrix X is 1, which restricts that each detection target can only be associated with one tracking target, and the sum of each column is also 1, which restricts that each tracking target can only be associated with one tracking target.

[0033] S3, training phase:

[0034] Using the data in step S1, the above model is trained, and the stochastic gradient descent method is used to train until the loss function converges; and a trained tracking model is obtained.

[0035] S4, tracking phase:

[0036] Using the trained model to track the real RGBT pedestrian video sequence.

[0037] The beneficial effects of the present application are:

[0038] The present application utilizes the complementarity of visible light and infrared image information, and designs an RGBT pedestrian multi-target tracking algorithm based on a graph neural network, which can realize stable tracking of pedestrian targets in complex scenes such as severe light changes and low visibility, and enhances the stability and robustness of the tracking model. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is the overall framework of the algorithm.

[0040] Figure 2 It is the structure of the RGBT fusion network.

[0041] Figure 3 It is the specific structure of the graph data association network.

[0042] Figure 4 It is an information propagation process diagram. DETAILED DESCRIPTION

[0043] The technical solutions of the present application will be described in detail below with reference to the drawings.

[0044] The present application proposes an RGBT pedestrian multi-target tracking algorithm based on a graph neural network. By fusing RGBT image information, the present application realizes feature association matching between targets by using graph neural network technology. Compared with traditional visible light multi-target tracking algorithms, the present application has a wider application scenario and still has strong stability when the tracking target is in a complex environment such as strong light, weak light and darkness.

[0045] Figure 1The overall framework of the algorithm in the application is shown. The input of the network is the continuous frame RGBT image. The RGBT image fusion network module is used to realize the information fusion of different modal images. Then the detection network is used to determine the size and position of the pedestrian target in the image. Finally, the graph data association network is used to realize the association between the detected targets.

[0046] The structure of the RGBT fusion network is shown in Figure 2 Two different modal images are directly spliced together, and then the DenseNet is used to realize the fusion of the two modal images. In the process of RGBT image fusion, the network evaluates the effectiveness of the information in the image for different modal images, and realizes the fusion of the two kinds of information by using adaptive information protection degree. The DenseNet structure in the network is shown in the figure, wherein the convolutional layer part adopts a convolutional layer with a 3*3 size convolution kernel, a step of 1 and mirror padding, a LeakyReLU activation function is connected after the convolutional layer, and a tanh activation function is connected as the activation function of the last convolutional layer.

[0047] The CenterNet target detector is used in the detector part to determine the position and size of the pedestrian target in the fusion result.

[0048] The process of initializing the graph model is as follows: the detection candidate boxes in the continuous frames f1 and f2 are abstracted as node sets N1 and N2 in the graph model, and the edges of the graph model are constructed by measuring the similarity between the nodes in N1 and N2. The node generation method is to select the top K detection candidate boxes as the nodes in the graph model, and the edge generation method is to comprehensively measure the similarity between the nodes in n1 and n2, including the appearance similarity and spatial similarity between the detection boxes. The appearance similarity is measured by the cosine similarity between the target feature tensors, and the spatial similarity is measured by calculating the center distance, intersection over union and width-height ratio between the detection boxes. In addition, in order to deal with the situation that some nodes cannot be connected in the edge construction process, the nodes are set as buffer nodes and saved in N1, and a certain buffer time is set to make the information of the nodes be preserved in a certain continuous frame, so as to deal with the situation that long-term association needs to be realized.

[0049] Figure 3The specific structure of the graph data associated network is shown. The feature information on the initialized graph model is constantly updated through the information propagation process, and gradually has higher order information. The overall feature updating process is divided into node feature updating and edge feature updating two processes. The edge features and node features in the graph model after the last round of updating will be output as prediction scores through a softmax layer. Not all cached nodes have a corresponding relationship in the graph, so in this paper, the node prediction score is used to evaluate whether the node corresponding target exists in the graph. If the score is lower than the threshold, it means that the node is a redundant node without a matching target. The edge prediction score is used to measure the degree of association between the two targets corresponding to the edge. The higher the score, the greater the probability of matching relationship between the targets. By finding the optimal matching relationship of the edge prediction score, the final tracking result can be obtained.

[0050] Figure 4 The information propagation process is shown. Through the information propagation process in this step, the features of nodes and edges in the graph model are constantly updated. Specifically, it can be divided into two steps of node feature updating and edge feature updating. In the node updating process, the features of its adjacent nodes and edges are aggregated to update its features, and the calculation formula is as follows:

[0051]

[0052] where v i n+1 and v i n represent the updated features of node i after the n+1th and nth rounds, respectively, v j n represents the n-th round feature of the j-th neighbor node of the above node, e i,j n represents the n-th round feature of the edge between nodes i and j. The features of node i itself and its neighbor node j, and the features of adjacent edges are concatenated and input into f node , which consists of a fully connected layer, a LN (Layer Normalization) normalization layer, and a ReLU activation function. After the features of all neighbor nodes of node i are calculated by f node , they are aggregated in an accumulative manner, and then divided by the number of neighbor nodes N i of node i. The obtained feature tensor is then passed through a network f node with the same structure as f out , and the updated node feature is obtained.

[0053] In the edge updating process, the features of its connected nodes and its own initialized features are aggregated to update its features, and the calculation formula is as follows:

[0054] ei,j n+1 = f edge ([e i,j 0 , e i,j n , v i n+1 , v j n+1 ])

[0055] where e i,j n+1 and e i,j n denote the edge e i,j the updated features of the (n+1)-th and n-th rounds, e i,j 0 denote the initialized features of the edge, v i n+1 and v j n+1 denote the features of its connected nodes i and j, respectively. f edge The network structure is the same as f node in the above steps.

Claims

1. A RGBT pedestrian multi-target tracking method based on a graph neural network, characterized in that, Comprise the following steps: S1, training data set construction: using data expansion method, using pseudo-infrared data generation method to expand the public pedestrian visible light multi-target tracking dataset, obtain the RGBT multi-target tracking training dataset; S2, build RGBT multi-target tracking model, including RGBT feature fusion network, detection network and graph data association network, specifically: S21, adopt DenseNet network to build RGBT feature fusion network, the input of DenseNet network is the spliced pedestrian visible light image and infrared data image, DenseNet network evaluates the effectiveness of information in the image of different modal images; S22, adopt CenterNet detection network to build detection network, realize pedestrian target detection, the input of CenterNet detection network is the fusion result output by RGBT feature fusion network; S23, build graph data association network based on graph neural network, realize the feature association between pedestrian targets according to the detection result, the specific steps are as follows: S231, initialize graph model: abstraction of detection candidate boxes in consecutive frames , into a node set in a graph model , , edges of the graph model are constructed by measuring similarity between nodes in and ; nodes are generated by selecting top K detection candidate boxes as nodes in the graph model, and edges are generated by measuring similarity between nodes in , , which specifically includes appearance similarity and spatial similarity between detection boxes; for the case that some nodes cannot be connected during edge construction, they are set as buffer nodes saved in N1, and a buffer time is set. S232, graph model information propagation update: Including node feature update and edge feature update; In the process of node update, the characteristics of its adjacent nodes and edges are aggregated to update its characteristics: ， wherein, and respectively represent the updated features of the node i in the n+1th and nth rounds, represent the jth neighbor node of the node in the n round feature, represent the n round feature of the edge between the two nodes i and j; the self feature of the node i is spliced with the features of the adjacent node j and the adjacent edge, and input into , The network is composed of a fully connected layer, a LN normalization layer and a ReLU activation function; after the features of all adjacent nodes of the node i are calculated by , they are aggregated in an accumulative manner, and then divided by the number of adjacent nodes of the node i , and the obtained feature tensor is input into a network with the same structure as , so as to obtain the updated node feature. In the process of edge update, the characteristics of its connected node characteristics and its own initialization characteristics are aggregated to update its characteristics: ， wherein, and denotes an edge the updated features of the n+1th and nth rounds, denotes the initialized features of the edge, and denotes the features of its connected nodes i and j, respectively, the network structure is the same as the network structure. S233, update the output of the graph model: The edge features and node features in the last round of updated graph model will be output as prediction scores through a softmax layer respectively; The node prediction score evaluates whether the node corresponding target exists in the matching relationship in the graph, if the score is lower than the threshold, it indicates that the node is a redundant node without matching target; The edge prediction score is used to measure the association degree between the two targets corresponding to the edge, the higher the score, the greater the probability of the existence of the matching relationship between the targets; S24, target matching: According to the obtained prediction score, the optimal matching scheme is obtained: ， ， wherein m represents the number of tracked targets, and n represents the number of targets in the current frame; represents the association cost between the ith tracked target and the jth detected target, i.e., the coefficient of the edge predicted by the graph neural network; X represents the assignment matrix, is an element in the matrix, representing the association state between the ith tracked target and the jth detected target, and its value ranges from "0-1"; "1" indicates that the two are associated, and "0" indicates that the two are not associated; the sum of each row of the assignment matrix X is 1, which constrains each detected target to be associated with only one tracked target, and the sum of each column is also 1, which constrains each tracked target to be associated with only one tracked target; S3, training stage: Use the training data in S1 to train the RGBT multi-target tracking model established in S2, and train it to convergence by using the stochastic gradient descent method; Obtain the trained tracking model; S4, tracking stage: Use the trained model to track the real RGBT pedestrian video sequence.

Images