Target tracking method based on causal relationship and attention mechanism

Abstract

The invention provides a target tracking method based on a causal relationship and an attention mechanism, and the method comprises the steps: obtaining a target feature through a feature extraction sub-network, calculating a trajectory feature according to a target tracking result at a previous moment, and constructing an AND-OR logic diagram of a driving task; inputting the trajectory features and an adjacent matrix containing AND-OR logic weights into a spatial graph coding layer, the output of the layer serving as the input of a time coding layer, the output of the time coding layer serving as the output of a space-time decoder, and the output serving as one of the inputs of the decoder; initializing an adjacent matrix according to the distance between the target detected in the current frame and the track, and inputting the target feature vector and the adjacent matrix into an image convolution layer; the feature vector of the current frame and the adjacent matrix are input into a decoder, the decoder outputs a tracking result of a current frame target through a prediction layer, and finally track information is updated and fed back to the input end of the encoder for prediction at the next moment. According to the method, the causal logical relationship from the driving task to the target state is introduced as a high-level semantic meaning, and the causal logical relationship and the target trajectory information are aggregated through the graph convolutional network, so that the capability of extracting the space-time and causal dependency relationship of the target motion at different moments is enhanced, and missing detection and false detection when the target is shielded for a long time are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and in particular to the tracking technology of road vehicles, and specifically discloses a target tracking method based on causality and attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art or prior art.

[0003] Vehicle target tracking is a key step in realizing autonomous driving functions. In recent years, with the development of artificial intelligence technology, vehicle target tracking methods have also made significant progress. However, in vehicle target tracking scenarios under road conditions, the target interacts with other vehicles, pedestrians, and static buildings during driving, resulting in occlusion, which makes the target visibility change frequently, leading to missed detection and false detection of the target. Therefore, it is urgent to design a high-precision real-time tracking method that can overcome the influence of occlusion.

[0004] Early target tracking methods include methods based on feature matching (such as template matching methods) and statistical laws (such as Kalman filtering and particle filtering). With the rapid development of deep neural networks, thanks to their more expressive features, tracking accuracy and tracking robustness have been improved. In particular, two-stage target tracking methods based on convolutional neural networks (CNN) have been continuously proposed. The two-stage target tracking method first detects the position of the target box, and then associates the target box with the existing tracking trajectory. The two-stage method can select the most advanced model for each stage to combine, which can usually improve the overall tracking performance. The representative algorithm is the DeepSort model that combines the YOLO detection model, the Kalman filter, and the Hungarian associator. However, when faced with frequent / long-term occlusion of the target, the CNN-based detection model cannot provide an effective target box, which reduces the IoU matching accuracy of the association model, resulting in tracking failure. The above methods cannot solve the problem of missed detection or false detection caused by occlusion. The fundamental reason is that they are all based on the assumption that the prior data and the actual data are independent and identically distributed. When the actual data does not conform to the distribution of the prior data, the parameters obtained based on the original distribution become invalid and the method fails. For example, when the target is occluded by an object with a completely different appearance, the probability distribution of the appearance features of the target position on the image changes, or when the target is occluded by an object with a similar appearance but opposite movement direction, the probability distribution of the motion model changes. Secondly, separating detection from tracking will make the detector unable to perceive the association between frames, and only consider the spatial / appearance features of the target image, ignoring the temporal causal relationship of the moving target. Therefore, a vehicle target tracking method that can relax the constraints on the independent and identically distributed assumption and consider the target's spatiotemporal characteristics is needed to overcome the tracking failure caused by occlusion in target tracking.

[0005] AOG (And-Or Graph), first proposed by Pearl in 1984, is a hierarchical symbolic graph structure representation, consisting of And-node, Or-node, Terminal-node, and edges between nodes. If the parent node is completed only after all child node events are completed, the parent node is an And-node; if the parent node can be completed only after any child node event is completed, the parent node is an Or-node; multi-level relationships are represented by vertical arrows from top to bottom; different logical subgraphs can be obtained by selecting different child nodes at the Or-node. The steps of using AOG for prediction are to first generate multiple logical subgraphs according to the jump probability between nodes within the range of interest, then select the optimal subgraph using optimization methods or evaluation criteria, and finally use the subgraph for inference prediction. In 2018, Xu et al. proposed a causal And-Or graph (C-AOG) target tracking method, using And-nodes to represent different perspectives of the vehicle and Or-nodes to represent different parts of the vehicle. The transition probability on the Or-node in the graph is known a priori, and the subgraph with the largest posterior probability is taken as the optimal subgraph. This method can learn the causal relationship between nodes from high-level semantics to low-level features, which effectively improves the tracking accuracy. However, due to the lack of expression and calculation of deep appearance features, when the similarity between targets is high, the posterior probability value of the non-target may be greater than the posterior probability value of the real target, resulting in judgment failure.

[0006] In order to overcome the defect of ignoring the temporal relationship of the target, the multi-target tracker TransMOT proposed by Chu in 2023 combines the CNN detection model with the Transformer's attention mechanism calculation structure. By transposing the time dimension of the feature matrix of the spatial graph encoding layer to the first dimension, and then inputting it into the standard encoder layer, it realizes the calculation of the temporal information attention score function; a sparse weighted graph is constructed using the spatial relationship between the targets, and then the graph convolution model is used for information aggregation. This method can effectively handle the multi-target tracking problem, but it has the following defects: (1) It only solves the temporal causal problem by changing the arrangement of the matrix, and cannot fully express the causal relationship between the targets in the time series, and cannot learn long-term temporal dependencies; (2) In the spatial graph encoding layer, the target position is simultaneously input into the graph convolutional network (GCN) and the other two linear layers to process the position relationship of the targets in the same frame, so the ability of GCN to extract features from the data structure cannot be fully utilized.

[0007] In summary, in the existing target tracking, there is a problem that tracking errors are prone to occur when the tracking target is blocked. Summary of the invention

[0008] The purpose of the present invention is to address the shortcomings of the above-mentioned existing AOG-based target tracking method and TransMOT method, and propose a target tracking method based on causality and attention mechanism to overcome the tracking failure when the vehicle target is occluded.

[0009] Design principle: The inherent causal logic relationship between things will not change due to changes in external environmental inputs, and does not rely on the independent and identically distributed assumption. Therefore, the present invention introduces causal logic relationships to reduce dependence on assumptions. The causal relationship can be represented by a directed acyclic graph. Considering the application scenario of road driving target tracking, there is a natural hierarchical relationship between the driving task and the target state. The And-Or Graph (AOG) is selected to represent the hierarchical logical relationship from high-level semantics to low-level features. The overall design scheme of this application is as follows.

[0010] A target tracking method, the target tracking method is based on a causal AND / OR logic graph and a spatiotemporal attention mechanism, and the target tracking method includes:

[0011] S1. Use the target feature extraction model to perform target detection and feature vector extraction.

[0012] S2. Calculate the fitting curve parameters of each trajectory.

[0013] S3. Calculate the relative motion features of each trajectory including the T frame state.

[0014] S4. Construct an AND / OR logic diagram according to the trajectory curve parameters and relative motion characteristics.

[0015] S5. Construct the target feature vector and graph adjacency matrix and input them into the graph convolution layer of the spatial graph encoding layer.

[0016] S6. After position encoding, the acquired trajectory features are input into the linear layer of the spatial graph encoding layer.

[0017] S7. The output of the spatial map coding layer is used as the input of the temporal coding layer, and the output of the temporal coding layer is the output of the encoder module.

[0018] S8. Input the features of the current frame detection target and the corresponding adjacency matrix into the decoder.

[0019] S9: The decoder output is used as the input of the fully connected prediction layer. The fully connected prediction layer outputs the target tracking result: the trajectory attributes of the target, the target position box, and feeds the target tracking result back to the input ends of S2 and S3 to update the trajectory information of the input ends.

[0020] Furthermore, the target feature extraction model in step S1 is constructed based on the YOLOv10-x network, including a trajectory feature extraction subnetwork and a color feature extraction subnetwork, which are used to perform target detection and feature vector extraction on the image.

[0021] Further, step S1 includes: S11, obtaining target detection results: inputting the image into the trained YOLOv10-x network, and the network output is the coordinate values ​​of the upper left corner and lower right corner of the bounding box of each detected target, the attribute confidence score and the attribute classification result; S12, calculating the color features of the HSV space: according to the target bounding box obtained in S11, all pixels in each target bounding box are converted from the RGB color space to the HSV space, and the mean and median differences Δh and Δs of the H and S channels in the target box are calculated respectively.

[0022] Among them, HSV (Hue, Saturation, Value) is a color space created based on the intuitive characteristics of color, also known as the hexagonal cone model.

[0023] Furthermore, in step S2, the fitting curve parameters of each trajectory are calculated according to the tracking result of the previous moment, including: S21, inputting the position matrix of the trajectory of the T-frame time period, and performing cubic polynomial fitting using the least squares method; S22, outputting a three-dimensional parameter vector after eliminating the constant term from the parameters calculated in S21.

[0024] Furthermore, in step S3, the relative motion features of each track including the T frame state are calculated according to the tracking result at the previous moment, including: S31, calculating the relative position of the next frame of the track to the previous frame including the center position and image coordinates; S32, outputting the relative position vector.

[0025] Further, step S4 includes: S41, according to the curve parameters obtained in S2, when the first two dimensions of the vector are not 0, it is judged to be turning, when the first two dimensions are both 0, it is judged to be straight driving, and when the three dimensions are all 0, it is stopped; S42, after the judgment result of S41, the vehicle driving scene is judged according to the relative motion characteristics output by S3: the relative position between the last two frames and the relative angle between the first two frames are calculated, and according to the angles being in different quadrants and being a steering type, it is judged to be a left turn, a right turn, a right turn and backward, or a left turn and backward; if it is a straight driving type, the relative displacement is calculated, and the displacement is a straight line forward, otherwise it is a straight line backward; S43, and an AND / OR logic graph AOG is constructed.

[0026] Further, step S5 includes: S51, sequentially splicing the target features, relative motion features, curve parameters and the values ​​of the AND / OR logic graph adjacency matrix to generate the state vector of each target, and forming a state matrix containing all targets; S52, taking the trajectory as a node, the sum of the AND / OR logic graphs AOG of the trajectory at the previous moments is used as the time causal weight, the intersection and union ratio between different targets at the same moment is used as the space weight, and the element value of the total adjacency matrix of the target is the sum of the time causal weight and the space weight;

[0027] S53. Input the state matrix and total adjacency matrix of the target into the graph convolutional neural network GCN of the spatial graph encoding layer, and the output of the graph convolutional neural network GCN is used as the Key vector of the standard Transformer model.

[0028] Further, step S6 includes: S61, standard position encoding of the position vector, and then adding it to the original position vector to generate an encoding vector; S62, inputting the encoding vector into the linear layer of the spatial graph encoding layer as the Query vector and Value vector in the standard Transformer model. The Query vector is the query vector Q in the attention mechanism, and the Value vector is the value vector V in the self-attention mechanism. In addition, in the Transformer model, there is also a Key vector, which is the key vector K in the self-attention mechanism, corresponding to three matrices: the query matrix W Q , value matrix W V and bond matrix W K .

[0029] Further, step S7 includes: S71, the first two dimensions of the output of the spatial map coding layer are transposed and then input into the temporal coding layer; S72, the structure of the temporal coding layer is the same as the standard Transformer encoder structure.

[0030] Furthermore, the spatial graph decoding layer module has the same structure as the spatial graph encoding layer module. In step S8, the features of the target detected in the current frame and the corresponding adjacency matrix are input into the spatial decoder, the tracking result of the target in the current frame is output, and the tracking result is fed back to the input end to update the trajectory information of the input end, specifically including: 81. The input of the graph convolution layer is the graph structure of the target detection result in the current frame: the target is the node, the position coordinates of the target frame are the feature vector, and the IoU (intersection over union, referred to as IoU) between the target frames is the corresponding element value of the adjacency matrix; S82. The output of the spatial decoder is used as the Query vector to calculate the cross attention with the Key vector and the Value vector output by the temporal coding layer; S83. Output through the softmax layer. The softmax function is used to normalize the attention weight, that is, it is a normalized exponential function.

[0031] Further, step S9 includes: inputting the output of step S83 into a prediction layer; the prediction layer includes two two-layer fully connected layers, which respectively output the trajectory attributes and the target position box coordinates of the target; and feeding the trajectory attributes and the target position box coordinates back to the spatial graph encoding layer input to calculate the trajectory related features.

[0032] Compared with the prior art, the beneficial effect of the present invention is as follows: This application aims at the problem of tracking errors that are prone to occur when the target is occluded in vehicle target tracking in road environments, and designs a target tracking method based on causal AND / OR logic graph and spatiotemporal attention mechanism. The causal logic relationship between the vehicle driving task and the target state is aggregated into the spatial information of the trajectory through a graph convolutional network, and the causal logic relationship, spatial information and time information are integrated using the attention mechanism of the Transformer architecture, realizing the function of state prediction with high-level semantic guidance. Compared with the traditional method of determining whether the target is occluded by the number of unupdated beats, the present invention handles the tracking prediction problem of occluded targets more flexibly, reduces the number of target tracking mismatches, and improves tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the overall process of the target tracking method based on causality and attention mechanism of the present invention;

[0034] Figure 2 Cause and effect logic diagram for vehicle driving;

[0035] Figure 3 This is the structure diagram of the spatial encoder. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Target Tracking Methods

[0038] A target tracking method, see Figure 1-Figure 3 The method is based on causality (AND / OR logic graph) and spatiotemporal attention mechanism, and is suitable for tracking rigid body targets on the road, such as vehicles. The technical idea of ​​the present invention is to calculate the trajectory fitting parameters and motion features based on the target features in the video frame extracted from the feature extraction network, and construct the driving task AND / OR logic graph. The trajectory and target features and the AND / OR graph adjacency matrix are input into the graph convolution network of the spatial graph encoding layer, and the target features are input into the linear layer of the spatial graph encoding layer. The output is used as the input of the temporal coding layer. Finally, the decoder generates the target tracking result of the current frame and updates the trajectory information. The target tracking method includes the following steps, and the following steps do not necessarily constitute a sequential order.

[0039] Step S1: Utilize the target feature extraction model to perform target detection and feature vector extraction.

[0040] Among them, the target feature extraction model is built based on the YOLOv10-x network, including a trajectory feature extraction subnetwork and a color feature extraction subnetwork, which are used to perform target detection and feature vector extraction on images.

[0041] S11. Obtain target detection results: Input the image into the trained YOLOv10-x network. The network outputs the coordinate values ​​of the upper left and lower right corners of the bounding box of each detected target, the attribute confidence score, and the attribute classification result. The specific examples are as follows.

[0042] A frame of image I in the video t Input the trained YOLOv10-x network and output the detection result matrix of the detected N0 targets. The matrix dimension is 300×6, where 300 refers to the number of predicted boxes output and 6 is the dimension. The 6 dimensions are: The attribute confidence score and attribute classification result are The coordinates of the upper left and lower right corners of the bounding box Output the detection of N0 targets. Where score represents the score value, cls represents the classification value, t represents the time t, and i represents the i-th target (i is a positive integer).

[0043] S12. Calculate the color features of the HSV space: According to the target bounding box obtained in S11, convert all pixels in each target bounding box from the RGB color space to the HSV space, and calculate the difference Δh and Δs between the mean and the median of the H and S channels in the target box, respectively.

[0044] HSV (Hue, Saturation, Value) is a color space created based on the intuitive characteristics of color, also known as the hexagonal pyramid model. In the HSV space, H is the least sensitive to lighting, and V is very sensitive, so the V channel is discarded and only the eigenvalues ​​of the H and S channels are calculated. All pixels within each bounding box are converted from RGB color space to HSV space according to the following formula:

[0045] V = max(R, G, B);

[0046] Δ=V-min(R,G,B);

[0047]

[0048] Calculate the difference between the mean and median of the H and S channels in the target frame, Δh and Δs, respectively. V, H, and S represent the eigenvalues ​​of the three channels in the HSV space, and R, G, and B represent the eigenvalues ​​of the three channels in the RGB space.

[0049] Step S2, calculate the fitting curve parameters of each trajectory; specifically, calculate the fitting curve parameters of each trajectory according to the tracking result at the previous moment. Including: S21, input the position matrix of the trajectory in the T frame time period, and use the least squares method to perform cubic polynomial fitting; S22, remove the constant term from the parameters calculated in S21 and output a three-dimensional parameter vector.

[0050] In the specific example, the calculation process of the fitting curve parameters is as follows.

[0051] (2.1) If the target of frame t Belongs to the kth trajectory The position coordinates of the upper left corner and lower right corner of the bounding box output in step S11 are Calculate the target geometric center coordinates as:

[0052] (2.2) Trajectory k The X-axis coordinate cubic matrix of the center position of the trajectory in the T frame time period is The Y-axis coordinate matrix is Wherein, k represents the kth trajectory, which is a positive integer.

[0053] (2.3) Using a cubic polynomial Fitting trajectory curve, using least squares method to calculate various coefficients, according to the formula X = (A T A) -1 A T T solves the equation AX = parameter vector X in T. Among them, are the cubic term, quadratic form and linear term coefficients respectively.

[0054] (2.4) Finally, the first three dimensions of the output parameter vector X

[0055] Step S3, calculating the relative motion features of each track including the T frame state. Specifically, calculating the relative motion features of each track including the T frame state according to the tracking result at the previous moment includes the following steps.

[0056] S31, calculate the relative position of the next frame of the trajectory to the previous frame, including the center position and image coordinates; a specific example is: calculate the relative position of the tth frame of the kth trajectory to the previous frame:

[0057] S32, output relative position vector:

[0058] Step S4, constructing an AND / OR logic diagram according to the trajectory curve parameters and the relative motion characteristics;

[0059] S41. According to the curve parameters obtained in S2, when the first two dimensions of the vector are not 0, it is judged as turning, when the first two dimensions are 0, it is judged as straight driving, and when all three dimensions are 0, it is stopped. In the specific example, the curve parameter vector is Determine the target driving category based on the value of the elements in the vector:

[0060] (a) When The driving category is steering;

[0061] (b) When The driving category is straight-line driving;

[0062] (c) When The driving category is stop.

[0063] S42, after the result of S41 is determined, the vehicle driving scene is determined according to the relative motion characteristics output by S3: the relative position between the last two frames and the relative angle between the first two frames are calculated, and according to whether the angle is in different quadrants and is a steering type, it is determined to be a left turn, a right turn, a right turn and backward, or a left turn and backward; if it is a straight-line driving type, the relative displacement is calculated, and the displacement is a straight-line forward, otherwise it is a straight-line backward; in a specific example: based on the driving category, the vehicle driving scene is determined according to the relative motion characteristics:

[0064] (a) Calculate the relative displacement, angle cosine and relative motion respectively:

[0065] The relative displacement is:

[0066] The cosine of an angle is:

[0067] The relative motion is:

[0068] (b) Solve the angle θ between the end point of the trajectory segment and the direction of motion revealed:

[0069]

[0070] (c) Determine the driving scene based on the driving category, relative displacement and the angle of change in the direction of motion according to Table 1:

[0071] Table 1 Driving scene judgment

[0072]

[0073]

[0074] S43, construct an And Or Graph (AOG). The logical relationships in the driving task are divided into three layers from top to bottom: driving scene, trajectory behavior, and target state. The nodes in these logical relationships are represented as And Or Graph nodes. The And Or Graph structure is as follows: Figure 2 As shown in the figure, the top-level root node is the driving task of the vehicle itself. Each driving task contains multiple different driving scenarios. The driving scenarios have an "or" relationship with the driving task nodes, so the driving task is an "or" node. Each driving scenario is also an "or" node containing different trajectory behaviors. The trajectory behavior refers to the trajectory curve parameters and the relative motion characteristics of the trajectory. The trajectory behavior node is composed of the bottom-level target state. The temporal order of the target state indicates that the states are in an "and" relationship, so the trajectory behavior node is an "and" node.

[0075] Step S5: construct the target feature vector and graph adjacency matrix, and input them into the graph convolution layer of the spatial graph encoding layer.

[0076] S51, the target features, relative motion features, curve parameters and the values ​​of the adjacency matrix of the AND or logical graph are sequentially concatenated to generate the state vector of each target, and form a state matrix containing all targets. In the specific example, the OR node in the AND or logical graph represents the classification label of the sublayer, and the attribute label value I k It is the driving scene classification label corresponding to the trajectory; the target features, relative motion features, curve parameters and attribute labels are spliced ​​to obtain the target trajectory Tra k The feature vector at frame t:

[0077] S52, take the trajectory as the node, the sum of the AND / OR logic graphs AOG of the trajectory in the previous moments as the temporal causal weight, the intersection and union ratio between different targets at the same moment as the spatial weight, and the element value of the target total adjacency matrix is ​​the sum of the temporal causal weight and the spatial weight. Specifically, take the target trajectory point of the tth frame as the node to establish the graph structure G(δ,E), the node set The boundary matrix E represents the relationship between trajectory points, and the adjacency matrix A of the graph is established t :

[0078] (a) The sum of the AOGs of the trajectory at the previous moments is used as the temporal causal weight matrix A Tra ;

[0079] (b) The intersection-and-union ratio between different targets in the same frame is used as the position weight matrix A of the same-frame adjacency matrix o .

[0080] The graph adjacency matrix A is the sum of the temporal causal matrix and the position weight matrix: A t =A Tra +A o .

[0081] S53, input the state matrix and total adjacency matrix of the target into the graph convolutional network (GCN) of the spatial graph encoding layer, and the output of the graph convolutional neural network (GCN) is used as the key vector of the standard Transformer model. The movement of the target is not only affected by its own state at the previous moment, but also by the state of other targets. Each node in the two-layer GCN can collect information from its neighbor nodes and non-neighbor nodes, and the two-layer GCN network is selected accordingly. Contains N g The feature matrix of the trajectory The target feature matrix Δ t and the adjacency matrix A t Input a two-layer GCN network. Where σ represents the feature vector of the trajectory at time t.

[0082] Step S6: After position encoding, the acquired trajectory features are input into the linear projection layer of the spatial graph encoding layer.

[0083] S61. Perform standard position encoding on the position vector:

[0084] in is the coordinate position, k is the trajectory number, d model is the total number of trajectories.

[0085] Then add it to the original position vector to generate the encoding vector of each target After sorting, we can get N g ×4-dimensional encoding matrix

[0086] S62. Input the encoded vector into the linear layer of the spatial graph encoding layer as the Query vector and the Value vector in the standard Transformer model. The Query vector is the query vector Q in the self-attention mechanism, and the Value vector is the value vector V in the self-attention mechanism.

[0087] First Input two linear projection layers respectively and output Q t and V t ; Then H t , Q t 、V t As the Key vector, Query vector and Value vector respectively, calculate the dot product attention value: Then, the dot product attention value and the trajectory feature vector are input into the residual connection and normalization layer, and its output is then passed through the feedforward layer and another residual connection and normalization to obtain the output of the spatial graph encoding layer.

[0088] Step S7: The output of the spatial image coding layer is used as the input of the temporal coding layer, and the output of the temporal coding layer is the output of the encoder module. The following steps are included.

[0089] S71, the first two dimensions of the output of the spatial image coding layer are transposed and then input into the temporal coding layer; S72, the structure of the temporal coding layer is the same as the standard Transformer encoder structure. The specific example is as follows.

[0090] Spatial image encoding layer output The T frame representing the time series is in the second dimension, and after transposing the first two dimensions, we get Put T frames representing the time series in the first dimension.

[0091] Then Enter the standard Transformer encoder layer, whose structure is as follows Figure 3 shown.

[0092] Finally, the temporal encoding layer outputs

[0093] Step S8, input the features of the current frame detection target and the corresponding adjacency matrix into the decoder. The spatial decoder module has the same structure as the spatial graph coding layer module. In step S8, the features of the current frame detection target and the corresponding adjacency matrix are input into the spatial decoder, the tracking result of the current frame target is output, and the tracking result is fed back to the input end to update the trajectory information of the input end, which specifically includes the following steps.

[0094] S81. The input of the graph convolution layer is the graph structure G(O,E) of the target detection result of the current frame: the target is the node, the position coordinates of the target frame are the feature vector, and the IoU between the target frames is the corresponding element value of the adjacency matrix; specific example: The characteristic matrix composed of the characteristic vectors of the node and the target frame position coordinates is Adjacency Matrix The corresponding element value is the IoU between the target boxes; the input of the linear layer is m t Among them, the uppercase bold O represents the node set, and the lowercase bold o in the set is all the target feature vectors at time t.

[0095] S82, output of spatial decoder As the query vector, the temporal encoding layer output As the Key vector and Value vector, calculate the cross attention:

[0096] Where dtemp represents the temporary value of the decoder at the current moment, out represents the output; in ten, t represents time, and en represents the encoder. yes The dimension of .

[0097] S83, output through softmax layer

[0098] Step S9: The decoder output is used as the input of the fully connected prediction layer, and the fully connected prediction layer outputs the target tracking result: the trajectory attribute of the target, the target position box, and feeds the target tracking result back to the input terminals S2 and S3 to update the trajectory information of the input terminals. The specific example includes the following steps.

[0099] S91, the output of step S83 Input prediction layer;

[0100] S92, the prediction layer consists of two fully connected layers, which output the trajectory attributes of the target respectively and the target location frame coordinates

[0101] S93, feeding back the trajectory attributes and the target position frame coordinates to the spatial graph encoding layer input to calculate trajectory related features.

[0102] In summary, the present invention is a target tracking method based on causal AND / OR logic graph and spatiotemporal attention mechanism, and the implementation scheme is: first, the target features are obtained by using the feature extraction subnetwork, the trajectory features are calculated according to the target tracking results of the previous moment, and the AND / OR logic graph of the driving task is constructed; the adjacency matrix is ​​initialized according to the distance between the target detected in the current frame and the trajectory, and then the target feature vector and the adjacency matrix are input into the graph convolution layer, and the target feature vector is input into the linear layer after position encoding; the feature vector of the target detected in the current frame and the corresponding adjacency matrix are input into the decoder, and the decoder output is output through the prediction layer to output the tracking result of the target in the current frame: the trajectory attribute of the target and the target position box; finally, the trajectory information is updated and fed back to the encoder input for prediction at the next moment. The present invention introduces the causal logic relationship from the driving task to the target state as a high-level semantics, aggregates the causal logic relationship with the target trajectory information through the graph convolution network, enhances the network's ability to extract the spatiotemporal and causal dependencies of the target movement at different times, and effectively avoids missed detection and false detection when the target is blocked for a long time.

[0103] Target tracking system

[0104] A target tracking system for executing the aforementioned target tracking method, the target tracking system comprising:

[0105] Position encoding: The acquired trajectory features are position encoded and added to the original position vector to generate the encoding vector of each target, and it is input into the linear projection layer of the spatial graph encoding layer to represent the position relationship between the trajectory and other trajectories in the current frame.

[0106] The spatial decoder has the same structure as the spatial graph encoding layer. It mainly consists of a graph convolutional network and two linear projection layers, which are used to map the trajectory and target features calculated by the spatiotemporal encoder into the association matrix between the target and the trajectory.

[0107] The prediction layer adopts a two-layer fully connected prediction layer, including a target trajectory attribute prediction layer and a target position box prediction layer, which is used to predict the target position and trajectory attributes.

[0108] The spatiotemporal encoder, including the spatial graph encoding layer and the temporal encoding layer, is used to extract the features of trajectories and targets and aggregate their corresponding causal logical relationships.

[0109] The above method and system can effectively solve the problem of tracking errors that easily occur when the tracking target is blocked in the prior art.

[0110] Computer readable storage medium

[0111] The present invention also provides a computer-readable storage medium on which computer instructions are stored, and when the computer instructions are executed, the steps of the above method are executed. The method is described in detail in the above part and will not be described here.

[0112] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carriers.

[0113] The computer program coding required for the operation of each part of the application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, VisualBasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages, etc. The program coding can be run completely on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and partly on the remote computer, or run completely on the remote computer or processing device. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0114] terminal

[0115] The present invention also provides a terminal, including a memory and a processor, wherein the memory stores data provider information and computer instructions that can be run on the processor, and the processor executes the steps of the above method when running the computer instructions. The method is described in detail in the above section and will not be repeated here.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target tracking method based on causality and attention mechanism, characterized in that: The target tracking method comprises: S1. Target detection and feature vector extraction using target feature extraction model; S2, calculating the fitting curve parameters of each trajectory; S3, calculating the relative motion characteristics of each trajectory including the T frame state; S4, constructing an AND / OR logic diagram according to trajectory curve parameters and relative motion characteristics; S5, construct the target feature vector and graph adjacency matrix, and input them into the graph convolution layer of the spatial graph encoding layer; S6, inputting the acquired trajectory features into the linear layer of the spatial graph encoding layer after position encoding; S7, the output of the spatial image coding layer is used as the input of the temporal coding layer, and the output of the temporal coding layer is the output of the encoder module; S8, input the features of the current frame detection target and the corresponding adjacency matrix into the decoder; S9: The decoder output is used as the input of the fully connected prediction layer. The fully connected prediction layer outputs the target tracking result: the trajectory attributes of the target, the target position box, and feeds the target tracking result back to the input ends of S2 and S3 to update the trajectory information of the input ends.

2. The target tracking method according to claim 1, characterized in that: The target feature extraction model in step S1 is built based on the YOLOv10-x network, including a trajectory feature extraction subnetwork and a color feature extraction subnetwork, which are used to perform target detection and feature vector extraction on the image.

3. The target tracking method according to claim 2, characterized in that: Step S1 includes: S11. Obtain target detection results: Input the image into the trained YOLOv10-x network. The network outputs the coordinate values ​​of the upper left corner and lower right corner of the bounding box of each detected target, the attribute confidence score, and the attribute classification result. S12. Calculate the color features of the HSV space: According to the target bounding box obtained in S11, convert all pixels in each target bounding box from the RGB color space to the HSV space, and calculate the difference Δh and Δs between the mean and the median of the H and S channels in the target box, respectively.

4. The target tracking method according to claim 1, characterized in that: In step S2, the fitting curve parameters of each trajectory are calculated according to the tracking result at the previous moment, including: S21, input the position matrix of the trajectory in the T frame time period, and perform cubic polynomial fitting using the least squares method; S22, output a three-dimensional parameter vector after removing the constant term from the parameters calculated in S21.

5. The target tracking method according to claim 1, characterized in that: In step S3, the relative motion features of each track including the T-frame state are calculated according to the tracking result at the previous moment, including: S31, calculating the relative position of the next frame of the trajectory to the previous frame, including the center position and image coordinates; S32. Output relative position vector.

6. The target tracking method according to claim 1, characterized in that: Step S4 includes: S41, according to the curve parameters obtained in S2, when the first two dimensions of the vector are not 0, it is judged as turning, when the first two dimensions are 0, it is judged as straight driving, and when the three dimensions are all 0, it is stopped; S42, after the result of S41 is determined, the vehicle driving scene is determined according to the relative motion characteristics output by S3: the relative position between the last two frames and the relative angle between the first two frames are calculated, and if the angles are in different quadrants and it is a turning type, it is determined to be a left turn, a right turn, a right turn and backward, or a left turn and backward; if it is a straight-line driving type, the relative displacement is calculated, and the displacement is a straight-line forward, otherwise it is a straight-line backward; S43. Construct an AND / OR logic graph AOG.

7. The target tracking method according to claim 1, characterized in that: Step S5 includes: S51, sequentially concatenate the target features, relative motion features, curve parameters, and the values ​​of the AND or logic graph adjacency matrix to generate a state vector for each target, thereby forming a state matrix containing all targets; S52, taking the trajectory as a node, the sum of the AND / OR logic graphs AOG of the trajectory at the previous moments is taken as the temporal causal weight, the intersection and union ratio between different targets at the same moment is taken as the spatial weight, and the element value of the total adjacency matrix of the target is the sum of the temporal causal weight and the spatial weight; S53. Input the state matrix and total adjacency matrix of the target into the graph convolutional neural network GCN of the spatial graph encoding layer, and the output of the graph convolutional neural network GCN is used as the Key vector of the standard Transformer model.

8. The target tracking method according to claim 1, characterized in that: Step S6 includes: S61, performing standard position encoding on the position vector, and then adding the encoded vector to the original position vector to generate an encoded vector; S62. Input the encoded vector into the linear layer of the spatial graph encoding layer as the Query vector and Value vector in the standard Transformer model respectively.

9. The target tracking method according to claim 1, characterized in that: Step S7 includes: S71, the first two dimensions of the output of the spatial image coding layer are transposed and input into the temporal coding layer; S72, the structure of the temporal coding layer is the same as the standard Transformer encoder structure.

10. The target tracking method according to claim 1, characterized in that: The spatial decoder module has the same structure as the spatial graph coding layer module. In step S8, the features of the detected target in the current frame and the corresponding adjacency matrix are input into the spatial decoder, the tracking result of the target in the current frame is output, and the tracking result is fed back to the input end to update the trajectory information of the input end, which specifically includes: S81, the input of the graph convolution layer is the graph structure of the target detection result of the current frame: the target is the node, the position coordinates of the target frame are the feature vector, and the IoU between the target frames is the corresponding element value of the adjacency matrix; S82, the output of the spatial decoder is used as the Query vector, and the cross attention is calculated with the Key vector and Value vector output by the temporal encoding layer; S83, output through the softmax layer.

11. The target tracking method according to claim 10, characterized in that: Step S9 includes: The output of step S83 is input into the prediction layer; The prediction layer consists of two two-layer fully connected layers, which output the target trajectory attributes and the target position box coordinates respectively; The trajectory attributes and target location box coordinates are fed back to the spatial graph encoding layer input to calculate trajectory related features.

Images