Target tracking method, device and equipment in multi-target multi-source information fusion scene

Abstract

The invention discloses a target tracking method, device and equipment in a multi-target multi-source information fusion scene, and relates to the field of target tracking. The method comprises the following steps: acquiring information data of a to-be-tracked target group; performing parallel processing on the information data by adopting a DBSCAN algorithm and a neural network to obtain a track association result; the track association result comprises a track measurement fragment and a target to which the track measurement fragment belongs; segmenting the track measurement segment based on a preset sliding window and step length, and determining a track residual error according to sub-track information obtained through segmentation by adopting an error feature estimation algorithm; based on the trained MsRFCFN, determining a fusion residual error according to the track residual error; and performing track reconstruction and fusion splicing processing according to the fusion residual error to obtain a fusion track for realizing target tracking. The invention aims to improve the fusion precision so as to improve the target tracking precision.

Description

Technical Field

[0001] The present application relates to the field of target tracking, and in particular to a target tracking method, apparatus and device in a multi-target and multi-source information fusion scenario. Background Art

[0002] Since the inception of aerospace engineering research in the 1960s, sensor target tracking has been a global focus. Its core approach involves using sensors to perceive the target and environment within a monitoring area. Based on noise-corrupted measurement data, the target's trajectory is correlated, fused, and filtered to estimate the target's motion state and track it within a specified timeframe. Currently, this technology has been widely applied in areas such as maritime and airborne early warning, surveillance and security, and autonomous driving, demonstrating its significant application value.

[0003] In practical applications, targets are diverse, numerous, and highly mobile, while also experiencing significant environmental noise. The limited perception and tracking capabilities of a single sensor make it difficult to cope with complex situations and meet the requirements for high precision, real-time performance, and robustness. Therefore, it is necessary to fully leverage the detection advantages of multi-source sensors and integrate multi-dimensional and multi-layered information, algorithms, and computing power for target tracking. Multi-source sensor information fusion systems can integrate sensor data from different types and multiple platforms. This not only complements the information from each sensor to obtain richer situational data, but also integrates measurement information, reducing errors from individual sensors and improving information reliability.

[0004] Due to inherent sensor errors, noise, and obstruction caused by obstacles, as well as the potential for overlapping tracks of multiple targets, the track segments collected by the sensors deviate from the actual target positions, reducing tracking accuracy. Therefore, it is particularly necessary to correlate the track segments collected by the sensors. Track correlation, a key prerequisite for multi-source sensor information fusion, aims to integrate and classify track segments from different sensors, identify observation data of the same target on multiple platforms, and establish a corresponding relationship between them and the target's actual position. Currently, common track correlation algorithms include nearest neighbor, K-nearest neighbor, and global nearest neighbor algorithms. These methods achieve target correlation by selecting measurement segments with the smallest statistical distance, resulting in a relatively simple computational process. However, in densely populated target areas, when targets have similar motion states or when there is overlap or separation, the correlation accuracy of these algorithms drops significantly, making it difficult to meet the requirements of actual tracking tasks.

[0005] Another approach is the probabilistic data association algorithm, which treats all measurement segments within the tracking area as real data and designs a criterion to calculate the probability that each observation is from a real target. Compared to the global nearest neighbor algorithm, this method not only has better association results but also has stronger anti-interference capabilities. However, it is mainly suitable for single-target tracking and performs poorly in situations where multiple targets have dense echoes. To this end, Bar-Shalom et al. proposed a joint probabilistic data association algorithm based on the probabilistic data association algorithm. This algorithm determines the optimal association pair by calculating the conditional probability between each set of target states and measurement segments. However, as the number of targets, sensor platforms, and environmental clutter increases, the computational complexity rapidly expands, making it difficult to meet real-time requirements. Summary of the Invention

[0006] The purpose of this application is to provide a target tracking method, device and equipment in a multi-target and multi-source information fusion scenario, which can improve the fusion accuracy and thus enhance the accuracy of target tracking.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a target tracking method in a multi-target multi-source information fusion scenario, comprising:

[0009] Acquire information data of a target group to be tracked; the information data is acquired based on multiple observation platforms; the information data includes: visible light images and track data information; the target group to be tracked includes multiple targets to be tracked;

[0010] The information data is processed in parallel using the DBSCAN algorithm and a neural network to obtain a track association result; the track association result includes: a track measurement segment and a target to which the track measurement segment belongs;

[0011] Segmenting the track measurement segments based on a preset sliding window and step size, and determining the track residual based on the segmented sub-track information using an error characteristic estimation algorithm;

[0012] Based on the trained MsRFCFN network, a fusion residual is determined according to the track residual; the trained MsRFCFN network is obtained by training parameters of the MsRFCFN network based on a loss function and adopting a BP algorithm with a goal of minimizing the loss convergence;

[0013] Track reconstruction and fusion splicing are performed based on the fusion residual to obtain a fused track; the fused track is used to achieve target tracking.

[0014] In a second aspect, the present application provides a target tracking device in a multi-target multi-source information fusion scenario, comprising:

[0015] An information data acquisition module is used to acquire information data of a target group to be tracked; the information data is acquired based on multiple observation platforms; the information data includes: visible light images and track data information; the target group to be tracked includes multiple targets to be tracked;

[0016] a processing module, configured to perform parallel processing on the information data using a DBSCAN algorithm and a neural network to obtain a track association result; the track association result includes: a track measurement segment and a target to which the track measurement segment belongs;

[0017] a track residual determination module, configured to segment the track measurement segments based on a preset sliding window and step size, and determine the track residual based on the segmented sub-track information using an error characteristic estimation algorithm;

[0018] A fusion residual determination module is used to determine a fusion residual based on the track residual based on a trained MsRFCFN network; the trained MsRFCFN network is obtained by training the parameters of the MsRFCFN network based on a loss function using a BP algorithm with the goal of minimizing the loss convergence;

[0019] The fusion track determination module is used to perform track reconstruction and fusion splicing processing based on the fusion residual to obtain a fusion track; the fusion track is used to achieve target tracking.

[0020] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target tracking method in the multi-target and multi-source information fusion scenario described above.

[0021] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0022] The present application provides a target tracking method, device and equipment in a multi-target and multi-source information fusion scenario. The present application combines the DBSCAN algorithm and a neural network to perform parallel processing of information data, which can alleviate the gradient vanishing and gradient explosion problems of the network, prevent overfitting, and make the determination of track association results more accurate. In response to the problems of low precision and poor generalization ability in track fusion, the present application uses an error feature estimation algorithm to process the track association results to achieve feature drive, and then combines the trained MsRFCFN network to perform residual fusion, reduce the risk of overfitting and improve learning efficiency. Finally, the track is reconstructed and fused and spliced ​​based on the fusion residual to obtain a fused track to achieve target tracking. Therefore, the present application can improve the fusion accuracy to improve the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 Flowchart of target tracking method in multi-target and multi-source information fusion scenario;

[0025] Figure 2 Schematic diagram of the architecture for track association algorithm;

[0026] Figure 3 This is a flow chart of clustering based on DBSCAN;

[0027] Figure 4 It is a schematic diagram of the specific structure of the CNN network model;

[0028] Figure 5 This is a schematic diagram of the specific structure of the CNN-LSTM parallel network model;

[0029] Figure 6 Schematic diagram of the flow chart corresponding to the track fusion algorithm driven by features and data in parallel;

[0030] Figure 7 Schematic diagram of feature extraction using multi-scale convolutional blocks;

[0031] Figure 8 Schematic diagram of the MsRFCFN fusion network structure;

[0032] Figure 9 This is a structural diagram of a target tracking device in a multi-target and multi-source information fusion scenario;

[0033] Figure 10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] In recent years, with the development of deep learning technology and the continuous updating of network models, it has also played a significant role in track association. The main idea is to extract features from track segments and then classify the tracks based on these features to achieve association. For example, methods such as convolutional neural networks, long short-term memory networks, Transformer models, and their variants have different applications and applicability in the field of multi-sensor multi-target tracking.

[0036] Multi-source sensor data fusion was first proposed in the 1970s. Its goal is to utilize observation data from different types of sensors on the same target, achieving information complementarity through processing and fusion. This results in a fused system with more comprehensive and accurate target information than a single sensor platform. Currently, three main approaches exist: traditional fusion methods such as weighted averaging, Bayesian estimation, and Kalman filtering; linear weighted fusion methods, including convex combination fusion and covariance-weighted fusion; and artificial intelligence algorithms such as fuzzy logic, neural networks, and deep learning. Weighted averaging is the simplest data fusion method, combining the information provided by each sensor and averaging it to obtain a fused value. Kalman filtering is primarily used for data fusion in linear systems. Because it requires less data to be stored during iteration, it offers better real-time performance for low-level data processing. Bayesian estimation performs a Bayesian evaluation on each sensor's data to measure accuracy, then uses a joint probability distribution function to calculate the target's correlation distribution. The fusion result is ultimately obtained by minimizing the joint probability distribution likelihood function. Linear weighted fusion methods perform a weighted summation of different local tracks to obtain a fused track. Using artificial intelligence algorithms to fuse data can better reflect the overall data distribution. In particular, neural networks have strong nonlinear processing capabilities, parallel computing capabilities, and adaptive learning capabilities. They can not only meet the requirements of multi-source sensor data fusion, but also reduce dependence on prior knowledge and effectively improve the reliability of fusion results in complex environments.

[0037] In summary, if the two key technologies of track association and fusion filtering can be applied jointly, and the data-driven and feature-driven algorithm models can be combined, the accuracy, real-time and robustness of target tracking will be further improved.

[0038] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0039] In an exemplary embodiment, Figure 1 As shown, a target tracking method in a multi-target multi-source information fusion scenario is provided, including:

[0040] Step 100: Acquire information data of the target group to be tracked. The information data is acquired based on multiple observation platforms; the information data includes: visible light images and track data information; the target group to be tracked includes multiple targets to be tracked.

[0041] Step 200: Use the DBSCAN algorithm and neural network to parallel process the information data to obtain track association results. The track association results include: track measurement segments and the targets to which the track measurement segments belong.

[0042] Among them, the DBSCAN algorithm and neural network are used to process the information data in parallel to obtain the track association results, including:

[0043] Step 201: Using a Kalman filter algorithm to filter the track data information, and performing motion state estimation based on the track measurement segments obtained after filtering to obtain state information; the state information includes: position information and speed information.

[0044] Step 202: clustering the track measurement segments using the DBSCAN algorithm to obtain classification results; the classification results include: track measurement segments of known targets and track measurement segments of unknown targets.

[0045] In one embodiment, the DBSCAN algorithm is used to cluster the track measurement segments to obtain classification results, which specifically include:

[0046] The DBSCAN algorithm is used to cluster the track measurement segments for a set number of times to obtain the clustering results. The clustering results are subjected to traversal statistical processing to determine the number of times the target appears to be the same target.

[0047] Determine whether the number of times is greater than the set threshold; the set threshold value is less than the set number of times.

[0048] If so, the track measurement segment is a track measurement segment of a known target; if not, the track measurement segment is a track measurement segment of an unknown target.

[0049] The clustering process for any number of times specifically includes:

[0050] All points in the track measurement segment are marked as unvisited points, and the clustering result set C is initialized to be empty. For any unvisited point P, determine whether point P is a core point. The process of determining whether it is a core point specifically includes:

[0051] Calculate the ε-neighborhood of point P; if the ε-neighborhood of point P contains at least MinPts points, and point P is also in the ε-neighborhood, then point P is taken as the core point, and all points in the ε-neighborhood are added to the clustering result set C.

[0052] For each point Q in the ε-neighborhood, determine whether point Q is a core point, and recursively repeat the core point determination process to expand the clustering result set C to obtain the clustering result.

[0053] Step 203: Perform target model recognition on the visible light image based on the CNN network model to obtain target type information; the CNN network model is trained based on the visible light image with known target type information; the CNN network model includes: an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer.

[0054] Step 204: Input the target type information and the track measurement segment of the unknown target into the trained CNN-LSTM parallel network model to obtain an association result; the association result is the target corresponding to the track measurement segment of the unknown target; the trained CNN-LSTM parallel network model is obtained by training the CNN-LSTM parallel network model based on the spliced ​​feature data and the track measurement segment of the known target; the spliced ​​feature data is obtained by feature splicing of the target type information and the state information.

[0055] Step 205: Perform track correlation processing on the correlation result and the track measurement segment of the unknown target to obtain a track correlation result.

[0056] Step 300: Segment the track measurement segments based on a preset sliding window and step size, and use an error characteristic estimation algorithm to determine the track residual based on the sub-track information obtained by segmentation.

[0057] The track measurement segments are segmented based on a preset sliding window and step size, and the track residual is determined based on the segmented sub-track information using an error characteristic estimation algorithm, specifically including:

[0058] The track measurement segments are segmented based on a preset sliding window and step size to obtain multiple segments of sub-track information; and a Bezier curve is used to linearly fit any two segments of sub-track information to obtain an approximate track of the real sub-track information.

[0059] The track residual between the sub-track information and the true sub-track information is extracted based on the approximate track.

[0060] The expression of the approximate track is:

[0061]

[0062] in, is the approximate track; is the Bernstein basis function; t is a variable parameter; R i is the control point of the approximate trajectory; n is the order of the Bezier curve; i is the order number of the Bezier curve.

[0063] Step 400: Based on the trained MsRFCFN network, a fusion residual is determined according to the track residual. The trained MsRFCFN network is obtained by training the parameters of the MsRFCFN network based on a loss function using a BP algorithm with the goal of minimizing the loss convergence.

[0064] Step 500: Perform track reconstruction and fusion splicing based on the fusion residual to obtain a fused track. The fused track is used to track the target.

[0065] Among them, the track reconstruction and fusion splicing processing are performed based on the fusion residual to obtain the fused track, which specifically includes:

[0066] The fusion residual is normalized to obtain the normalized fusion residual.

[0067] Using the formula:

[0068]

[0069] The track is reconstructed based on the normalized fusion residual to obtain the fused sub-track.

[0070] in, is the fusion sub-track; is the approximate track; is the normalized fusion residual; a is the vector composed of the minimum value of the track residual; b is the vector composed of the maximum value of the track residual.

[0071] Based on the status of the fused sub-tracks, fusion and splicing processing is performed to obtain the fused track; the expression corresponding to the status of the fused sub-tracks is:

[0072]

[0073] The expression of the fused track is:

[0074]

[0075] in, is the state of the fused sub-track at time i1; is the total number of local sub-tracks at time i1; is the state value of the j-th sub-track; T f To fuse the tracks; is the state of the fused sub-track at the first moment; is the state of the fused sub-track when the track length is N.

[0076] In one embodiment, the MsRFCFN network includes a feature extraction module and a fusion module.

[0077] The feature extraction module includes a basic convolution part and a multi-scale convolution part; wherein, the basic convolution part includes a first layer of small-scale convolution, a second layer of small-scale convolution, a first layer of dilated convolution and a second layer of dilated convolution.

[0078] The convolution kernel size of the first layer of small-scale convolution and the second layer of small-scale convolution are both 3×3, the number of convolution kernels is 64, and the activation function is ReLU.

[0079] The convolution kernel size of the first and second layers of dilated convolution are both 3×3, the number of convolution kernels is 64, and the activation function is ReLU; the dilation ratio of the first layer of dilated convolution is 2, and the dilation ratio of the second layer of dilated convolution is 3.

[0080] The multi-scale convolution part contains a set number of multi-scale convolution blocks; the first layer of the fusion module is a 1×1 convolution layer, the number of convolution kernels is 128, and the activation function is ReLU; the second layer of the fusion module is a 3×3 convolution layer, the number of convolution kernels is 1, and the activation function is ReLU.

[0081] As an optional implementation, the loss function adopts the average root mean square error function; the expression of the average root mean square error function is:

[0082]

[0083] Where ARMSE is the average root mean square error; N is the track length; M is the number of tracks in the dataset; is the y-axis position of the qth true track at time p; is the y-axis position of the fused track corresponding to the qth real track at time p.

[0084] This application combines the DBSCAN algorithm with a convolutional neural network and a long short-term memory neural network. The observed track segments are classified by the DBSCAN algorithm to obtain a set of track segments whose targets have been determined, i.e., a labeled track data set, and a set of track segments whose targets have not been determined. The labeled track data set is used to train the CNN-LSTM network model, and the trained network model is then used to classify and output the set of track segments whose targets have not been determined, thereby completing the association of all track segments.

[0085] By segmenting the associated track segments into sub-tracks, the error characteristics of the sub-tracks are estimated and extracted to obtain residuals, completing feature-driven processing. A fusion network is then designed, using the residuals between the local sub-tracks and the true track as network inputs and the residuals between the approximate track and the true track as the desired output. The network is trained to improve fusion accuracy. Finally, the local sub-tracks are spliced ​​together to complete the fusion of the entire local sub-track.

[0086] In practical applications, the operating steps of the method mentioned in this application are as follows.

[0087] Step 1: Track association algorithm based on the fusion of DBSCAN and deep learning.

[0088] In the multi-sensor multi-target tracking problem, due to the interference and clutter in the observation environment and the strong maneuverability of the target, the track information observed by the sensor may be interrupted. At the same time, multiple sensors will receive a large amount of track information from the same target. In order to accurately track the target track, it is necessary to establish a one-to-one correspondence between the observation tracks of the same target by different sensors, that is, to perform track association on the multi-sensor observation information.

[0089] With the advancement of target tracking technology, using a single sensor to observe a target is no longer sufficient. A more comprehensive target observation requires the use of multiple sensors working in parallel, such as those utilizing visible light sensors and millimeter-wave radars. Traditional track association algorithms classify track segments using only their position information. However, after Kalman filtering, observed tracks can obtain not only target position information but also target velocity information, significantly aiding in temporal association of track segments. However, these algorithms fail to utilize velocity features, resulting in low association accuracy. Furthermore, few traditional algorithms can process image information, making it difficult to fuse multi-source information in scenarios where multiple sensors are co-observing. Therefore, deep learning theory is required. Neural network-based track association methods require labeled track data as a training set. However, in practice, it is impossible to predict which target a sensor-observed track segment belongs to, making it difficult to create a training set to train the neural network. To address the above shortcomings, this application proposes a track association algorithm based on the combination of DBSCAN and neural network in an observation scenario with multiple platforms, each equipped with visible light sensors and millimeter wave radars, to achieve track association in multi-source sensor and multi-target scenarios.

[0090] The DBSCAN algorithm (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm that can identify clusters of arbitrary shapes while effectively handling noisy data points. Unlike traditional partition-based clustering methods, the DBSCAN algorithm divides the data space into distinct regions by density and can find high-density clusters within these regions, while marking low-density regions as noise.

[0091] The core idea of ​​the DBSCAN algorithm is to use "density reachability" to perform clustering. First, let's define the key concepts in the algorithm:

[0092] ε-neighborhood: For a data point P, its ε-neighborhood refers to the set of points whose Euclidean distance from P is less than or equal to a set threshold ε, that is:

[0093] N ε (P)={Q∈D|dist(P,Q)≤ε}.

[0094] Where D represents the data set, P and Q represent data points, dist(P,Q) represents the Euclidean distance between two points, and N ε (P) represents all points in the ε-neighborhood of point P.

[0095] Core point: If the ε-neighborhood of point P contains at least MinPts data points, including point P itself, then point P is considered a core point, that is:

[0096]

[0097] Then point P is a core point.

[0098] Boundary point: If the number of points contained in the ε-neighborhood of a point is less than MinPts, but it itself is also in the ε-neighborhood of a core point, then the point is a boundary point.

[0099] Noise point: If a point is neither a core point nor a boundary point, it is considered a noise point.

[0100] Specifically, the DBSCAN algorithm performs clustering through the following steps:

[0101] Mark all points as unvisited points, initialize the clustering result set C to be empty, start from an unvisited point P in the data set, and calculate the ε-neighborhood N of P ε (P), if N ε (P) has at least MinPts points, then P is a core point. If P is a core point, then N ε All points in (P) are added to the same cluster C. For N ε For each point Q in (P), if Q can also be judged as a core point, then obtain Q's ε-neighborhood N ε (Q), put N ε All unvisited points in (Q) are added to cluster C, and the process is repeated recursively until no new core points can be found and no more points can be expanded, completing the expansion of cluster C. Finally, for each point that does not have MinPts neighboring points, it is determined whether it is in the neighborhood of the core point. If so, it is considered a boundary point, otherwise it is marked as noise.

[0102] The DBSCAN algorithm, relying on density expansion and noise filtering mechanisms, can automatically identify high-density target groups while simultaneously eliminating isolated or sparse outliers, making it highly suitable for track correlation scenarios. In complex noisy environments, the DBSCAN algorithm can effectively distinguish real targets from interference data. For dynamic target tracking, as targets continuously move through space, the DBSCAN algorithm can continuously expand the density of connected track points, connecting the trajectories of the same target in series. This eliminates the need to know the number of targets in advance and can adapt to complex situations such as changes in target shape and speed, thereby ensuring track continuity and accuracy.

[0103] A convolutional neural network (CNN) is a deep learning model specifically designed for processing grid-structured data. Its core concept is to effectively extract multi-level feature representations of input data through local receptive fields, weight sharing, and spatial downsampling. A CNN consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. Its core operations are convolution in the convolutional layers and pooling in the pooling layers.

[0104] The convolutional layer is the core layer of the convolutional neural network. The convolution operation is the network extracting image features. Compared with the calculation method of the fully connected layer, the number of parameters in the operation process is reduced. The formula is as follows:

[0105]

[0106] in, Represents the value of the (i', j')th position in the kth output feature map, C1 represents the number of channels of the input feature map, M1 and N1 represent the height and width of the convolution kernel, represents the weight of the k-th convolution kernel at the (m1,n1) position on the c-th input channel, Represents the pixel value at position (i,+m1-1,j'+n1-1 in the cth input feature map, b k Represents the bias of the kth convolution kernel. In the convolution layer, the convolution kernel slides across the input features. The sliding range depends on the size of the convolution kernel. Each sliding step performs a convolution calculation and obtains the corresponding feature value. The convolution layer adopts the strategy of local connection and weight sharing. Local connection means that neurons only focus on a local area of ​​the input. Information outside this area is not involved in the calculation. Each output feature only comes from a part of the input. Weight sharing means that the same convolution kernel repeatedly uses the same parameters to perform operations on the entire input. Through this design, convolutional neural networks greatly reduce the number of parameters and computational workload required to learn, while still ensuring good feature extraction capabilities.

[0107] Activation functions are a crucial component of convolutional neural networks. They introduce nonlinearity into the output of each layer, enabling the network to process more complex feature relationships. Without activation functions, no matter how many layers are added, the final representation is limited to a simple linear transformation. A commonly used activation function is ReLU (rectified Linear Unit), which converts values ​​less than zero to zero while leaving values ​​greater than zero unchanged. This not only speeds up computation but also effectively addresses the vanishing gradient problem in traditional neural networks.

[0108] The pooling layer, also known as the downsampling layer, can reduce the amount of data and computational complexity while helping to extract more stable features. The pooling operation is to find a representative value in a certain area of ​​the feature map. Generally, there are maximum pooling, average pooling, and random pooling. The most common method is maximum pooling, which selects the maximum value in a small area, so that the most significant features can be retained. Average pooling and maximum pooling are similar in operation, but the difference is that it selects the average value of the values ​​in a small area as the output value after pooling; random pooling multiplies the data in a small area with the probability value of a random probability matrix and then outputs the sum. Through pooling, the network becomes less sensitive to small changes in position, which helps to improve the robustness of the model.

[0109] When tracking a target, the track data observed by sensors is mostly time series. Long Short-Term Memory (LSTM) is a special type of recurrent neural network specifically designed to address the gradient vanishing and exploding problems and long-term dependency learning difficulties of traditional recurrent neural networks when processing long sequences of data. LSTM dynamically controls the forgetting, updating, and output of information by introducing a gating mechanism and cell state. The cell state runs through the entire time step and is used to store long-term information. The gating unit includes a forget gate, an input gate, and an output gate. The Sigmoid function generates weights from 0 to 1 to regulate the flow of information. The calculation of each LSTM unit at time step t is as follows:

[0110] Forget gate: determines which information to discard from the cell state. The formula is as follows:

[0111] f t =σ(W f ·[h t-1 ,x t ]+b f ).

[0112] Among them, f t represents the output of the forget gate, σ is the Sigmoid activation function, W f is the weight matrix of the forget gate, h t-1 is the hidden state of the previous moment, x t is the current input vector, b fis the bias term of the forget gate.

[0113] Input gate: determines which new information is stored in the cell state. The formula is as follows:

[0114] i t =σ(W i ·[h t-1 ,x t ]+b i ).

[0115]

[0116] Among them, i t represents the output of the input gate, is the candidate cell state, W i ,W C The weight matrices corresponding to the input gate and candidate state, b i ,b C correspond to the bias terms respectively.

[0117] Combined with the output results of the forget gate and the input gate, the current cell state is updated:

[0118]

[0119] Among them, C t Represents the cell state at the current moment, and ⊙ represents element-by-element multiplication, that is, the Hadamard product of the two matrices.

[0120] Output gate: determines what information is output from the cell state to the hidden state. The formula is as follows:

[0121] o t =σ(W o ·[h t-1 ,x t ]+b o ).

[0122] h t =o t ⊙tanh(C t ).

[0123] Among them, t represents the output of the output gate, h t Represents the hidden state at the current moment, W o ,b o are the weight matrix and bias respectively.

[0124] The LSTM unit filters out currently useless information through the forget gate and only retains valid information. Compared with traditional recurrent neural networks, LSTM only uses valid information for gradient updates, thereby alleviating the network's gradient vanishing and gradient exploding problems.

[0125] This application combines the DBSCAN algorithm and neural network to propose a new association algorithm. The new algorithm combines the visible light images collected by visible light sensors and the track data information collected by millimeter wave radars to improve the association accuracy. The architecture of the algorithm is as shown in Figure 2 . When a target group appears, the millimeter wave radar and visible light sensor work simultaneously. The visible light sensor collects the visible light images of the targets, and the millimeter wave radar collects the track data information of the targets. The two types of data are processed in parallel, and finally the track association result is obtained and the accuracy is calculated. First, the track data information obtained by the millimeter wave radar is filtered by the Kalman filtering algorithm to remove the measurement noise of the track, and at the same time, the motion state of the target is estimated to obtain the position and velocity information of the target. Then, the track measurement segment of the target is passed to the DBSCAN algorithm for classification. According to the classification result, the track measurement segment is divided into two parts. One part is the track measurement segment of the known target, and the other part is the track measurement segment of the unknown target.

[0126] Assume that at time k, the set of track measurement segments after Kalman filtering is X k , which is expressed as:

[0127] X k = {x 1,k , x 2, k, …, x [[ID=,19]] o,k}.

[0128] Where o is the total number of track measurement segments collected by the sensor at time k, x 1,k is the first track measurement segment collected by the sensor at time k, x 2,k and so on. Send X k to the classification module, and its process structure is as shown in Figure 3 .

[0129] At this time, the set of track measurement segments X k has filtered out the measurement noise through the Kalman filtering module and estimated the position and velocity information of the target. However, since it has not been associated at this time, it is unknown which target the track measurement segment belongs to. The track measurement segment is sent to the DBSCAN clustering module for n times of clustering. The n clustering results are traversed and counted, and the number of times a that each sample x i,k (i ∈ 1~o) is considered to belong to the same target j1 in the n times of clustering is counted. Then set a threshold m, and m should satisfy m < n. When a > m, it can be considered that the track measurement segment x i,kBelonging to target j1, all track measurement segments that can meet the above conditions are regarded as track measurement segments with determined target, and their characteristics include the target's position, speed and target; the remaining track measurement segments that do not meet the above conditions are regarded as track segments with undetermined target, and their characteristics only include the target's position and speed.

[0130] Define the set of track measurement segments that have been determined to belong to the target at time k as Where w is the number of track measurement segments whose target has been determined at time k. Each element in the set is a track segment, which can be expressed as:

[0131]

[0132] Each track measurement segment in this set contains 5 features, where x and y represent the position coordinates of the target. Indicates the speed information of the target, and j1 indicates the track measurement segment Which target does it belong to?

[0133] Define the set of track measurement segments whose target is not determined at time k as Where e is the number of track measurement segments whose targets are not determined at time k. Each element in the set is a track measurement segment, which can be expressed as:

[0134]

[0135] Each track measurement segment of this set contains 4 features, and the meaning of each feature is the same as above, e, w, o satisfies o = w + e.

[0136] For the visible light images collected by the visible light sensor, a CNN network model is built. The CNN network is trained using a labeled image training set. The network parameters are repeatedly adjusted so that the CNN network model can more accurately identify the target model. After the training is completed, the CNN network model can be used as a target recognition module to extract features from infrared images. Finally, the target model t' of each infrared image is classified through the Softmax layer in the network. The specific structure of the CNN network model used in this application is shown in Figure 4 .

[0137] like Figure 4As shown in Figure 1, in this model, the input image size is 128×128×3. The first layer is a convolutional layer, which convolves the input image using the SAME convolution mode. This means that appropriate zero padding is added around the input image to ensure that the size of the feature map after convolution is consistent with the input image size. The convolution kernel size is set to 3×3, the number of kernels is set to 32, and the stride is set to 1. The feature map size after convolution is 128×128×3. A maximum pooling layer, Pool-1, is added after convolution layer Conv-1, with a pooling window size of 2×2. The third and fourth layers of the network model are convolution layers Conv-2 and Conv-3, which use the same convolution operation as Conv-1, with a convolution kernel size of 3×3, the number of kernels is set to 64, and the stride is set to 1. A maximum pooling layer, Pool-2, is added after Conv-3, with a pooling window size also set to 2×2. The 6th layer of the network model is the convolution layer Conv-4, which uses the same convolution operation as Conv-1. The size of the convolution kernel is set to 3×3, the number of convolution kernels is set to 128, and the step size is set to 1. After the convolution, the maximum pooling layer Pool-3 is added, and the size of the pooling window is set to 2×2. Two fully connected layers FCL-1 and FCL-2 are added after Pool-3. The number of neurons in the fully connected layer FCL-1 is set to 256, and the number of neurons in the fully connected layer FCL-2 is set to the number of models of the detected target. A Dropout layer is also added between the two fully connected layers, and the dropout rate is set to 0.5 to prevent the neural network from overfitting. A Softmax layer is added at the end of the network model to convert the output of the fully connected layer into a probability distribution for classifying the target model. In order to accelerate the training convergence of the model, this application performs batch normalization (BatchNormalization, BN layer) after all convolution layers. At the same time, the BN layer can also prevent the gradient of the convolution layer from disappearing and the network model from overfitting.

[0138] By processing the visible light image corresponding to each track measurement segment through the CNN network model, the target model feature t' corresponding to each track measurement segment can be obtained. The target model feature is combined with the aforementioned features to further improve the accuracy of track association. At this time, the track measurement segments with confirmed and unconfirmed targets are respectively expressed as:

[0139]

[0140] The track association problem can also be considered as a classification problem of track measurement segments. This application uses the CNN-LSTM parallel network model to learn the changes in track characteristics, obtain association results and calculate the association accuracy. The specific network structure diagram is shown in Figure 5 .

[0141] Before feature extraction, first perform feature splicing on To label the data, middle The first five dimensions of information, namely the position, speed and target model corresponding to the track segment, are used as the five features of the network training set. The sixth dimension feature, namely the target to which the track segment belongs, is used as the label of the network training set to complete the production of the network training set. This serves as the test set after network training, completing the creation of a track dataset. The track dataset is then normalized, scaling the track data to between 0 and 1 to eliminate the impact of varying feature dimensions on the network. Because the track data is a long time series, it needs to be segmented. This application uses a sliding window of size 6 to segment the long track sequence into small data blocks, accelerating network convergence and enabling better feature extraction.

[0142] Next, the normalized dataset is fed into a CNN-LSTM parallel network model for feature learning. The input data size is 6×7, which is the product of the sliding window size and the number of features. In the CNN module, the first layer is the convolutional layer Conv-1, with 32 convolution kernels and a size of 1×3. After convolution, a feature matrix of size 32×3 is obtained. A sigmoid activation function is added after the convolutional layer to accelerate network convergence. The second layer is a max pooling layer to reduce network computational complexity. The pooling window is set to 2×2, reducing the size of the feature matrix to 16×3. Finally, a Flatten layer is added to flatten the feature matrix into a spatial feature matrix to facilitate concatenation with the temporal feature matrix output by the LSTM module. The flattened feature matrix size is 48×1. In the LSTM module, the first layer is an LSTM layer with 32 neurons. After LSTM feature extraction, a 6×32 feature matrix is ​​obtained. Dropout is then applied, with the dropout parameter set to 0.5. Finally, the Flatten layer is still added to flatten the feature matrix into a time series feature matrix. The size of the flattened feature matrix is ​​192×1. The 48×1 spatial feature matrix obtained by the CNN module and the 192×1 time series feature module obtained by the LSTM module are spliced. The splicing function uses the torch.cat function under the PyTorch framework to obtain a 240×1 feature matrix. The fully connected layer FCL is then used to process the spliced ​​feature matrix to improve the network model's ability to express features. The number of neurons in the fully connected layer is set to the number of targets in the track dataset. Finally, the Softmax layer is used for classification to output the network's associated results. The CNN-LSTM parallel network model uses the spliced ​​feature matrix. After training, input the concatenated features get The target to which the track segment belongs is determined, and the track association is completed. Finally, the accuracy of the association algorithm is calculated.

[0143] Step 2: Feature and data parallel driven track fusion algorithm.

[0144] After each platform completes the track association, in order to reduce the error in the target state measurement and other uncertainty information and obtain a higher-precision target position, it is necessary to fuse the local tracks obtained from multiple platforms measuring the same target. The data-driven fusion algorithm establishes a deep neural network and uses a pre-labeled local track data set to update and learn the network parameters to achieve information fusion. This application proposes a feature- and data-parallel-driven track fusion algorithm to address the problems of low fused track accuracy, poor generalization ability, and poor fusion performance for local tracks with large differences in accuracy in data-driven track fusion.

[0145] Assume that the target's true track T in the two-dimensional plane r Expressed as:

[0146]

[0147] This application uses two sensor platforms s1 and s2 to observe the target track as an example to perform track fusion. The track fusion of multiple platforms can be obtained in the same way. The local track measured by the two sensor platforms can be expressed as:

[0148] T s1 =T r +E s1 .

[0149] T s2 =T r +E s2 .

[0150] Among them, T s1 ,T s2 are the local tracks detected by the two sensors, E s1 ,E s2 are the errors of the two local tracks relative to the true track, including measurement error, error in data preprocessing, and noise. The purpose of track fusion is to process the two local tracks through the fusion algorithm to obtain the fused track T f , which can be expressed as:

[0151] T f =T r +E f .

[0152] Among them, E fThe error between the fused track and the real track is the smaller the error is, the better the track fusion effect is. Therefore, this application allows the network to focus on learning the error features directly, but in actual applications, the error E s1 ,E s2 It cannot be obtained directly, and an error feature estimation algorithm is required to process the track to obtain an estimated value of the track error, i.e., the residual. The feature and data parallel driving mentioned in this application refer to: first, based on prior knowledge and the error feature estimation algorithm, an error model of the track is established, the error feature is estimated to obtain the residual feature, and the track fusion algorithm is converted from learning track features to learning residual features, thus completing feature driving; then a training set is established, a fusion network is designed, and the residual features obtained through the local track and the real track are used as the input and expected output of the network, respectively, to train the network, thus completing data driving.

[0153] The track fusion algorithm driven by feature and data parallelism includes two stages: offline training and online fusion. Figure 6 As shown. In the offline training stage, a smaller sliding window and step size are first set, and the local track and the true track obtained by the sensor platform are divided into small sub-tracks, and the track residual is obtained through the error feature estimation algorithm. This application designs a multi-scale residual full convolution fusion network (Multi-scale Residuals Full Convolution Fusion Network, MsRFCFN), which is used to forward propagate the residuals of the input local sub-tracks and calculate the fusion residuals of the two sensor platform sub-tracks. Next, MsRFCFN uses the loss function to calculate the difference between the fusion residual and the true residual, and then uses the BP algorithm to backpropagate the difference and update the parameters. After multiple trainings, the network loss finally converges to the minimum value.

[0154] In the online fusion stage, the local track is first divided into sub-tracks according to the set sliding window and step size, and the track residual is obtained through the error characteristic estimation algorithm. The track residual is then input into the trained MsRFCFN network to obtain the fusion residual through forward propagation. The track is reconstructed on the fusion residual output by the network to obtain the fused sub-track, and then the sub-tracks are fused and spliced ​​to finally obtain the complete fused track.

[0155] Sub-track segmentation.

[0156] The local track is divided into multiple sub-tracks according to the set sliding window and step size for estimating the error characteristics. Assuming that the length of the complete local track is N, combined with the real-time performance and accuracy standards of the fusion algorithm, the sliding window length is set to L w , the step size is N s After segmentation, one of the sub-tracks of the local track of sensor platforms s1, s2 can be expressed as

[0157] Error characteristic estimation.

[0158] Error characteristic estimation is mainly achieved through two steps. First, the local sub-track is fitted to obtain the approximate track of the true sub-track; then the track residual between the local sub-track and the true sub-track is extracted based on the approximate track.

[0159] Track approximation methods include linear fitting, nonlinear fitting, and filtering algorithms. Nonlinear fitting and filtering algorithms take a long time to calculate and require certain prior knowledge. In order to improve calculation efficiency and reduce dependence on prior knowledge, this application uses Bezier curves to linearly fit two local tracks to obtain approximate tracks. First, the two local sub-tracks are fitted with n-order Bezier curves, which can be expressed mathematically as:

[0160]

[0161] Among them, P i ,Q i is the control point, is the Bernstein basis function, and t∈[0,1] is a variable parameter.

[0162] Next, the control points of the local sub-track are weighted fused to generate an approximate sub-track. The control point R i :

[0163] R i =αP i +(1-α)Q i i=0,1,…,n。

[0164] Where α∈[0,1] is the fusion weight, which can be dynamically adjusted according to the sensor confidence or the covariance of the measurement error.

[0165] Approximate sub-track after fusion Expressed as:

[0166]

[0167] Local sub-track and the real sub-track Subtract the approximate sub-tracks to obtain the track residuals The track residual is used as an estimate of the true error.

[0168]

[0169] Combined with local track T s1 ,T s2 The mathematical representation of the local sub-track error can be obtained The quantitative relationship between it and the track residual is:

[0170]

[0171] It is the difference between the local sub-track residual and the corresponding true error. The approximate accuracy of the Bezier curve is higher, so the difference between the approximate sub-track and the true sub-track is The error is very small, that is, the local sub-track residual is close to the true error, and the residual can be used to replace the error for subsequent training. At this point, the estimation and extraction of error features are completed, that is, feature driving is completed.

[0172] Residual fusion.

[0173] After completing the feature drive, it is necessary to design a deep learning network to achieve residual fusion. Before the residual data is fed into the neural network, it needs to be normalized. Compared with two-dimensional single-channel data, one-dimensional dual-channel data can maintain the connection between channels, and at the same time prevent the introduction of redundant parameters during the convolution process, thereby reducing the risk of overfitting and improving learning efficiency. Therefore, this application combines the residuals of the two parts of the local sub-tracks into a matrix to form one-dimensional dual-channel data, and then performs normalization:

[0174]

[0175] Among them, O I is the normalized residual merging matrix, and a is the The vector of the minimum values ​​in The vectors formed by the maximum values ​​in are all the same size as O I The same, the fraction here represents element-by-element division. I It can be used as the input of the network.

[0176] In the MsRFCFN fusion network structure, this application designs a multi-scale convolution block (Multi-scale Convolution Block, MsCB) for feature extraction, such as Figure 7 As shown in the figure, from left to right, they are the deep small receptive field path, medium receptive field path, large receptive field path, and residual connection path. The deep small receptive field path consists of one layer of 1×1 convolution and two layers of 3×3 convolution, enhancing the extraction of local features. The medium receptive field path consists of one layer of 1×1 convolution and one layer of 3×3 convolution. The large receptive field path consists of one layer of 3×3 convolution. The residual connection path is a direct connection path. Finally, the four paths are channel-wise concatenated to form the output. The parameters (i, j, k) represent the number of convolution kernels in the branch.

[0177] This multi-scale convolutional block, through a parallel multi-branch structure, fuses receptive fields of different scales, enabling it to simultaneously capture local details and global context. It also automatically adapts to target features of varying sizes, enhancing the model's robustness to changes in target size and making it suitable for multi-scale tasks. The first two branches employ 1×1 convolutions for channel compression and replace single-layer large convolutions with multiple layers of small convolutions. For example, two layers of 3×3 convolutions replace one layer of 5×5 convolution, reducing the number of network parameters and complexity by approximately 30%. The residual connection design also ensures the trainability of deep networks, speeding up convergence and preventing vanishing or exploding gradients.

[0178] The MsRFCFN fusion network structure designed in this application is as follows Figure 8 As shown in Figure 2. It includes feature extraction module and fusion module.

[0179] The feature extraction module consists of two parts, namely the basic convolution part and the multi-scale convolution part. The basic convolution part includes two layers of identical small-scale convolutions, the convolution kernel size is 3×3, the number of convolution kernels is 64, and the activation function is ReLU. Small-scale convolution can preliminarily extract local detail features of the image. Next are two layers of dilated convolutions, the convolution kernel size is 3×3, the number of convolution kernels is 64, and the activation function is ReLU. The dilation rate of the first layer of dilated convolution is set to 2, and the dilation rate of the second layer of dilated convolution is set to 3. By setting different dilation rates, the receptive field can be expanded and multi-scale contextual information can be captured. The multi-scale convolution part contains 4 multi-scale convolution blocks, and the parameters are set to (32,32,32), (64,64,64), (96,96,96), (128,128,128), respectively. The residual features that have been preliminarily extracted are further extracted to obtain multi-scale features. At this time, the output feature map size is L w ×384, the number of channels is the number of convolution kernels in the last layer.

[0180] The first layer of the fusion module is a 1×1 convolution layer with 128 convolution kernels and ReLU as the activation function. It reduces the dimension of the feature map output by the feature extraction module and fuses the cross-channel information. The second layer is a 3×3 convolution layer with 1 convolution kernel and ReLU as the activation function. It further fuses the spatial features to obtain the single-channel fusion residual. Size L w ×1.

[0181] There is no pooling layer in the fully convolutional network structure, and the length of the output data of each layer is the same, which avoids feature loss during the conversion between high-dimensional and low-dimensional features.

[0182] Track fusion and splicing.

[0183] After obtaining the normalized fusion residual After that, it needs to be restored to the fused sub-track The expression is:

[0184]

[0185] In order to improve the utilization of data during the sub-track extraction process, the selected step size is small, which is smaller than the sliding window length, resulting in overlaps between adjacent sub-tracks extracted by the sliding window. Therefore, when the fused sub-tracks are restored and spliced, the overlapping parts need to be averaged to further improve the fusion accuracy. The state of the fused track at time i is set to The calculation method is:

[0186]

[0187] Indicates the state value of the j-th sub-track containing the point at the point, M i represents the total number of local sub-tracks containing this point, and the complete fused track can be obtained as:

[0188]

[0189] During the offline training phase, the loss function is the average root mean square error (ARMSE). The ARMSE of the track in the y direction is expressed as follows, and the same is true for the x direction:

[0190]

[0191] This application conducted a comparative simulation verification on the nearest neighbor algorithm, joint probabilistic data association algorithm, K-MEANS clustering algorithm in traditional association algorithms, and the association algorithm proposed in this application under the same simulation scenario. The association accuracy is shown in Table 1.

[0192] Table 1 Association accuracy table

[0193] Association Algorithm Nearest Neighbor Algorithm Joint Data Association Algorithm K-MEANS clustering algorithm This application algorithm Association accuracy 82.38％ 89.37％ 78.66％ 96.24％

[0194] Based on Table 1, it can be seen that compared with other traditional algorithms, the association accuracy of the algorithm proposed in this application is improved. The algorithm comprehensively extracts the temporal and spatial features of the track data, and extracts the target model features in the image. After pre-classifying the track segments through DBSCAN, the neural network is trained and tested. This algorithm effectively solves the problem that in actual association application scenarios, the target to which the track segment belongs cannot be predicted, making it difficult to prepare a neural network training set, thereby improving the association accuracy.

[0195] In the same simulation scenario, this application conducted comparative simulation verification on the covariance cross fusion algorithm, the expanded dimension KF fusion algorithm, the variational Bayesian adaptive Kalman filter algorithm (VBAKF) in the traditional fusion algorithm, and the fusion algorithm proposed in this application. The calculated ARMSEs are shown in Table 2, in km.

[0196] Table 2 ARMSE table corresponding to each fusion algorithm

[0197] Fusion Algorithm Covariance Crossover Extended Dimension KF VBAKF This application algorithm ARMSE / km 0.94 0.87 1.12 0.53

[0198] Table 2 shows that the accuracy of our fusion algorithm is higher than that of several other fusion algorithms. This algorithm extracts the error estimation characteristics of the sub-tracks and establishes a residual model, which drives the track fusion based on the error characteristics. The error characteristics are then passed to the MSRFCFN network model for residual fusion. Finally, the overlapping points in multiple sub-tracks are combined to form a complete track, resulting in a fused track, further improving the fusion accuracy of the approximate track.

[0199] In an exemplary embodiment, Figure 9 As shown, a target tracking device in a multi-target multi-source information fusion scenario is provided, comprising:

[0200] The information data acquisition module is used to acquire information data of the target group to be tracked; the information data is acquired based on multiple observation platforms; the information data includes: visible light images and track data information; the target group to be tracked includes multiple targets to be tracked.

[0201] The processing module is used to use the DBSCAN algorithm and the neural network to perform parallel processing on the information data to obtain the track association result; the track association result includes: the track measurement segment and the target to which the track measurement segment belongs.

[0202] The track residual determination module is used to segment the track measurement segments based on the preset sliding window and step size, and use the error characteristic estimation algorithm to determine the track residual according to the sub-track information obtained by segmentation.

[0203] The fusion residual determination module is used to determine the fusion residual based on the trained MsRFCFN network and the track residual; the trained MsRFCFN network is obtained by training the parameters of the MsRFCFN network based on the loss function and adopting the BP algorithm with the goal of minimizing the loss convergence.

[0204] The fusion track determination module is used to reconstruct the track and perform fusion splicing processing based on the fusion residual to obtain the fusion track; the fusion track is used to achieve target tracking.

[0205] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target tracking method in a multi-target and multi-source information fusion scenario is implemented.

[0206] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0207] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0208] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0209] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0211] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0212] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0213] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A target tracking method in a multi-target multi-source information fusion scenario, characterized in that: include: Obtain information data of the target group to be tracked; The information data is obtained based on multiple observation platforms; The information data includes: visible light images and track data information; the target group to be tracked includes multiple targets to be tracked; The information data is processed in parallel using the DBSCAN algorithm and a neural network to obtain a track association result; the track association result includes: a track measurement segment and a target to which the track measurement segment belongs; Segmenting the track measurement segments based on a preset sliding window and step size, and determining the track residual based on the segmented sub-track information using an error characteristic estimation algorithm; Based on the trained MsRFCFN network, a fusion residual is determined according to the track residual; the trained MsRFCFN network is obtained by training parameters of the MsRFCFN network based on a loss function and adopting a BP algorithm with a goal of minimizing the loss convergence; Track reconstruction and fusion splicing are performed based on the fusion residual to obtain a fused track; the fused track is used to achieve target tracking.

2. The target tracking method in a multi-target multi-source information fusion scenario according to claim 1, characterized in that: The DBSCAN algorithm and neural network are used to process the information data in parallel to obtain the track association results, which specifically include: The track data information is filtered using a Kalman filter algorithm, and motion state estimation is performed based on the track measurement segments obtained after the filtering process to obtain state information; the state information includes: position information and speed information; Clustering the track measurement segments using the DBSCAN algorithm to obtain classification results; the classification results include: track measurement segments of known targets and track measurement segments of unknown targets; Performing target model recognition on the visible light image based on a CNN network model to obtain target type information; the CNN network model is trained based on the visible light image with known target type information; the CNN network model includes: an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer; The target type information and the track measurement segment of the unknown target are input into the trained CNN-LSTM parallel network model to obtain an association result; the association result is the target corresponding to the track measurement segment of the unknown target; the trained CNN-LSTM parallel network model is obtained by training the CNN-LSTM parallel network model based on the spliced ​​feature data and the track measurement segment of the known target; the spliced ​​feature data is obtained by feature splicing the target type information and the state information; The association result is subjected to track association processing with the track measurement segment of the unknown target to obtain a track association result.

3. The target tracking method in a multi-target multi-source information fusion scenario according to claim 2, characterized in that: The DBSCAN algorithm is used to cluster the track measurement segments to obtain classification results, which specifically include: Using the DBSCAN algorithm to perform clustering processing on the track measurement segments for a set number of times to obtain a clustering result; Performing traversal statistical processing on the clustering results to determine the number of times the targets are the same target; Determine whether the number of times is greater than a set threshold; the value of the set threshold is less than the value of the set number of times; If yes, the track measurement segment is a track measurement segment of a known target; If not, the track measurement segment is a track measurement segment of an unknown target; The clustering process for any number of times specifically includes: Mark all points in the track measurement segment as unvisited points, and initialize the clustering result set C to be empty; For any unvisited point P, determine whether point P is a core point. The process of determining whether it is a core point specifically includes: Calculate the ε-neighborhood of point P; if the ε-neighborhood of point P contains at least MinPts points, and point P is also in the ε-neighborhood, then point P is taken as the core point, and all points in the ε-neighborhood are added to the clustering result set C; For each point Q in the ε-neighborhood, determine whether point Q is a core point, and recursively repeat the core point determination process to expand the clustering result set C to obtain the clustering result.

4. The target tracking method in a multi-target multi-source information fusion scenario according to claim 1, characterized in that: MsRFCFN network includes feature extraction module and fusion module; The feature extraction module includes a basic convolution part and a multi-scale convolution part; wherein the basic convolution part includes a first layer of small-scale convolution, a second layer of small-scale convolution, a first layer of dilated convolution and a second layer of dilated convolution; The convolution kernel size of the first and second layers of small-scale convolution is 3×3, the number of convolution kernels is 64, and the activation function is ReLU. The convolution kernel size of the first and second layers of dilated convolution is 3×3, the number of convolution kernels is 64, and the activation function is ReLU. The dilation ratio of the first layer of dilated convolution is 2, and the dilation ratio of the second layer of dilated convolution is 3. The multi-scale convolution part includes a set number of multi-scale convolution blocks; The first layer of the fusion module is a 1×1 convolution layer, the number of convolution kernels is 128, and the activation function is ReLU; the second layer of the fusion module is a 3×3 convolution layer, the number of convolution kernels is 1, and the activation function is ReLU.

5. The target tracking method in a multi-target multi-source information fusion scenario according to claim 1, characterized in that: The track measurement segments are segmented based on a preset sliding window and step size, and the track residual is determined based on the segmented sub-track information using an error characteristic estimation algorithm, specifically including: Segmenting the track measurement segment based on a preset sliding window and step size to obtain multiple sub-track information segments; Bezier curve is used to linearly fit any two sub-track information to obtain the approximate track of the real sub-track information; A track residual between the sub-track information and true sub-track information is extracted based on the approximate track.

6. The target tracking method in a multi-target multi-source information fusion scenario according to claim 5, characterized in that: The expression of the approximate track is: in, is the approximate track; is the Bernstein basis function; t is a variable parameter; R i is the control point of the approximate trajectory; n is the order of the Bezier curve; i is the order number of the Bezier curve.

7. The target tracking method in a multi-target multi-source information fusion scenario according to claim 1, characterized in that: Track reconstruction and fusion splicing are performed based on the fusion residual to obtain a fused track, specifically including: Normalizing the fusion residual to obtain a normalized fusion residual; Using the formula: Reconstructing the track according to the normalized fusion residual to obtain a fused sub-track; in, is the fusion sub-track; is the approximate track; is the normalized fusion residual; a is the vector composed of the minimum value of the track residual; b is the vector composed of the maximum value of the track residual; Based on the status of the fused sub-tracks, fusion and splicing processing is performed to obtain the fused track; The expression corresponding to the state of the fused sub-track is: The expression of the fused track is: in, is the state of the fused sub-track at time i1; is the total number of local sub-tracks at time i1; is the state value of the j-th sub-track; T f To fuse the tracks; is the state of the fused sub-track at the first moment; is the state of the fused sub-track when the track length is N.

8. The target tracking method in a multi-target multi-source information fusion scenario according to claim 1, characterized in that: The loss function adopts the average root mean square error function; the expression of the average root mean square error function is: Where ARMSE is the average root mean square error; N is the track length; M is the number of tracks in the dataset; is the y-axis position of the qth true track at time p; is the y-axis position of the fused track corresponding to the qth real track at time p.

9. A target tracking device in a multi-target multi-source information fusion scenario, characterized in that: include: An information data acquisition module is used to acquire information data of the target group to be tracked; The information data is obtained based on multiple observation platforms; The information data includes: visible light images and track data information; the target group to be tracked includes multiple targets to be tracked; a processing module, configured to perform parallel processing on the information data using a DBSCAN algorithm and a neural network to obtain a track association result; the track association result includes: a track measurement segment and a target to which the track measurement segment belongs; a track residual determination module, configured to segment the track measurement segments based on a preset sliding window and step size, and determine the track residual based on the segmented sub-track information using an error characteristic estimation algorithm; A fusion residual determination module is used to determine a fusion residual based on the track residual based on a trained MsRFCFN network; the trained MsRFCFN network is obtained by training the parameters of the MsRFCFN network based on a loss function using a BP algorithm with the goal of minimizing the loss convergence; The fusion track determination module is used to perform track reconstruction and fusion splicing processing based on the fusion residual to obtain a fusion track; the fusion track is used to achieve target tracking.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target tracking method in a multi-target multi-source information fusion scenario according to any one of claims 1 to 8.

Images