A multi-target tracking method for unmanned system across cameras based on multi-information matching

Abstract

The application discloses a multi-target tracking method for multi-camera of unmanned system based on multi-dimensional information matching, and relates to the fields of image processing and motion analysis.The main scheme is as follows: a target detection model is used to obtain the position and category of a detection frame of the target; an image processing method is used to identify the appearance color of the target; a feature extraction network is established to capture the deep appearance feature of the target; the target is tracked under a single camera by using multi-dimensional constraints such as appearance feature information, position information, attribute constraint, motion information constraint and maximum similarity threshold constraint; and every 60 frames, the multi-target tracking results of multiple single cameras are sent to a multi-camera multi-target fusion module, and the target is matched by using multi-dimensional information such as appearance feature information, attribute constraint, camera space topology constraint, target motion time constraint and maximum similarity threshold constraint, so as to realize the multi-target tracking of the unmanned system across cameras.

Description

Technical Field

[0001] This invention relates to a multi-target tracking method, and more particularly to a multi-camera multi-target tracking method for unmanned systems based on multi-source information matching. Background Technology

[0002] With the development of digital technology, multi-target tracking has been widely applied in video surveillance, autonomous driving, intelligent transportation systems, and modern military fields, becoming a hot research topic. The main task of multi-target tracking is to associate numerous targets in a video or video stream, assigning each target a unique identifier (ID), and continuously maintaining the target's ID. Currently, single-camera multi-target tracking systems are relatively mature, but due to the limited monitoring range of a single camera, continuous target tracking is impossible, resulting in significant limitations. Cross-camera multi-target tracking systems, by increasing the number of cameras, can overcome the inherent shortcomings of single-camera multi-target tracking systems, achieving long-term continuous target tracking, which has strong practical significance. However, due to problems such as complex background interference, target occlusion and disappearance, and significant changes in target scale and pose under different cameras, cross-camera multi-target tracking remains an unsolved problem even after years of effort, requiring further research.

[0003] Currently, most cross-camera target tracking methods involve two stages:

[0004] (1) Local Trajectory Generation Stage. This stage tracks each detected target within a single camera and generates a local trajectory for it. Single-camera target tracking methods are mainly divided into traditional methods and deep learning-based methods. Traditional target tracking methods include optical flow, particle filtering, and Kalman filtering. These methods have relatively fast tracking speeds, but they generally only obtain the motion features of the target by establishing a motion model, lacking relevant processing for changes in target scale and failing to extract the visual appearance features of the target in the image region, thus generally resulting in poor tracking performance. With the continuous development of artificial intelligence, deep learning-based methods have achieved better results in multi-target tracking problems and have gradually become the mainstream algorithm in the field of multi-target tracking. Classic algorithms include SORT, Deep SORT, and MOTDT. These methods generally first detect moving targets, then predict their motion trajectories through filtering algorithms, establish motion and appearance models, and simultaneously perform correlation matching of multiple targets from both motion and appearance feature perspectives, greatly improving the accuracy of multi-target tracking. However, the accuracy of these tracking methods depends on the accuracy of target detection; if the detection algorithm is ineffective, the tracking performance will also be unsatisfactory.

[0005] (2) Cross-camera trajectory matching stage. This stage matches all local trajectories of multiple targets across all cameras to generate their complete trajectories. Some methods continuously match local trajectories between adjacent cameras, eventually achieving trajectory matching across all cameras. Other methods iteratively match local trajectories across all cameras using methods such as bipartite graph matching or linear programming. Furthermore, some research attempts to find a global solution for trajectory matching using Bayes' theorem or graphical models, obtaining the global trajectory of each target by maximizing the posterior probability or finding the network flow from the source node to the destination node. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a multi-target tracking method for unmanned systems across cameras by matching multiple information, which addresses the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, this invention discloses a multi-camera multi-target tracking method for unmanned systems based on multi-source information matching, comprising: a target detection module, a feature extraction module, a color recognition module, a single-camera multi-target tracking module, and a cross-camera multi-target fusion module; wherein the target detection module is used for target detection, the feature extraction module is used for target feature extraction, the color recognition module is used for target color extraction, the single-camera multi-target tracking module is used to obtain single-camera multi-target tracking results, and the cross-camera multi-target fusion module is used to complete cross-camera multi-target tracking;

[0008] The five modules perform the following steps:

[0009] Step 1, Object Detection; A deep learning-based object detection model is established to obtain the location and category of all objects in an image from a single camera; the object detection method includes:

[0010] YOLOv5 is used as the object detection model to detect the location and category of the object. After obtaining the location and category of the object, a non-maximum suppression algorithm is used to obtain the detection results, thus completing the object detection.

[0011] Step 2, Target Color Extraction; All targets detected in Step 1 are traversed, and image processing methods are used to identify the appearance color of the targets; the target color extraction method includes:

[0012] Image processing methods are used to identify the target color. The image region containing the target is converted from RGB space to HSV space. The number of pixels occupied by each color is counted. By comparison, the color that occupies the most area is identified as the target color.

[0013] Step 3, target feature extraction; all targets detected in Step 1 are traversed, and ResNet18 network structure is used as the backbone network to extract multi-dimensional features of the targets.

[0014] The method for extracting the target features includes:

[0015] A feature extraction network is constructed to extract target features. The feature extraction network uses a ResNet18 network structure as the backbone network, consisting of 17 convolutional layers and one fully connected layer. The specific steps are as follows:

[0016] Step 3-1, use the output U∈R of the last convolutional layer Conv17 in the feature extraction network. 512×7×7 As the input to the attention fusion pool module in the feature extraction module, R 512×7×7 Represents the target feature matrix;

[0017] Step 3-2: Divide the input U of the attention fusion pool module into 7 blocks, each block using V. i1 ∈R 512×7 , i1∈(1,7) means, where i1 represents the i1th block;

[0018] Step 3-3, each V block i1 The output pool is obtained by passing the data through a max pooling layer and a fully connected layer, respectively. i1 and W i1 The formula is as follows:

[0019] Pool i1 =max(V i1 )∈R 512

[0020] W i1 =FC(V i1 )∈R 512×2

[0021] Where max(V) i1 ) represents the maximum vector matrix of regional target features, FC(V) i1 ) represents the elements of the connection layer matrix;

[0022] Steps 3-4, convert each output W i1 Divided into W according to the method of extracting by row i11 and W i12 Two parts, W i11 Weight information is generated after the Sigmoid function is applied, and then compared with W. i12 Performing a dot product yields a result similar to that of the Pool. i1 Perform addition, implement skip connections, and obtain the result Y. i1 ∈R 512 ;

[0023] Steps 3-5, obtain the result Y i1 By concatenating the components, we obtain Y∈R 512×7 ;

[0024] Steps 3-6, for the 7 V blocks i1 Performing the same concatenation operation, we finally obtain the feature vector Z∈R. 512 .

[0025] Step 4, Target Position Prediction and Analysis; using the Kalman filter algorithm, the target position is predicted and updated based on the constant velocity model and the linear observation model; the method for target position prediction and analysis includes:

[0026] The Kalman filter algorithm is used to predict and update the target position using a constant velocity model and a linear observation model.

[0027] The prediction formula is as follows:

[0028] x'=Fx

[0029] P' = FPF T +Q

[0030] Where x is the mean value of the target at the previous frame, expressed as an 8-dimensional vector. The characterization is defined as follows: where m and n represent the center coordinates of the target location bounding box, r is the aspect ratio, and h is the height. Let m, n, r, h be the velocity information corresponding to the image coordinates, x' be the mean value of the target predicted at the current frame time, F be the state transition matrix, P be the covariance of the target at the previous frame time, which is used to represent the uncertainty of the target position information and is characterized by an 8×8 diagonal matrix, Q be the motion estimation error, and P' be the covariance of the target at the current time.

[0031] The updated formula is as follows:

[0032]

[0033] x″=x'+K(z-Hx')

[0034] P″=(I-KH)P'

[0035] Where H is the transformation matrix from state variables to detection variables in dimension 4×8, R is the detection noise covariance, z is the target location detection box, x″ is the updated mean of the target, and P″ is the updated covariance of the target; the updated mean and updated covariance of the target are used to predict the target's motion position in the next frame.

[0036] Step 5, Multi-target tracking; Using appearance feature information, position information, and multi-dimensional constraints, perform multi-target tracking of the target under a single camera; Obtain the single-camera multi-target tracking result;

[0037] The single-camera multi-target tracking method includes:

[0038] The target in the current frame is used as the detector, and targets in previous frames are used as trackers. During the tracking process, i.e., the matching process between the detector and the tracker, the tracking status of all trackers is recorded. If a tracker fails to match for more than a threshold time, it is marked as deleted. The detector only performs association matching with trackers in the non-deleted state. The matching method includes:

[0039] Two matching methods are used; the first matching uses the cosine distance between the feature vectors of the detector and the tracker as a similarity criterion, and the second matching uses the generalized intersection-union ratio between the detector position and the tracker's predicted position as a similarity criterion.

[0040] First, a cost matrix is ​​constructed using a similarity metric. Second, the cost matrix is ​​constrained using category information, color information, motion information, and the maximum similarity threshold. Finally, a bipartite graph matching algorithm, namely Hungarian matching, is used to perform correlation matching on the cost matrix to complete multi-target tracking with a single camera.

[0041] The multi-target tracking method includes:

[0042] After obtaining the cost matrix for the first tracking by calculating the cosine distance between the feature vectors of the detector and the tracker, the cost matrix is ​​constrained by category, color, motion information and maximum threshold respectively; the second tracking steps are similar to the first tracking.

[0043] The method for constraining the cost matrix includes:

[0044] Step 5-1, the cost matrix is ​​represented by Cost∈R M*N The expression represents M detectors and N trackers, where c is the element in the cost matrix. i，j Let represent the cosine distance between the feature vectors of the i-th detector and the j-th tracker;

[0045] Step 5-2: Constrain the cost matrix; first, determine whether the category or color of the i-th detector and the j-th tracker are consistent. If they are inconsistent, then adjust the corresponding c... i，j Set to infinity;

[0046] Step 5-3: Calculate the Mahalanobis distance d between the position of the i-th detector and the predicted position of the j-th tracker. i，j The formula is as follows:

[0047] d i，j =(d i -s j ) T V j -1 (d i -s j )

[0048] Where, d i s represents the position of the i-th detection box. j V represents the predicted position of the j-th tracker. j This represents the covariance matrix between the detected location and the predicted tracking location;

[0049] Step 5-4: When calculating the Mahalanobis distance between the i-th detector and the j-th tracker, determine whether the Mahalanobis distance is greater than a threshold; if it is greater than the threshold, then set the corresponding c... i，j Set it to infinity to constrain the motion information;

[0050] Step 5-5, determine c i，j Is it greater than the threshold? If it is, then change the corresponding c. i，j Set it to infinity.

[0051] The multi-dimensional constraints include: attribute constraints, motion information constraints, and maximum similarity threshold constraints.

[0052] Step 6, cross-camera multi-target tracking: At set intervals, the single-camera multi-target tracking results of each camera are sent to the cross-camera multi-target fusion module. The module uses multi-dimensional information, including appearance feature information, attribute constraints, camera spatial topology constraints, target motion time constraints, and maximum similarity threshold constraints, to match targets and complete cross-camera multi-target tracking.

[0053] The cross-camera multi-target fusion module is used to associate the same target between different cameras and assign the same identifier ID to the same target.

[0054] For target information within the set time interval, the target is first compared with the previous matching result. If a match has been made, the target identifier is modified to the matched identifier. If the target has not been matched before, the cosine distance between targets from different cameras is calculated using feature vectors to obtain the cost matrix. Then, the cost matrix is ​​constrained using target attribute information, spatial topology information between cameras, and temporal information of target motion. After that, the Hungarian matching algorithm is used to associate targets under different cameras, and the matching result is recorded for the next cross-camera target tracking.

[0055] Beneficial effects:

[0056] (1) This invention uses ResNet as the backbone network when extracting target features. Compared with other classic convolutional neural networks (such as AlexNet and VGG-Net), ResNet achieves a relatively good balance between real-time performance and accuracy. In order to further mine the deep features of the target, this invention designs an attention fusion pool module. Compared with reducing three-dimensional features to one-dimensional features through simple average pooling or max pooling, this module can better retain key features and eliminate the influence of redundant information on the algorithm performance, greatly improving the model's representation ability.

[0057] (2) This invention employs two strategies, cosine distance and generalized intersection-union (CIU), to measure the correlation between multiple targets under a single camera. Cosine distance can better measure the appearance correlation between targets compared to Euclidean distance, while CIU can better measure the positional correlation between targets compared to CIU. The combination of the two methods helps to reduce the occurrence of mismatches or insufficient matches.

[0058] (3) This invention constrains target matching under a single camera by using multiple conditions such as category information, color information, motion information, and maximum threshold, reducing unnecessary matching and lowering the probability of false matching. By using target attribute information, camera spatial topology information, and target motion temporal information to constrain target matching across cameras in multiple dimensions, the accuracy of association matching is greatly improved.

[0059] (4) While performing single-camera multi-target tracking, the present invention sends the tracking results of each single camera to the cross-camera multi-target fusion module every 60 frames to associate the targets between different cameras, balancing real-time performance and accuracy. It not only consumes less resources and has a fast calculation speed, but also has high accuracy and robustness. Attached Figure Description

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0061] Figure 1 This is a schematic diagram of the multi-target tracking method across cameras according to the present invention.

[0062] Figure 2 This is a schematic diagram of the single-camera multi-target tracking process of the present invention.

[0063] Figure 3 This is a schematic diagram of the target feature extraction network structure of the present invention. Detailed Implementation

[0064] The principle of this invention is as follows: This invention provides a multi-camera multi-target tracking method based on multi-source information matching, mainly divided into five modules: target detection, feature extraction, color recognition, single-camera multi-target tracking, and cross-camera multi-target fusion. First, single-camera targets are detected, features extracted, color recognized, and tracked. Then, every 60 frames, the tracking results of each single camera are sent to the cross-camera multi-target fusion module to perform cross-camera fusion of multiple targets and record the tracking results. This process is repeated to achieve cross-camera multi-target tracking.

[0065] This invention provides a multi-camera multi-target tracking method based on multi-source information matching, mainly consisting of five modules: target detection, feature extraction, color recognition, single-camera multi-target tracking, and cross-camera multi-target fusion. For example... Figure 1 As shown, this method first detects, extracts features, recognizes colors, and tracks the target from a single camera. Then, every 60 frames, the tracking results from each single camera are sent to a cross-camera multi-target fusion module to perform cross-camera fusion of multiple targets and record the tracking results. This process is repeated to achieve cross-camera multi-target tracking.

[0066] This invention utilizes deep learning to establish an object detection model, acquiring the location and category of all targets in an image from a single camera. YOLOv5 is used as the object detection model, which is based on a neural network model to detect the location and category of targets from an entire image, achieving end-to-end object detection. After obtaining the location information features, to filter out high-confidence localization results, exclude multiple localizations of the same target, and eliminate low-confidence results, a Non-Maximum Suppression (NMS) algorithm is employed to obtain the detection results.

[0067] All targets are traversed, and the colors of the targets are identified using traditional image processing methods.

[0068] The process iterates through all targets, extracting their visual features using a feature extraction network. This invention uses a ResNet network as the backbone, then feeds the targets into an attention fusion pool module. The main function of this module is to fuse the three-dimensional features of the targets into one-dimensional features. Compared to simple max pooling or average pooling, this attention fusion pool module not only eliminates a large amount of redundant information and retains key information, but also highlights the weight of key information within the overall information, further extracting attention from the image.

[0069] The Kalman filter algorithm is used to predict the motion position of a target in the previous frame in the current frame.

[0070] Perform single-camera multi-target tracking, such as Figure 2As shown, for convenience, the target in the current frame is called the detector, and the targets in previous frames are called trackers. During the matching process, this invention records the tracking status of all trackers. If a tracker has not been matched for more than 60 frames, it means that the target has left the camera area, and therefore, there is no need to match the detector with this tracker anymore; the tracker is marked as deleted. Therefore, in actual matching, the detector will only associate and match with trackers that are not deleted. The main process is as follows:

[0071] To measure the similarity between targets, this invention employs two matching steps. The first matching uses the cosine distance between the feature vectors of the detector and tracker as the similarity criterion, while the second matching uses the Generalized Intersection Over Union (GIOU) between the detector position and the tracker's predicted position as the similarity criterion. Specifically, a cost matrix is ​​first constructed using the similarity criterion. Then, category information, color information, motion information, and a maximum similarity threshold are used to constrain the cost matrix. Finally, a bipartite graph matching algorithm, namely Hungarian matching, is used to perform correlation matching on the cost matrix to achieve multi-target tracking.

[0072] The target state is updated using a Kalman filter algorithm for tracking and matching in the next frame of a single camera.

[0073] Every 60 frames, the results of multi-target tracking from all individual cameras are sent to the cross-camera target fusion module. The main function of this module is to associate the same targets across different cameras and assign them the same identifier ID. For the target information every 60 frames, the target is first compared with the previous matching results. If the target has been matched before, no further matching is needed; simply change its identifier to the matched identifier. If the target has not been matched before, the cosine distance between the targets from different cameras is calculated using feature vectors to obtain the cost matrix. Then, the cost matrix is ​​constrained using target attribute information, spatial topology information between cameras, and temporal information of target motion. Finally, the Hungarian matching algorithm is used to associate the targets from different cameras, and the matching results are recorded for use in the next cross-camera target tracking.

[0074] An example of a method for tracking multiple targets across multiple cameras includes the following steps:

[0075] Step 1: Target Color Extraction. Image processing methods are used to identify the target color, converting the image region containing the target from RGB space to HSV space. Images acquired by cameras in natural environments are easily affected by natural lighting, occlusion, and shadows, making them sensitive to brightness. Since all three components of the RGB color space are closely related to brightness, changes in brightness will cause corresponding changes in all three components. Therefore, this invention does not use the RGB space when identifying the target color. The HSV space can intuitively express the hue, vividness, and lightness of a color, facilitating color comparison. It consists of three parts: hue, saturation, and lightness. Because different colors have different ranges in the HSV space, this invention uses OpenCV tools to capture the distribution areas of common color channels, thus counting the pixels occupied by each color. By comparison, the color occupying the largest area is identified as the target color.

[0076] Step 2: Target feature extraction, such as Figure 3 As shown, the feature extraction network uses a ResNet18 network structure as its backbone. It mainly consists of 17 convolutional layers (Conv1-Conv17) and one fully connected layer (FC18). This method uses the output U∈R of the last convolutional layer Conv17. 512×7×7 This serves as the input to the attention fusion pool module. The attention fusion pool module first divides the input U into 7 blocks, each using V... i ∈R 512×7 Let i∈(1,7) represent the condition. Then, each block is processed through a max pooling layer and a fully connected layer to obtain the output Pool. i and W i The formula is as follows:

[0077] Pool i =max(V i )∈R 512

[0078] W i =FC(V i )∈R 512×2

[0079] Each W i Divided into W i1 and W i2 W i1 Weight information is generated after the Sigmoid function is applied, and then compared with W. i2 Perform a dot product to highlight W. i2 The key information in the pool, the results obtained are related to the pool i Perform addition, implement skip connections, and obtain the result Y. i ∈R 512 .

[0080]

[0081] The obtained Y i By concatenating the components, we obtain Y∈R 512×7 , compared with V just now i The operations are similar, and finally the feature vector Z∈R is obtained. 512 .

[0082] Step 3: Target Position Prediction and Analysis. In single-camera multi-target tracking, the Kalman filter algorithm is used to predict and update the target position using a constant velocity model and a linear observation model. The prediction formula is as follows:

[0083] x'=Fx

[0084] P' = FPF T +Q

[0085] Where x is the mean value of the target at the previous frame, expressed as an 8-dimensional vector. The values ​​represent the target's bounding box center coordinates, aspect ratio, height, and corresponding velocity information in the image coordinates. x' is the predicted mean value of the target in the current frame. When initializing the mean value for the first frame, since there is no previous frame, its (m, n, r, h) is set to the target's position information in the first frame, and the corresponding velocity is initialized to 0. F represents the state transition matrix. P is the target's covariance in the previous frame, used to represent the uncertainty of the target's position information, characterized by an 8×8 diagonal matrix. The larger the matrix value, the higher the uncertainty. Q is the motion estimation error.

[0086] The updated formula is as follows:

[0087]

[0088] x″=x'+K(z-Hx')

[0089] P″=(I-KH)P'

[0090] Where H is the transformation matrix from the state variable to the detection variable in dimension 4×8, R is the detection noise covariance, z is the target location detection box, x″ is the updated mean of the target, and P″ is the updated covariance of the target. The updated mean and covariance of the target are used to predict the target's motion position in the next frame.

[0091] Step 4: Single-camera multi-object tracking. In single-camera multi-object tracking, after obtaining the cost matrix for the initial tracking by calculating the cosine distance between the feature vectors of the detector and the tracker, the cost matrix is ​​constrained by category, color, motion information, and a maximum threshold. The secondary tracking steps are similar. Specifically, the cost matrix is ​​represented by Cost∈R M*NThe expression represents M detectors and N trackers, where c is the element in the cost matrix. i，j Let represent the cosine distance between the feature vectors of the i-th detector and the ,-th tracker.

[0092] When constraining the cost matrix, first determine whether the category or color of the i-th detector is consistent with that of the ,j-th tracker. If they are inconsistent, then adjust the corresponding c. i，j Set to infinity (in practice, set to 1×10). 6 (The same applies below).

[0093] Then, the Mahalanobis distance d between the position of the i-th detector and the predicted position of the -th tracker is calculated. i，j The formula is as follows:

[0094] d i，j =(d i -s j ) T V j -1 (d i -s j )

[0095] Where, d i s represents the position of the i-th detection box. j V represents the predicted position of the j-th tracker. j This represents the covariance matrix between the detected location and the predicted tracking location.

[0096] When the Mahalanobis distance between the i-th detector and the j-th tracker is calculated, it is determined whether this Mahalanobis distance is greater than the user-defined maximum Mahalanobis distance threshold. If it is greater, the corresponding c is adjusted. i，j Setting it to infinity constrains the motion information.

[0097] Finally, determine c. i，j Is it greater than the custom maximum cosine distance threshold? If it is, then the corresponding c... i，j Set it to infinity.

[0098] Based on the above steps, the cost matrix is ​​constrained, reducing the matching between unrelated targets.

[0099] Step 5: Cross-camera multi-target tracking. In cross-camera multi-target tracking, the association method is similar to that of single-camera multi-target tracking. It involves constructing a cost matrix, constraining the cost matrix, and using the Hungarian algorithm to associate the cost matrices to obtain matching results. However, since the cross-camera situation is more complex than the single-camera situation, the constraints under the single-camera method are no longer used in the cross-camera case. This invention designs different constraints for the cross-camera situation, specifically as follows: First, the target's attribute information is used as a constraint, including category and color. If the attributes of targets are inconsistent, they are definitely not the same target, and no further matching is needed, thus achieving attribute constraints. Second, the spatial topological positions between multiple cameras are used as prior knowledge. Multiple cameras are numbered. When tracking a target with a single camera, the target's motion direction is estimated through its trajectory under the single camera, and the number of the previous camera it appeared before entering the current camera, and the number of the next camera it enters after leaving the current camera, i.e., where the target came from and where it is going. When performing cross-camera matching, the cost matrix is ​​constrained by the spatial motion states of detectors and trackers. Specifically, assuming a detector is located at camera 2, the origin of the detector is first determined by obtaining the previous camera number, let's say 1. Then, a first positional constraint is applied: this detector is matched only with trackers located at camera 1. Since trackers at camera 1 may be moving to camera 2 or other cameras, a second constraint is applied: this detector is matched only with trackers located at camera 1 that are moving to camera 2, further narrowing the matching range and achieving spatial constraint. Next, the target's motion time information is used for constraint. The target's velocity is estimated by comparing the positional change of the target's center point between consecutive frames and the duration of each frame. The minimum velocity of the target under the current camera is obtained, and the distance between the current camera position and the next camera position the target is moving to is calculated. The maximum time it takes for the target to travel from the current camera to the next camera is obtained by combining the distance between cameras and the minimum velocity. Therefore, in actual matching, it is necessary to determine the time interval between the detector and the tracker. If this time interval is greater than the maximum duration, it indicates that they are not the same target, and no matching is required, thus achieving time constraint. Finally, the similarity between targets is used for constraint. If the cosine distance between the detector and the tracker is greater than the specified maximum threshold, it indicates that the differences between the targets are large and the similarity is low, meaning they are not the same target, and no further matching is performed, thus achieving further constraint.

[0100] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a multi-target tracking method for unmanned systems using multi-source information matching across cameras, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0101] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0102] This invention provides a concept and method for multi-target tracking across cameras in an unmanned system using multi-source information matching. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A multi-target tracking method for unmanned systems across cameras using multi-modal information matching, the method comprising: include: Target detection module, feature extraction module, color recognition module, single-camera multi-target tracking module, and cross-camera multi-target fusion module; The target detection module is used for target detection, the feature extraction module is used for target feature extraction, the color recognition module is used for target color extraction, the single-camera multi-target tracking module is used to obtain the single-camera multi-target tracking result, and the cross-camera multi-target fusion module is used to complete cross-camera multi-target tracking. Perform the following steps: Step 1, Target Detection; A target detection model based on deep learning methods is established to obtain the location and category of all targets in an image from a single camera; Step 2, target color extraction; traverse all targets detected in Step 1 and use image processing methods to identify the appearance color of the targets; Step 3, target feature extraction; All targets detected in step 1 are traversed, and ResNet18 network structure is used as the backbone network to extract multi-dimensional features of the targets. Step 4, Target position prediction and analysis; using the Kalman filter algorithm, the target position is predicted and updated using a constant velocity model and a linear observation model; Step 5, multi-target tracking; Multi-target tracking using a single camera is achieved by utilizing appearance feature information, location information, and multi-dimensional constraints. Obtain multi-target tracking results from a single camera; Step 6, multi-target tracking across cameras; At set intervals, the single-camera multi-target tracking results of each camera are sent to the cross-camera multi-target fusion module. The module uses multi-dimensional information, including appearance feature information, attribute constraints, camera spatial topology constraints, target motion time constraints, and maximum similarity threshold constraints, to match targets and complete cross-camera multi-target tracking. The specific method for multi-target tracking under a single camera in step 5 includes: The target in the current frame is used as the detector, and targets in previous frames are used as trackers. During the tracking process, i.e., the matching process between the detector and the tracker, the tracking status of all trackers is recorded. If a tracker fails to match for more than a threshold time, it is marked as deleted. The detector only performs association matching with trackers in the non-deleted state. The matching method includes: Two matching methods are used; the first matching uses the cosine distance between the feature vectors of the detector and the tracker as a similarity criterion, and the second matching uses the generalized intersection-union ratio between the detector position and the tracker's predicted position as a similarity criterion. First, a cost matrix is ​​constructed using a similarity metric. Second, the cost matrix is ​​constrained using category information, color information, motion information, and the maximum similarity threshold. Finally, a bipartite graph matching algorithm, namely Hungarian matching, is used to perform correlation matching on the cost matrix to complete multi-target tracking with a single camera. The multi-target tracking method includes: After obtaining the cost matrix for the first tracking by calculating the cosine distance between the feature vectors of the detector and the tracker, the cost matrix is ​​constrained by category, color, motion information and maximum threshold respectively; the second tracking steps are the same as the first tracking. The method for constraining the cost matrix includes: Step 5-1, the cost matrix is ​​used The expression represents the number of detectors (M) and trackers (N), where M represents M detectors and N represents N trackers. The elements in the cost matrix are... Let represent the cosine distance between the feature vectors of the i-th detector and the j-th tracker; Step 5-2: Constrain the cost matrix; first, determine whether the category or color of the i-th detector and the j-th tracker are consistent. If they are inconsistent, then adjust the corresponding... Set to infinity; Step 5-3: Calculate the Mahalanobis distance between the position of the i-th detector and the predicted position of the j-th tracker. The formula is as follows: ； in, This indicates the position of the i-th detection box. This represents the predicted position of the j-th tracker. This represents the covariance matrix between the detected location and the predicted tracking location; Step 5-4: When calculating the Mahalanobis distance between the i-th detector and the j-th tracker, determine whether the Mahalanobis distance is greater than a threshold; if it is greater than the threshold, then... Set it to infinity to constrain the motion information; Step 5-5, Determine Is it greater than the threshold? If it is, then the corresponding... Set to infinity; The cross-camera multi-target fusion module described in step 6 is used to associate the same target between different cameras and assign the same identifier ID to the same target; For target information within the set time interval, the target is first compared with the previous matching result. If a match has been made, the target identifier is modified to the matched identifier. If the target has not been matched before, the cosine distance between targets from different cameras is calculated using feature vectors to obtain the cost matrix. Then, the cost matrix is ​​constrained using target attribute information, spatial topology information between cameras, and temporal information of target motion. After that, the Hungarian matching algorithm is used to associate targets under different cameras, and the matching result is recorded for the next cross-camera target tracking.

2. The method for multi-target tracking across cameras in an unmanned system based on multi-source information matching according to claim 1, characterized in that, The target detection method described in step 1 includes: YOLOv5 is used as the object detection model to detect the location and category of the object. After obtaining the location and category of the object, a non-maximum suppression algorithm is used to obtain the detection results, thus completing the object detection.

3. The multi-target tracking method for unmanned systems across cameras based on multi-source information matching according to claim 2, characterized in that, The method for extracting the target color in step 2 includes: Image processing methods are used to identify the target color. The image region containing the target is converted from RGB space to HSV space. The number of pixels occupied by each color is counted. By comparison, the color that occupies the most area is identified as the target color.

4. The multi-target tracking method for unmanned systems across cameras based on multi-source information matching according to claim 3, characterized in that, The target feature extraction method described in step 3 includes: A feature extraction network is constructed to extract target features; the feature extraction network adopts the ResNet18 network structure as the backbone network; it includes 17 convolutional layers and one fully connected layer.

5. The multi-target tracking method for unmanned systems across cameras based on multi-source information matching according to claim 4, characterized in that, The target feature extraction method described in step 3 includes: Step 3-1: Use the output of the last convolutional layer Conv17 in the feature extraction network. As the input to the attention fusion pool module in the feature extraction module, wherein, Represents the target feature matrix; Step 3-2: Divide the input U of the attention fusion pool module into 7 blocks, each block using... It means that among them Indicates the first piece; Step 3-3, each piece The outputs are obtained by passing the material through a max pooling layer and a fully connected layer, respectively. and The formula is as follows: ； ； in, This represents the maximum vector matrix of regional target features. Represents the elements of the connection layer matrix; Steps 3-4, for each output Divided according to the method of extracting by row and Two parts, Weight information is generated after the Sigmoid function is applied, and... Performing a dot product yields the same result as... Perform addition, implement skip connections, and obtain the result. ; Steps 3-5: Obtain the results By splicing them together, we can obtain ; Steps 3-6, for the 7 pieces Perform the same concatenation operation to obtain the feature vector. .

6. The multi-target tracking method for unmanned systems across cameras based on multi-source information matching according to claim 5, characterized in that, The method for target location prediction and analysis in step 4 includes: The Kalman filter algorithm is used to predict and update the target position using a constant velocity model and a linear observation model. The prediction formula is as follows: ； ； Where x is the mean value of the target at the previous frame, expressed as an 8-dimensional vector. The characterization is defined as follows: where m and n represent the center coordinates of the target location bounding box, r is the aspect ratio, and h is the height. The corresponding coordinates in the image Speed ​​information, The mean value predicted for the target at the current frame time. Represents the state transition matrix; Let covariance be the target's value at the previous frame, used to represent the degree of uncertainty in the target's position information. The diagonal matrix characterization, where Q is the motion estimation error. This represents the covariance of the target at the current moment; The updated formula is as follows: ； ； ； in, For dimension The transformation matrix from state variables to detection variables is given, and R is the detection noise covariance. The target location detection box, The target is the updated mean. The updated covariance of the target; the updated mean and updated covariance of the target are used to predict the target's motion position in the next frame.

7. The multi-target tracking method for unmanned systems across cameras based on multi-source information matching according to claim 6, characterized in that, The multi-dimensional constraints mentioned in step 5 include: attribute constraints, motion information constraints, and maximum similarity threshold constraints.

Images