A multi-target tracking method, device and electronic equipment

Abstract

The application provides a multi-target tracking method and device and electronic equipment to improve the accuracy of multi-target tracking. In the embodiments of the application, the track and target are associated by fusing the motion matching cost and the appearance matching cost. The motion matching cost can ensure the track continuity of the target under short-time occlusion or rapid motion, and the appearance matching cost can solve the identity confusion problem of the target intersection by using the semantic information of the target. The application adopts the Hungarian algorithm to solve the optimal matching relationship, which can minimize the total matching cost between all tracks and detected targets. Thus, the matching relationship between the target and the track is accurately and effectively determined, and the accuracy of multi-target tracking is improved.

Description

Technical Field

[0001] This application relates to the fields of image processing, computer vision, and multi-target tracking technology, and in particular to a multi-target tracking method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of computer vision technology and artificial intelligence, video target tracking technology has been widely used in fields such as intelligent video surveillance, autonomous driving, human-computer interaction, and remote sensing image analysis. Multi-target tracking, as a core task, aims to detect targets of interest in consecutive frames of a video sequence and assign a unique identifier to each target, thereby forming a complete motion trajectory.

[0003] Currently, mainstream multi-target tracking methods typically employ detection-based tracking. This method first uses a target detection model to detect targets in the current frame and obtain their position information. Then, it uses a data association algorithm to match the detected targets in the current frame with the existing trajectories in historical frames. However, this approach has low accuracy and cannot accurately track multiple targets. Summary of the Invention

[0004] This application provides a multi-target tracking method, apparatus, and electronic device to improve the accuracy of multi-target tracking.

[0005] In a first aspect, embodiments of this application provide a multi-target tracking method, the method comprising: The current frame image is acquired, the position information of each target in the current frame image is determined by the target detection model, and the current appearance feature vector of each target is extracted by the recognition model; the current tracked trajectories are acquired. Based on the pre-saved historical states of each trajectory, the estimated position information of each trajectory in the current frame image is predicted; based on the estimated position information of each trajectory and the position information of each target, the motion matching cost of each target matching each trajectory is determined; the historical appearance feature vector associated with each trajectory is obtained, and based on the similarity between the historical appearance feature vector of each trajectory and the current appearance feature vector of each target, the appearance matching cost of each target matching each trajectory is determined. Based on the motion matching cost and the appearance matching cost, the target of each trajectory matching is determined using the Hungarian algorithm.

[0006] The above-mentioned technical solution has the following advantages or beneficial effects: In this embodiment, the trajectory and target are associated by fusing motion matching cost and appearance matching cost. Motion matching cost can ensure the trajectory continuity of the target under short-term occlusion or rapid movement, while appearance matching cost can use the semantic information of the target to solve the identity confusion problem when targets cross. Furthermore, this application uses the Hungarian algorithm to solve the optimal matching relationship, which can minimize the total matching cost between all trajectories and the detected target. Thus, the matching relationship between the target and the trajectory is accurately and effectively determined, improving the accuracy of multi-target tracking.

[0007] In one possible implementation, predicting the estimated position information of each trajectory in the current frame image based on the pre-saved historical states of each trajectory includes: Based on the pre-saved historical states of each trajectory, the estimated position information and state covariance matrix of each trajectory in the current frame image are predicted; The step of determining the motion matching cost for each target to match each trajectory based on the estimated position information of each trajectory and the position information of each target includes: For each trajectory, based on the intersection-union ratio (IoU) between the estimated position information of the trajectory and the position information of each target, a first matching cost is determined for matching the trajectory with each target. Based on the sum of the variances of each component in the determined state covariance matrix of the trajectory, a second matching cost representing the uncertainty of the estimated position information of the trajectory is determined. For each target, based on the first matching cost and the second matching cost between the trajectory and the target, and their corresponding weights, a motion matching cost between the trajectory and the target is determined.

[0008] The above-mentioned technical solution has the following advantages or beneficial effects: This application embodiment determines the second matching cost by calculating the sum of the variances of each component in the state covariance matrix, which can quantify the uncertainty of the trajectory estimation position. When the prediction error of a trajectory increases due to prolonged occlusion or violent movement, the covariance matrix determined based on that trajectory will increase accordingly, thereby increasing the matching cost, reducing the probability of the trajectory being incorrectly associated, effectively suppressing identity switching phenomena, and thus improving the accuracy of multi-target tracking.

[0009] In one possible implementation, after acquiring the current frame image and before determining the location information of each target in the current frame image using a target detection model, the method further includes: The current frame image is cut into multiple overlapping sub-image slices; The step of determining the position information of each target in the current frame image through a target detection model includes: The target detection model is used to detect targets in each sub-image slice and the current frame image to obtain the first candidate position information of the locally detected target in each sub-image slice and the second candidate position information of the target in the current frame image. Each first candidate position information is mapped to the current frame image to obtain each third candidate position information. The non-maximum suppression algorithm is used to merge each third candidate position information and each second candidate position information to obtain the position information of each target in the current frame image.

[0010] The above-mentioned technical solution has the following advantages or beneficial effects: The embodiments of this application, through slicing, are equivalent to indirectly increasing the pixel density of the local area without changing the model input resolution, so that small targets that were originally blurry in the whole image compression have more significant features in the slice, thereby significantly reducing the false negative rate.

[0011] In one possible implementation, determining the location information of each target in the current frame image using a target detection model includes: The location information, category, and confidence level of each target in the current frame image are determined by a target detection model; wherein the category includes vehicles, pedestrians, and other categories. After extracting the current appearance feature vectors of each target through the recognition model, and before determining the motion matching cost of each target matching each trajectory, the method further includes: For each target, determine the current appearance feature vector of the target and its similarity with the target feature vector that is saved in advance for the category corresponding to the target. If the determined similarity is lower than the similarity threshold, or the confidence level of the target is lower than the confidence threshold, then the target is determined to be a misidentified target and the target is deleted.

[0012] The above-mentioned technical solution has the following advantages or beneficial effects: In this embodiment, the combination of confidence threshold and appearance feature similarity for joint screening can effectively intercept low-quality detection targets. This embodiment introduces similarity comparison with preset category feature vectors, which is equivalent to adding semantic consistency verification. This can significantly reduce the false detection rate caused by background clutter or similar objects. Furthermore, by deleting misidentified targets before calculating motion matching costs, it reduces erroneous matching attempts between invalid trajectories and noisy targets, thus reducing the computational burden of association algorithms such as the Hungarian algorithm.

[0013] In one possible implementation, determining the target for each trajectory matching using the Hungarian algorithm based on the motion matching cost and appearance matching cost of each target matching each trajectory includes: Based on the motion matching cost of each target matching each trajectory, the Hungarian algorithm is used to determine the motion target matching each trajectory; based on the appearance matching cost of each target matching each trajectory, the Hungarian algorithm is used to determine the appearance target matching each trajectory. For each trajectory, the target matching the trajectory is determined based on the moving target matching the trajectory and the appearance target matching the trajectory.

[0014] The above-mentioned technical solution has the following advantages or beneficial effects: the reliability of motion information and appearance information varies in different scenarios. Motion matching is more effective when the camera is stable and the target's motion pattern is obvious, while appearance matching is more advantageous when the target reappears after occlusion or when motion is blurred. This application's embodiment, by using the Hungarian algorithm for independent matching, can fully exploit the maximum potential of both the motion model and the appearance model, avoiding the direct interference of noise from one feature with the matching process of another. Even if one modality performs poorly, the other modality can still provide reliable matching results, thereby significantly improving the accuracy of multi-target tracking.

[0015] In one possible implementation, determining the target matching the trajectory based on the moving target matching the trajectory and the appearance target matching the trajectory includes: If a moving target matching the trajectory is the same as a visual target matching the trajectory, then the same target is determined to be the target matching the trajectory.

[0016] In one possible implementation, determining the target matching the trajectory based on the moving target matching the trajectory and the appearance target matching the trajectory includes: If the moving target matching the trajectory and the appearance target matching the trajectory are different, and the sum of the variances of each component of the state covariance matrix of the trajectory in the current frame image is greater than a first threshold, then the moving target matching the trajectory is determined as the target matching the trajectory; or, If the moving target matched by the trajectory is different from the appearance target matched by the trajectory, then it is determined whether the IoU between the position information of the moving target matched by the trajectory and the estimated position information of the trajectory is greater than a second threshold. If it is greater than the second threshold, the moving target matched by the trajectory is determined to be the target matched by the trajectory. If it is not greater than the second threshold, the appearance target matched by the trajectory is determined to be the target matched by the trajectory.

[0017] The above technical solution has the following advantages or beneficial effects: by adopting the method provided in the embodiments of this application, when the trajectory is in a state of high uncertainty, the final target is determined based on the moving target matching the trajectory, or, based on whether the IoU between the position information of the moving target matching the trajectory and the estimated position information of the trajectory is greater than a second threshold, the reliability of the motion prediction is determined, and then the final matching target is determined, thereby improving the accuracy of the matching target determination.

[0018] In one possible implementation, the historical appearance state vector of any trajectory contains the appearance state sub-vectors of the corresponding tracked target in multiple historical frames; determining the appearance matching cost of each target with each trajectory based on the similarity between the historical appearance feature vectors of each trajectory and the current appearance feature vectors of each target includes: For each trajectory, the similarity between each appearance state sub-vector contained in the trajectory and the current appearance feature vector of each target is determined. Based on multiple similarities and the weights corresponding to the historical frames corresponding to each appearance state sub-vector, the appearance matching cost of each target matching the trajectory is determined. The longer the interval between the historical frame and the current frame image, the smaller the corresponding weight.

[0019] The above-described technical solution has the following advantages or beneficial effects: The embodiments of this application calculate multiple similarities and perform weighted fusion, resulting in an appearance matching cost with higher statistical reliability than a single similarity value. The weighted summation method smooths out abnormal similarity values ​​from individual historical frames, making the final appearance matching cost more accurately reflect the correlation between the target and the trajectory. This helps to more accurately distinguish different targets in dense scenes or when there are many similar targets.

[0020] Secondly, embodiments of this application also provide a multi-target tracking device, the device comprising: The acquisition module is used to acquire the current frame image, determine the position information of each target in the current frame image through the target detection model, and extract the current appearance feature vector of each target through the recognition model; and acquire the currently tracked trajectories. The processing module is configured to: predict the estimated position information of each trajectory in the current frame image based on the pre-saved historical states of each trajectory; determine the motion matching cost of each target matching each trajectory based on the estimated position information of each trajectory and the position information of each target; obtain the historical appearance feature vector associated with each trajectory; determine the appearance matching cost of each target matching each trajectory based on the similarity between the historical appearance feature vector of each trajectory and the current appearance feature vector of each target; and determine the target matching each trajectory using the Hungarian algorithm based on the motion matching cost and the appearance matching cost.

[0021] In one possible implementation, the processing module is specifically used to predict the estimated position information and state covariance matrix of each trajectory in the current frame image based on the pre-saved historical states of each trajectory. The processing module is specifically configured to, for each trajectory, determine a first matching cost between the trajectory and each target based on the IoU between the estimated position information of the trajectory and the position information of each target, and determine a second matching cost characterizing the uncertainty of the estimated position information of the trajectory based on the sum of the variances of each component in the determined state covariance matrix of the trajectory; and for each target, determine a motion matching cost between the trajectory and the target based on the first matching cost and the second matching cost and their corresponding weights.

[0022] In one possible implementation, the processing module is further configured to cut the current frame image into multiple overlapping sub-image slices; The processing module is specifically used to perform target detection on each sub-image slice and the current frame image through a target detection model, to obtain the first candidate position information of the locally detected target in each sub-image slice and the second candidate position information of the target in the current frame image, to map each first candidate position information to the current frame image to obtain each third candidate position information, and to merge each third candidate position information and each second candidate position information through a non-maximum suppression algorithm to obtain the position information of each target in the current frame image.

[0023] In one possible implementation, the processing module is specifically used to determine the location information of each target in the current frame image, the category of each target, and the confidence level of each target through a target detection model; wherein the category includes vehicles, pedestrians, and other categories; The processing module is further configured to determine the similarity between the current appearance feature vector of each target and the target feature vector that is pre-saved for the category corresponding to the target. If the determined similarity is lower than the similarity threshold, or the confidence level corresponding to the target is lower than the confidence threshold, then the target is determined to be a misidentified target and the target is deleted.

[0024] In one possible implementation, the processing module is specifically configured to: determine the moving target matching each trajectory based on the motion matching cost of each target matching each trajectory using a Hungarian algorithm; determine the appearance target matching each trajectory based on the appearance matching cost of each target matching each trajectory using a Hungarian algorithm; and determine the target matching each trajectory based on the moving target matching the trajectory and the appearance target matching the trajectory.

[0025] In one possible implementation, the processing module is specifically configured to determine that the same target is the target matched by the trajectory if the moving target matching the trajectory is the same as the appearance target matching the trajectory.

[0026] In one possible implementation, the processing module is specifically configured to: if the moving target matching the trajectory and the appearance target matching the trajectory are different, and the sum of the variances of each component of the trajectory in the state covariance matrix of the current frame image is greater than a first threshold, then determine the moving target matching the trajectory as the target matching the trajectory; or, if the moving target matching the trajectory and the appearance target matching the trajectory are different, determine whether the IoU between the position information of the moving target matching the trajectory and the estimated position information of the trajectory is greater than a second threshold; if it is greater than the second threshold, then determine the moving target matching the trajectory as the target matching the trajectory; if it is not greater than the second threshold, then determine the appearance target matching the trajectory as the target matching the trajectory.

[0027] In one possible implementation, the processing module is specifically used to determine, for each trajectory, the similarity between each appearance state sub-vector contained in the trajectory and the current appearance feature vector of each target, and to determine the appearance matching cost of each target matching the trajectory based on multiple similarities and the weight corresponding to the historical frame corresponding to each appearance state sub-vector; wherein, the longer the interval between the historical frame and the current frame image, the smaller the corresponding weight.

[0028] Thirdly, embodiments of this application also provide an electronic device, which includes at least a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the steps of the multi-target tracking method as described in any of the preceding claims. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram illustrating the process of a multi-target tracking method provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of a process for obtaining multiple sub-image slices in an embodiment of the present application. Figure 3 A detailed process diagram of multi-target tracking provided for an embodiment of this application; Figure 4aThis is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 4b This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 4c This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 4d This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 5a This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 5b This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 5c This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 5d This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The present application will now be described in further detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0032] To improve the accuracy of multi-target tracking, embodiments of this application provide a multi-target tracking method, apparatus, and electronic device.

[0033] The multi-target tracking method includes: an electronic device acquiring a current frame image; determining the position information of each target in the current frame image through a target detection model; extracting the current appearance feature vector of each target through a recognition model; acquiring each currently tracked trajectory; predicting the estimated position information of each trajectory in the current frame image based on the pre-saved historical state of each trajectory; determining the motion matching cost of each target matching each trajectory based on the estimated position information of each trajectory and the position information of each target; acquiring the historical appearance feature vector associated with each trajectory; determining the appearance matching cost of each target matching each trajectory based on the similarity between the historical appearance feature vector of each trajectory and the current appearance feature vector of each target; and determining the target matched by each trajectory using the Hungarian algorithm based on the motion matching cost and the appearance matching cost.

[0034] Figure 1 This application provides a schematic diagram of a multi-target tracking method, which includes the following steps: S101: Obtain the current frame image, determine the position information of each target in the current frame image through the target detection model, and extract the current appearance feature vector of each target through the recognition model; obtain the currently tracked trajectories.

[0035] The multi-target tracking method provided in this application is applied to an electronic device, which can be a smart device such as a PC or a server.

[0036] To perform multi-target tracking, the electronic device can first acquire the image to be processed at the current moment, i.e., the current frame image, from the video acquisition device, and perform normalization preprocessing on the current frame image to adapt it to the input of the target detection model. The electronic device locally stores a pre-trained target detection model, and can input the current frame image into the pre-trained target detection model. The target detection model determines the position information of each potential target in the current frame image through forward inference. In one possible implementation, the position information can be the coordinates of a bounding box that completely contains the target, including pedestrians and vehicles. The electronic device also locally stores a pre-trained recognition model. In this embodiment, the electronic device also uses the pre-trained recognition model to detect each detected target region and extract the current appearance feature vector that can characterize the target's identity. In one possible implementation, the target detection model is a YOLOv12 detector.

[0037] The electronic device also acquires the currently tracked trajectories. In one possible implementation, it can retrieve and obtain a set of currently active tracking trajectories from a database. This set records the motion state of each target in historical frames, the unique identifier (ID) of the trajectory, etc., providing basic data for subsequent data association and status updates.

[0038] S102: Based on the pre-saved historical states of each trajectory, predict the estimated position information of each trajectory in the current frame image; based on the estimated position information of each trajectory and the position information of each target, determine the motion matching cost of each target matching each trajectory; obtain the historical appearance feature vector associated with each trajectory, and based on the similarity between the historical appearance feature vector of each trajectory and the current appearance feature vector of each target, determine the appearance matching cost of each target matching each trajectory.

[0039] The electronic device can pre-store historical states for each trajectory. In one possible implementation, the historical state of each trajectory records the motion data of the corresponding target in multiple past video frames, covering position, velocity, and acceleration. For each trajectory, the electronic device can predict the estimated position information of the target in the current frame image using a motion model (such as Kalman filtering). The electronic device calculates the spatial distance between the estimated position information and the position information of each target detected in the current frame image, thereby quantifying the motion matching cost. In one possible implementation, the Mahalanobis distance between the estimated position information and the position information of each target can be calculated separately.

[0040] In one possible implementation, the electronic device locally stores historical appearance feature vectors associated with each trajectory. A historical appearance feature vector associated with a particular trajectory is defined as the appearance feature vector of the target corresponding to that trajectory in the most recently successfully detected frame preceding the current frame. For example, if a target is detected in frames 2, 3, and 5, and a trajectory for that target is generated based on the targets in frames 2, 3, and 5, then the historical appearance feature vector associated with that trajectory is the appearance feature vector of that target in frame 5. The electronic device can, for each trajectory, determine the appearance matching cost between that trajectory and each target based on the similarity between the historical appearance feature vector associated with that trajectory and the current appearance feature vector of each detected target.

[0041] In one possible implementation, a motion matching cost matrix can be obtained based on the motion matching cost of each target matching each trajectory. The elements of the motion matching cost matrix consist of the motion matching cost between each target and each trajectory. The rows and columns of the matrix correspond to the set of targets and the set of trajectories detected in the current frame image, respectively. For example, the rows of the matrix correspond to the detected targets (target 1, target 2... target m), and the columns correspond to the trajectories (trajectory 1, trajectory 2... trajectory n). The element in the x-th row and y-th column of the matrix represents the motion matching cost between target x and trajectory y. Similarly, an appearance matching cost matrix can be obtained based on the appearance matching cost of each target matching each trajectory. The elements of the appearance matching cost matrix consist of the appearance matching cost between each target and each trajectory. The rows and columns of the matrix correspond to the set of targets and the set of trajectories detected in the current frame image, respectively. For example, the rows of the matrix correspond to the detected targets (target 1, target 2... target m), and the columns correspond to the trajectories (trajectory 1, trajectory 2... trajectory n). The element in the x-th row and y-th column of the matrix represents the appearance matching cost between target x and trajectory y.

[0042] S103: Based on the motion matching cost and the appearance matching cost, the target of each trajectory matching is determined using the Hungarian algorithm.

[0043] After obtaining the motion matching cost and appearance matching cost of each target and each trajectory, the electronic device can use the Hungarian algorithm to perform matching and determine the target that matches each trajectory.

[0044] In one possible implementation, an association cost matrix can be constructed based on motion matching cost and appearance matching cost. For example, a motion matching cost matrix can be constructed based on the motion matching cost of each target matching each trajectory, and an appearance matching cost matrix can be constructed based on the appearance matching cost of each target matching each trajectory. The association cost matrix is ​​then constructed based on the motion matching cost matrix, the appearance matching cost, and their corresponding weights. For instance, a first matrix can be obtained by multiplying the motion matching cost matrix and its corresponding first weight, and a second matrix can be obtained by multiplying the appearance matching cost matrix and its corresponding second weight. The association cost matrix is ​​then constructed based on the sum of the first and second matrices. Based on the association cost matrix, the Hungarian algorithm is used to determine the targets matched to each trajectory.

[0045] Compared to greedy algorithms that only focus on the current locally optimal pairing, the Hungarian algorithm takes a holistic approach, seeking the globally optimal solution that minimizes the sum of the combined costs of all matched pairs. In scenarios where motion and appearance are combined to fuse the cost matrix, this means the algorithm avoids situations where subsequent key targets cannot be matched due to locally optimal choices, significantly improving the continuity and accuracy of tracking.

[0046] In this embodiment, the trajectory and target are associated by fusing motion matching cost and appearance matching cost. Motion matching cost ensures the trajectory continuity of the target under short-term occlusion or rapid movement, while appearance matching cost utilizes the semantic information of the target to solve the identity confusion problem when targets cross paths. Furthermore, this application employs the Hungarian algorithm to solve for the optimal matching relationship, which minimizes the total matching cost between all trajectories and the detected target. This accurately and effectively determines the matching relationship between the target and the trajectory, improving the accuracy of multi-target tracking.

[0047] To improve tracking accuracy, based on the above embodiments, in this embodiment, predicting the estimated position information of each trajectory in the current frame image based on the pre-saved historical states of each trajectory includes: Based on the pre-saved historical states of each trajectory, the estimated position information and state covariance matrix of each trajectory in the current frame image are predicted; The step of determining the motion matching cost for each target to match each trajectory based on the estimated position information of each trajectory and the position information of each target includes: For each trajectory, based on the Intersection over Union (IoU) ratio between the estimated position information of the trajectory and the position information of each target, a first matching cost is determined for matching the trajectory with each target. A second matching cost, representing the uncertainty of the estimated position information of the trajectory, is determined based on the sum of the variances of each component in the determined state covariance matrix of the trajectory. For each target, a motion matching cost is determined based on the first matching cost and the second matching cost between the trajectory and the target, along with their corresponding weights.

[0048] To perform multi-target tracking accurately and effectively, electronic devices can predict the estimated position information of the target corresponding to the trajectory in the current frame image and the state covariance matrix for each trajectory using a motion model (such as Kalman filtering). The state covariance matrix is ​​used to represent the uncertainty of the estimated position information.

[0049] When determining the motion matching cost, for each trajectory, based on the IoU between the estimated position information of the trajectory and the position information of each detected target, a first matching cost is determined for matching the trajectory with each target. A second matching cost, representing the uncertainty of the estimated position information of the trajectory, is determined based on the sum of the variances of each component in the determined state covariance matrix of the trajectory. For each target, based on the first matching cost and the second matching cost, and their corresponding weights, the motion matching cost between the trajectory and the target is determined. In one possible implementation, the product of the first matching cost and the corresponding third weight can be determined (for ease of distinction, this product can be called the first product), and the product of the second matching cost and the corresponding fourth weight can be determined (for ease of distinction, this product can be called the second product). The motion matching cost between the trajectory and the target is determined based on the sum of the first product and the second product.

[0050] The above-mentioned technical solution has the following advantages or beneficial effects: This application embodiment determines the second matching cost by calculating the sum of the variances of each component in the state covariance matrix, which can quantify the uncertainty of the trajectory estimation position. When the prediction error of a trajectory increases due to prolonged occlusion or violent movement, the covariance matrix determined based on that trajectory will increase accordingly, thereby increasing the matching cost, reducing the probability of the trajectory being incorrectly associated, effectively suppressing identity switching phenomena, and thus improving the accuracy of multi-target tracking.

[0051] To accurately and effectively perform multi-target tracking, based on the above embodiments, in this embodiment of the application, after acquiring the current frame image and before determining the position information of each target in the current frame image through the target detection model, the method further includes: The current frame image is cut into multiple overlapping sub-image slices; The step of determining the position information of each target in the current frame image through a target detection model includes: The target detection model is used to detect targets in each sub-image slice and the current frame image to obtain the first candidate position information of the locally detected target in each sub-image slice and the second candidate position information of the target in the current frame image. Each first candidate position information is mapped to the current frame image to obtain each third candidate position information. The non-maximum suppression algorithm is used to merge each third candidate position information and each second candidate position information to obtain the position information of each target in the current frame image.

[0052] Since target detection models typically require downsampling and compression when processing high-resolution images, this can easily lead to the loss of small target features and thus missed detections. This application proposes a slice-assisted detection strategy. To improve the accuracy of target detection, after acquiring the current frame image, the electronic device can divide the current frame image into multiple sub-image slices with overlapping regions. In one possible implementation, the Slicing-Aided Hyper Inference (SAHI) method can be used to cut the current frame image into multiple sub-image slices with overlapping regions.

[0053] Figure 2 This is a schematic diagram illustrating a process for obtaining multiple sub-image slices, as provided in an embodiment of this application.

[0054] like Figure 2 As shown, six sub-image slices with overlapping regions can be generated.

[0055] Each sub-image slice and the unsegmented current frame image are input into the target detection model. The process of determining the location information through the target detection model includes a dual-path detection mechanism: the target detection model is used to infer the location of each sub-image slice and the original current frame image, respectively, to obtain the first candidate location information of the locally detected target within each sub-image slice, and the second candidate location information of the target as a whole in the current frame image. The electronic device maps each first candidate location information to the coordinate system of the current frame image based on the coordinate offset of each sub-image slice relative to the current frame image, generating third candidate location information. Finally, a candidate set containing all third and second candidate location information is constructed. The overlap and confidence scores between candidate boxes are calculated using a non-maximum suppression algorithm, redundant detection boxes are removed, and duplicate targets are merged to obtain the final location information of each target in the current frame image.

[0056] This application provides a multi-target tracking method based on slice-assisted reasoning, a visual language model, and parallel dual-stream matching conflict decision-making. This method deeply integrates SAHI slice-assisted reasoning with RemoteCLIP cross-modal semantic filtering. In the association stage, it proposes parallel motion and appearance dual-stream matching and conflict decision-making based on motion stability indicators. Combined with a two-stage association strategy based on hierarchical confidence, it constructs a tracking framework with high accuracy, high recall, and high robustness. The greatest advantage of this application is that it enables UAVs to see more clearly and track more accurately in high-altitude aerial photography tasks such as patrols and traffic monitoring. It significantly reduces the "losing" and "misidentification" phenomena caused by targets that are too small, have complex backgrounds, or are temporarily obscured, making the target trajectory in the surveillance video more continuous and the identity more stable. This directly improves the reliability of automated analysis, reduces the frequency of manual review and intervention, and effectively saves manpower and time costs in practical applications such as security and traffic management.

[0057] The above-mentioned technical solution has the following advantages or beneficial effects: The embodiments of this application, through slicing, are equivalent to indirectly increasing the pixel density of the local area without changing the model input resolution, so that small targets that were originally blurry in the whole image compression have more significant features in the slice, thereby significantly reducing the false negative rate.

[0058] To improve the accuracy of target detection, based on the above embodiments, in this embodiment, determining the position information of each target in the current frame image through the target detection model includes: The location information, category, and confidence level of each target in the current frame image are determined by a target detection model; wherein the category includes vehicles, pedestrians, and other categories. After extracting the current appearance feature vectors of each target through the recognition model, and before determining the motion matching cost of each target matching each trajectory, the method further includes: For each target, determine the current appearance feature vector of the target and its similarity with the target feature vector that is saved in advance for the category corresponding to the target. If the determined similarity is lower than the similarity threshold, or the confidence level of the target is lower than the confidence threshold, then the target is determined to be a misidentified target and the target is deleted.

[0059] In this embodiment, after identifying the location information of each target, the target detection model also identifies the category of each target and the confidence level of each target. The categories include vehicles, pedestrians, etc. The confidence level of a target can be the confidence level of the target belonging to a corresponding determined category, or the confidence level of the target's location information belonging to a corresponding determined location.

[0060] After acquiring the targets detected by the target detection model, the electronic device can obtain the target feature vector pre-saved for the category corresponding to each target, determine the similarity between the current appearance feature vector of the target and the target feature vector, and if the determined similarity is lower than a preset similarity threshold, the target can be determined as a misidentified target and deleted. If the confidence level of the target determined by the target detection model is lower than a preset confidence threshold, the target can be determined as a misidentified target and deleted. In one possible implementation, the visual features of the target can be determined using a RemoteSensingContrastiveLanguage-ImagePre-training (RemoteCLIP) model, thereby determining the current appearance feature vector of the target.

[0061] In this embodiment, the visual features of the detected target are compared with predefined category features (such as "car" and "pedestrian"), and cosine similarity is calculated. Based on the set high and low thresholds and category consistency rules, false detection boxes that are semantically inconsistent or have low confidence are automatically removed, and a clean and highly reliable final detection result set is output.

[0062] The above-mentioned technical solution has the following advantages or beneficial effects: In this embodiment, the combination of confidence threshold and appearance feature similarity for joint screening can effectively intercept low-quality detection targets. This embodiment introduces similarity comparison with preset category feature vectors, which is equivalent to adding semantic consistency verification. This can significantly reduce the false detection rate caused by background clutter or similar objects. Furthermore, by deleting misidentified targets before calculating motion matching costs, it reduces erroneous matching attempts between invalid trajectories and noisy targets, thus reducing the computational burden of association algorithms such as the Hungarian algorithm.

[0063] To accurately and effectively perform multi-target tracking, based on the above embodiments, in this embodiment, the step of determining the target matching each trajectory using the Hungarian algorithm based on the motion matching cost and appearance matching cost of each target matching each trajectory includes: Based on the motion matching cost of each target matching each trajectory, the Hungarian algorithm is used to determine the motion target matching each trajectory; based on the appearance matching cost of each target matching each trajectory, the Hungarian algorithm is used to determine the appearance target matching each trajectory. For each trajectory, the target matching the trajectory is determined based on the moving target matching the trajectory and the appearance target matching the trajectory.

[0064] To accurately and effectively track multiple targets, electronic devices can use the Hungarian algorithm to determine the moving targets matched with each trajectory, based on the motion matching cost of each target matching with each trajectory. In one possible implementation, a motion matching cost matrix can be determined based on the motion matching cost of each target matching with each trajectory. Based on this motion matching cost matrix, the Hungarian algorithm is applied to solve for the minimum weight matching of the bipartite graph, thereby finding the optimal allocation scheme that minimizes the total matching cost globally, and finally determining the moving targets matched with each trajectory.

[0065] Electronic devices can determine the appearance targets for each trajectory based on the appearance matching costs of each target matching each trajectory, using the Hungarian algorithm. In one possible implementation, an appearance matching cost matrix can be determined based on the appearance matching costs of each target matching each trajectory. Based on this appearance matching cost matrix, the Hungarian algorithm is applied to solve the minimum weight matching of the bipartite graph, thereby finding the optimal allocation scheme that minimizes the total matching cost globally, and finally determining the appearance targets for each trajectory.

[0066] For each trajectory, the target matching the trajectory is determined based on the moving target and the appearance target matching the trajectory. In one possible implementation, the moving target matching the trajectory, the appearance target matching the trajectory, the historical state associated with the trajectory, the historical appearance feature vector, the location information of the detected moving target, the current appearance feature vector of the moving target, the location information of the detected appearance target, and the current appearance feature vector of the appearance target are input into a large language model to obtain the target matching the trajectory output by the large language model.

[0067] The above-mentioned technical solution has the following advantages or beneficial effects: the reliability of motion information and appearance information varies in different scenarios. Motion matching is more effective when the camera is stable and the target's motion pattern is obvious, while appearance matching is more advantageous when the target reappears after occlusion or when motion is blurred. This application's embodiment, by using the Hungarian algorithm for independent matching, can fully exploit the maximum potential of both the motion model and the appearance model, avoiding the direct interference of noise from one feature with the matching process of another. Even if one modality performs poorly, the other modality can still provide reliable matching results, thereby significantly improving the accuracy of multi-target tracking.

[0068] To accurately and effectively perform multi-target tracking, based on the above embodiments, in this embodiment, determining the target matching the trajectory based on the moving target matching the trajectory and the appearance target matching the trajectory includes: If a moving target matching the trajectory is the same as a visual target matching the trajectory, then the same target is determined to be the target matching the trajectory.

[0069] In this embodiment of the application, if the moving target matching the trajectory and the appearance target matching the trajectory are the same, then the electronic device can identify the same target as the target matching the trajectory.

[0070] To accurately and effectively perform multi-target tracking, based on the above embodiments, in this embodiment, determining the target matching the trajectory based on the moving target matching the trajectory and the appearance target matching the trajectory includes: If the moving target matching the trajectory and the appearance target matching the trajectory are different, and the sum of the variances of each component of the state covariance matrix of the trajectory in the current frame image is greater than a first threshold, then the moving target matching the trajectory is determined as the target matching the trajectory; or, If the moving target matched by the trajectory is different from the appearance target matched by the trajectory, then it is determined whether the IoU between the position information of the moving target matched by the trajectory and the estimated position information of the trajectory is greater than a second threshold. If it is greater than the second threshold, the moving target matched by the trajectory is determined to be the target matched by the trajectory. If it is not greater than the second threshold, the appearance target matched by the trajectory is determined to be the target matched by the trajectory.

[0071] To perform multi-target tracking, in this embodiment, when the moving target and the appearance target matching the trajectory are inconsistent, a conflict resolution strategy can be executed to determine the final matching target. This strategy includes any of the following implementation methods: First implementation method: Decision based on state uncertainty: Calculate the sum of the variances of each component in the state covariance matrix of the trajectory in the current frame image. If the sum of variances is greater than a first threshold, the trajectory is determined to be in a high-dynamic or high-uncertain state, and the moving target matching the trajectory is determined as the final matching target. Second implementation method: Decision based on spatial consistency: Calculate the IoU between the position information of the moving target matching the trajectory and the estimated position information of the trajectory. If the IoU is greater than a second threshold, it indicates that the motion prediction has spatial credibility, and the moving target matching the trajectory is determined as the final matching target. If the IoU is not greater than the second threshold, the appearance target matching the trajectory is determined as the final matching target.

[0072] In another possible implementation, various motion metrics, such as trajectory history stability and motion direction consistency, can be used to determine whether to adopt motion matching or appearance matching. Trajectory history stability can be used to measure whether a trajectory is a "mature and reliable" target, rather than newly appearing or soon-to-disappear noise. For example, the number of times the trajectory has successfully matched a corresponding target in the past K frames can be counted. If this number is greater than a preset threshold, the motion target matching the trajectory is determined as the final matched target; otherwise, the appearance target matching the trajectory is determined as the final matched target. Motion direction consistency can also be used to measure whether the target's current actual movement conforms to its historical movement trend. If the motion direction of the target matching the trajectory is different from that of the trajectory, the appearance target matching the trajectory is determined as the final matched target; otherwise, the motion target matching the trajectory is determined as the final matched target.

[0073] In this embodiment, for each track being tracked, a Kalman filter is used to predict its estimated position information and motion uncertainty in the current frame. Subsequently, two cost matrices are constructed in parallel: a motion cost matrix, calculated based on the intersection-over-union (IoU) ratio between the predicted and detected bounding boxes and the prediction uncertainty; and an appearance cost matrix, calculated based on the cosine distance between the appearance features stored in the track and the appearance features of the detected bounding boxes. The Hungarian algorithm is used to match these two cost matrices, resulting in two sets of solutions: a "motion matching result" and an "appearance matching result." Comparing these two sets of results, intelligent conflict decision-making is performed on contradictory matching pairs: by comprehensively comparing various motion indicators such as the current IoU size, prediction uncertainty, trajectory historical stability, and motion direction consistency, the system decides whether to adopt the motion matching or appearance matching result, ensuring optimal association.

[0074] The above technical solution has the following advantages or beneficial effects: by adopting the method provided in the embodiments of this application, when the trajectory is in a state of high uncertainty, the final target is determined based on the moving target matching the trajectory, or, based on whether the IoU between the position information of the moving target matching the trajectory and the estimated position information of the trajectory is greater than a second threshold, the reliability of the motion prediction is determined, and then the final matching target is determined, thereby improving the accuracy of the matching target determination.

[0075] To improve the accuracy of multi-target tracking, based on the above embodiments, in this embodiment, the historical appearance state vector of any trajectory includes the appearance state sub-vectors of the corresponding tracked target in multiple historical frames; the step of determining the appearance matching cost of each target with each trajectory based on the similarity between the historical appearance feature vectors of each trajectory and the current appearance feature vectors of each target includes: For each trajectory, the similarity between each appearance state sub-vector contained in the trajectory and the current appearance feature vector of each target is determined. Based on multiple similarities and the weights corresponding to the historical frames corresponding to each appearance state sub-vector, the appearance matching cost of each target matching the trajectory is determined. The longer the interval between the historical frame and the current frame image, the smaller the corresponding weight.

[0076] In this embodiment, the historical appearance state vector associated with the trajectory can include appearance state sub-vectors of the corresponding tracked target in multiple historical frames. For example, if a target associated with a certain trajectory is detected in frames 2, 3, 5, and 7, then the historical appearance state vector associated with the trajectory can include appearance state sub-vectors of the target in the most recently detected preset number of frames, such as appearance state sub-vectors of the target in frames 3, 5, and 7. When determining the appearance matching cost of each target matching each trajectory, for each trajectory, the similarity between each appearance state sub-vector contained in the trajectory and the current appearance state vector of each target can be determined. For each target, the appearance matching cost of matching the target with the trajectory is determined based on the similarity between each appearance state sub-vector contained in the trajectory and the current appearance state vector of the target, and the weight corresponding to the historical frame of each appearance state sub-vector. The longer the interval between the historical frame and the current frame image, the smaller the corresponding weight.

[0077] The above-described technical solution has the following advantages or beneficial effects: The embodiments of this application calculate multiple similarities and perform weighted fusion, resulting in an appearance matching cost with higher statistical reliability than a single similarity value. The weighted summation method smooths out abnormal similarity values ​​from individual historical frames, making the final appearance matching cost more accurately reflect the correlation between the target and the trajectory. This helps to more accurately distinguish different targets in dense scenes or when there are many similar targets.

[0078] In this embodiment, the semantically filtered detection boxes are divided into two groups based on their confidence levels. In the first stage, only high-confidence detection boxes are used to perform the parallel matching and conflict decision-making process with all trajectories, prioritizing stable tracking of the main target. In the second stage, the remaining low-confidence detection boxes are used to repeat the matching process with the unmatched trajectories from the first stage. This aims to recover targets temporarily lost due to occlusion or image blurring, and improves the recall rate by relaxing the matching threshold and favoring appearance matching. For successfully matched trajectories, their Kalman filter state is updated using the corresponding detection boxes, and their appearance feature library is smoothly updated. New trajectories are created for unmatched successful detections, while old trajectories that have failed to match for multiple consecutive frames are terminated and removed. The electronic device also includes an anomaly recovery mechanism, expanding the search range for trajectories exhibiting abnormal behavior to attempt recapture.

[0079] Figure 3This is a detailed schematic diagram illustrating a multi-target tracking process provided in an embodiment of this application.

[0080] Depend on Figure 3 It can be seen that the input current frame image can be obtained. Based on SAHI slicing assistance, the current frame image is divided into multiple sub-image slices with overlapping regions. The multiple sub-image slices and the current frame image are input into the YOLOv12 detection network for target detection. Then, the detection results of each slice are mapped back to the original image and merged through non-maximum suppression to obtain candidate target boxes. For the obtained candidate target boxes, the visual features of each candidate box are extracted using the RemoteCLIP visual language model, and cosine similarity is calculated with predefined category text features. Based on the set similarity threshold and category consistency rules, semantically inconsistent or low-confidence candidate boxes are eliminated, and the semantically filtered target detection results are output. The semantically filtered detection results are input into the tracking module. First, a motion cost matrix is ​​constructed based on IoU and Kalman filtering (the position cost is evaluated using the predicted boxes obtained by Kalman prediction and their uncertainty). At the same time, the external features are extracted based on re-Identification (ReID). The system embeds and constructs an appearance cost matrix using feature cosine distance. Then, it applies the Hungarian algorithm in parallel to both cost matrices, generating motion matching and appearance matching result sets respectively. For trajectory-detection pairs that conflict between the motion matching and appearance matching result sets, conflict decisions are made based on motion stability indicators such as current IoU and prediction uncertainty to select either motion matching or appearance matching, thus completing data association. For semantically filtered detection boxes, parallel dual-stream matching and conflict decision-making are performed in two stages according to confidence levels from high to low. In the first stage, motion and appearance matching and conflict decision-making are performed only on high-confidence detection boxes and all active trajectories. In the second stage, motion and appearance matching and conflict decision-making are performed on the remaining low-confidence detection boxes and unmatched trajectories to improve the recall rate of occluded or recurring targets. Trajectories are updated or initialized based on the above association results, and the state and appearance features of targets are dynamically maintained through a Kalman filter and appearance feature library to achieve online multi-target tracking.

[0081] Figure 4a , Figure 4b , Figure 4c , Figure 4d This is a schematic diagram illustrating the result of multi-target tracking provided in an embodiment of this application.

[0082] like Figure 4a , Figure 4b , Figure 4c , Figure 4dAs shown, Frame 1, Frame 10, Frame 40 and Frame 60 contain information about multiple tracked trajectories, including the trajectory ID, such as ID8, and record the category of the target corresponding to the tracked trajectory, such as car.

[0083] Figure 5a , Figure 5b , Figure 5c , Figure 5d This is a schematic diagram illustrating the result of another multi-target tracking method provided in an embodiment of this application.

[0084] like Figure 5a , Figure 5b , Figure 5c , Figure 5d As shown, Frames 1, 6, 17 and 27 contain information about multiple tracked trajectories, including the trajectory ID, such as ID2, and record the category of the target corresponding to the tracked trajectory, such as car or truck.

[0085] In detection-based tracking methods, missed detections by the detector can cause the target to completely "disappear" in subsequent frames, resulting in broken tracks. False detections, on the other hand, can assign tracking IDs to irrelevant objects or even generate false tracks, severely impacting system reliability and tracking accuracy. Therefore, improving the recall and accuracy of the detection module is a core requirement for ensuring the stable operation of multi-target tracking systems. Early target detection methods often relied on sliding windows combined with handcrafted features and classifiers for frame-by-frame searches. While handcrafted features such as Scale-Invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG) offer real-time advantages in simple scenes, they are less adaptable to small targets, multi-scale scenarios, and occluded scenes. With the rise of deep learning, two-stage and one-stage detectors have achieved end-to-end learning through convolutional neural networks. Two-stage detectors include Regions with Convolutional Neural Networks (R-CNN), Fast Regions with Convolutional Neural Networks (Fast R-CNN), and Faster Regions with Convolutional Neural Networks (Faster R-CNN). One-stage detectors include Single Shot MultiBox Detector (SSD), which significantly improves detection performance. However, in the detection of small targets in high-resolution aerial videos, it is still difficult to balance recall and inference speed.

[0086] Existing detection methods still suffer from insufficient recall for small targets in high-altitude aerial photography scenarios. Related technologies divide images into multiple overlapping slices, inferring separately and then merging them, significantly improving the recall rate for small targets while maintaining near real-time performance, but also increasing false positives. The method provided in this application combines a visual language model for cross-modal semantic filtering, effectively suppressing false positives while improving recall through slice-assisted inference. Furthermore, existing multi-target tracking algorithms often fuse motion and appearance features in a serial cascade or simple weighted manner. For example, related technologies propose a method to reduce duplicate counts through multi-frame IOU cascade matching, but lack an active decision-making mechanism for conflicts between parallel motion and appearance matching, and fail to deeply integrate cross-modal semantic verification with the detection-association process, making it difficult to achieve both high recall and low false positives in scenarios with dense small targets, complex backgrounds, and drastic motion changes. This patent performs motion and appearance dual-stream matching simultaneously during the association stage, and uses motion stability index to determine the conflict matching results. Combined with a two-stage association strategy with graded confidence, it effectively solves the problems of low recall, insufficient accuracy and poor robustness in scenarios with dense small targets, strong occlusion and complex backgrounds.

[0087] This application aims to address the shortcomings of existing multi-target tracking technologies in handling missed detections, false detections, and correlation conflicts of small targets in high-resolution aerial photography scenarios. It proposes a multi-target tracking method based on slice-assisted reasoning, a visual language model, and parallel dual-stream matching conflict decision-making. First, the designed SAHI slice-assisted reasoning module divides the original frame into multiple overlapping slices according to a preset size and overlap ratio, detects each slice separately, and then maps them back to the original image for merging. This effectively improves the recall rate and localization accuracy of small targets while reducing the computational overhead and missed detection risk of full-image detection. Second, the constructed RemoteCLIP cross-modal semantic filtering module extracts the visual features of candidate boxes and calculates cosine similarity with predefined category text embeddings. Based on high and low thresholds and category consistency rules, it eliminates semantically inconsistent and low-confidence false detection boxes, effectively suppressing background noise and false detections. Finally, the proposed parallel dual-stream matching and conflict decision module constructs a motion cost matrix based on IoU and Kalman prediction and an appearance cost matrix based on ReID feature cosine distance in the association stage, respectively. The Hungarian algorithm is applied in parallel to generate two matching results. Then, conflict decision is made based on motion stability index. Combined with a two-stage association strategy with hierarchical confidence, it effectively solves problems such as small target loss, high dynamic occlusion and insufficient matching accuracy in complex background scenes.

[0088] Figure 6 This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application, as shown below. Figure 6 As shown: The device includes: The acquisition module 601 is used to acquire the current frame image, determine the position information of each target in the current frame image through the target detection model, and extract the current appearance feature vector of each target through the recognition model; and acquire the currently tracked trajectories. The processing module 602 is configured to: predict the estimated position information of each trajectory in the current frame image based on the pre-saved historical states of each trajectory; determine the motion matching cost of each target matching each trajectory based on the estimated position information of each trajectory and the position information of each target; obtain the historical appearance feature vector associated with each trajectory; determine the appearance matching cost of each target matching each trajectory based on the similarity between the historical appearance feature vector of each trajectory and the current appearance feature vector of each target; and determine the target matching each trajectory using the Hungarian algorithm based on the motion matching cost and the appearance matching cost.

[0089] In one possible implementation, the processing module 602 is specifically used to predict the estimated position information and state covariance matrix of each trajectory in the current frame image based on the pre-saved historical states of each trajectory. The processing module 602 is specifically configured to, for each trajectory, determine a first matching cost between the trajectory and each target based on the IoU between the estimated position information of the trajectory and the position information of each target, and determine a second matching cost representing the uncertainty of the estimated position information of the trajectory based on the sum of the variances of each component in the determined state covariance matrix of the trajectory; and for each target, determine a motion matching cost between the trajectory and the target based on the first matching cost and the second matching cost and the corresponding weights.

[0090] In one possible implementation, the processing module 602 is further configured to cut the current frame image into multiple overlapping sub-image slices; The processing module 602 is specifically used to perform target detection on each sub-image slice and the current frame image through a target detection model, to obtain the first candidate position information of the locally detected target in each sub-image slice and the second candidate position information of the target in the current frame image, to map each first candidate position information to the current frame image to obtain each third candidate position information, and to merge each third candidate position information and each second candidate position information through a non-maximum suppression algorithm to obtain the position information of each target in the current frame image.

[0091] In one possible implementation, the processing module 602 is specifically used to determine the location information of each target in the current frame image, the category of each target, and the confidence level of each target through a target detection model; wherein the category includes vehicles, pedestrians, and other categories; The processing module 602 is further configured to determine the similarity between the current appearance feature vector of the target and the target feature vector that is pre-saved for the category corresponding to the target for each target. If the determined similarity is lower than the similarity threshold, or the confidence level corresponding to the target is lower than the confidence threshold, then the target is determined to be a misidentified target and the target is deleted.

[0092] In one possible implementation, the processing module 602 is specifically configured to: determine the motion target matching each trajectory based on the motion matching cost of each target matching each trajectory using the Hungarian algorithm; determine the appearance target matching each trajectory based on the appearance matching cost of each target matching each trajectory using the Hungarian algorithm; and determine the target matching each trajectory based on the motion target matching the trajectory and the appearance target matching the trajectory.

[0093] In one possible implementation, the processing module 602 is specifically configured to determine that if a moving target matching the trajectory is the same as an appearance target matching the trajectory, then the same target is the target matching the trajectory.

[0094] In one possible implementation, the processing module 602 is specifically configured to: if the moving target matching the trajectory and the appearance target matching the trajectory are different, and the sum of the variances of each component of the trajectory in the state covariance matrix of the current frame image is greater than a first threshold, then determine the moving target matching the trajectory as the target matching the trajectory; or, if the moving target matching the trajectory and the appearance target matching the trajectory are different, determine whether the IoU between the position information of the moving target matching the trajectory and the estimated position information of the trajectory is greater than a second threshold; if it is greater than the second threshold, then determine the moving target matching the trajectory as the target matching the trajectory; if it is not greater than the second threshold, then determine the appearance target matching the trajectory as the target matching the trajectory.

[0095] In one possible implementation, the processing module 602 is specifically used to determine, for each trajectory, the similarity between each appearance state sub-vector contained in the trajectory and the current appearance feature vector of each target, and to determine the appearance matching cost of each target matching the trajectory based on multiple similarities and the weight corresponding to the historical frame corresponding to each appearance state sub-vector; wherein, the longer the interval between the historical frame and the current frame image, the smaller the corresponding weight.

[0096] Figure 7 This application provides a schematic diagram of an electronic device structure based on an embodiment of the present application. In addition to the above embodiments, this application also provides an electronic device, such as... Figure 7 As shown, it includes: processor 701, communication interface 702, memory 703 and communication bus 704, wherein processor 701, communication interface 702 and memory 703 communicate with each other through communication bus 704. The memory 703 stores a computer program, which, when executed by the processor 701, causes the processor 701 to perform any of the above method steps.

[0097] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0098] The communication interface 702 is used for communication between the above-mentioned electronic device and other devices.

[0099] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0100] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0101] This application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform any of the above method steps.

[0102] This application provides a computer program product, which includes an executable program that, when executed by a processor, implements the method described herein.

[0103] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0104] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A multi-target tracking method, characterized in that, The method includes: The current frame image is acquired, the position information of each target in the current frame image is determined by the target detection model, and the current appearance feature vector of each target is extracted by the recognition model; the current tracked trajectories are acquired. Based on the pre-saved historical states of each trajectory, the estimated position information of each trajectory in the current frame image is predicted; based on the estimated position information of each trajectory and the position information of each target, the motion matching cost of each target matching each trajectory is determined; the historical appearance feature vector associated with each trajectory is obtained, and based on the similarity between the historical appearance feature vector of each trajectory and the current appearance feature vector of each target, the appearance matching cost of each target matching each trajectory is determined. Based on the motion matching cost and the appearance matching cost, the target of each trajectory matching is determined using the Hungarian algorithm.

2. The method according to claim 1, characterized in that, The step of predicting the estimated position information of each trajectory in the current frame image based on the pre-saved historical states of each trajectory includes: Based on the pre-saved historical states of each trajectory, the estimated position information and state covariance matrix of each trajectory in the current frame image are predicted; The step of determining the motion matching cost for each target to match each trajectory based on the estimated position information of each trajectory and the position information of each target includes: For each trajectory, based on the intersection-union ratio (IoU) between the estimated position information of the trajectory and the position information of each target, a first matching cost is determined for matching the trajectory with each target. Based on the sum of the variances of each component in the determined state covariance matrix of the trajectory, a second matching cost representing the uncertainty of the estimated position information of the trajectory is determined. For each target, based on the first matching cost and the second matching cost between the trajectory and the target, and their corresponding weights, a motion matching cost between the trajectory and the target is determined.

3. The method according to claim 1, characterized in that, After acquiring the current frame image and before determining the position information of each target in the current frame image using the target detection model, the method further includes: The current frame image is cut into multiple overlapping sub-image slices; The step of determining the position information of each target in the current frame image through a target detection model includes: The target detection model is used to detect targets in each sub-image slice and the current frame image to obtain the first candidate position information of the locally detected target in each sub-image slice and the second candidate position information of the target in the current frame image. Each first candidate position information is mapped to the current frame image to obtain each third candidate position information. The non-maximum suppression algorithm is used to merge each third candidate position information and each second candidate position information to obtain the position information of each target in the current frame image.

4. The method according to claim 1, characterized in that, The step of determining the position information of each target in the current frame image through a target detection model includes: The location information, category, and confidence level of each target in the current frame image are determined by a target detection model; wherein the category includes vehicles, pedestrians, and other categories. After extracting the current appearance feature vectors of each target through the recognition model, and before determining the motion matching cost of each target matching each trajectory, the method further includes: For each target, determine the current appearance feature vector of the target and its similarity with the target feature vector that is saved in advance for the category corresponding to the target. If the determined similarity is lower than the similarity threshold, or the confidence level of the target is lower than the confidence threshold, then the target is determined to be a misidentified target and the target is deleted.

5. The method according to claim 1, characterized in that, The step of determining the target for each trajectory using the Hungarian algorithm based on the motion matching cost and appearance matching cost of each target matching each trajectory includes: Based on the motion matching cost of each target matching each trajectory, the Hungarian algorithm is used to determine the motion target matching each trajectory; based on the appearance matching cost of each target matching each trajectory, the Hungarian algorithm is used to determine the appearance target matching each trajectory. For each trajectory, the target matching the trajectory is determined based on the moving target matching the trajectory and the appearance target matching the trajectory.

6. The method according to claim 5, characterized in that, The step of determining the target matching the trajectory based on the moving target matching the trajectory and the appearance target matching the trajectory includes: If a moving target matching the trajectory is the same as a visual target matching the trajectory, then the same target is determined to be the target matching the trajectory.

7. The method according to claim 5, characterized in that, The step of determining the target matching the trajectory based on the moving target matching the trajectory and the appearance target matching the trajectory includes: If the moving target matching the trajectory and the appearance target matching the trajectory are different, and the sum of the variances of each component of the state covariance matrix of the trajectory in the current frame image is greater than a first threshold, then the moving target matching the trajectory is determined as the target matching the trajectory; or, If the moving target matched by the trajectory is different from the appearance target matched by the trajectory, then it is determined whether the IoU between the position information of the moving target matched by the trajectory and the estimated position information of the trajectory is greater than a second threshold. If it is greater than the second threshold, the moving target matched by the trajectory is determined to be the target matched by the trajectory. If it is not greater than the second threshold, the appearance target matched by the trajectory is determined to be the target matched by the trajectory.

8. The method according to claim 1, characterized in that, The historical appearance state vector of any trajectory contains the appearance state sub-vectors of the corresponding tracked target in multiple historical frames; the determination of the appearance matching cost for each target to match each trajectory based on the similarity between the historical appearance feature vectors of each trajectory and the current appearance feature vectors of each target includes: For each trajectory, the similarity between each appearance state sub-vector contained in the trajectory and the current appearance feature vector of each target is determined. Based on multiple similarities and the weights corresponding to the historical frames corresponding to each appearance state sub-vector, the appearance matching cost of each target matching the trajectory is determined. The longer the interval between the historical frame and the current frame image, the smaller the corresponding weight.

9. A multi-target tracking device, characterized in that, The device includes: The acquisition module is used to acquire the current frame image, determine the position information of each target in the current frame image through the target detection model, and extract the current appearance feature vector of each target through the recognition model; and acquire the currently tracked trajectories. The processing module is configured to: predict the estimated position information of each trajectory in the current frame image based on the pre-saved historical states of each trajectory; determine the motion matching cost of each target matching each trajectory based on the estimated position information of each trajectory and the position information of each target; obtain the historical appearance feature vector associated with each trajectory; determine the appearance matching cost of each target matching each trajectory based on the similarity between the historical appearance feature vector of each trajectory and the current appearance feature vector of each target; and determine the target matching each trajectory using the Hungarian algorithm based on the motion matching cost and the appearance matching cost.

10. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the steps of the multi-target tracking method as described in any one of claims 1-8.

Images