Same-view-field cross-lens real-time target tracking system and method based on visual matching

Abstract

The invention provides a same-view-field cross-lens real-time target tracking system and method based on visual matching, and relates to the technical field of multi-target tracking, and the system comprises an acquisition module, a target detection module, a feature extraction module, a trajectory tracking module, a database management module and a cross-lens matching module. Wherein the trajectory tracking module can track a detected target in combination with target feature information, generate an identity label and output trajectory information, so as to ensure identity continuity and trajectory integrity of the target under the same lens; the database management module is used for uniformly storing the identity label, the feature information and the track information of the target and providing reliable data support for cross-lens tracking; and the cross-lens matching module is used for matching targets under different lenses based on the information in the database management module so as to realize cross-lens association of multiple lenses in the same view field. Through cooperation of multiple modules, multi-lens cross-lens real-time tracking under the same field of view can be effectively realized, and the accuracy and real-time performance of target matching are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-target tracking, in particular to a same-viewfield cross-lens real-time target tracking system and method based on visual matching. BACKGROUND

[0002] Intelligent monitoring camera systems have been widely used in smart cities, intelligent security, transportation and other fields. Cross-lens multi-target tracking technology aims to realize continuous real-time identification and tracking of multiple targets under different camera angles. This technology can realize seamless tracking of targets in complex scenes and has important practical significance.

[0003] Cross-lens multi-target tracking in the same viewfield puts higher requirements on the real-time matching. This is because multiple cameras need to capture targets in the same scene within the same time period and determine in real time whether they are the same object in different angles. If there is a delay or matching error, problems such as target ID switching, track interruption or repeated counting may occur. At the same time, the background of the same scene is highly similar, and the scale, angle, lighting and occlusion of targets in different angles are different, making it more challenging to establish a multi-angle correspondence relationship in real time and perform information fusion.

[0004] The prior art discloses various cross-lens target tracking methods. One of them is a method based on trajectory information matching. This method generates local trajectories in a single camera and matches trajectory segments across cameras to obtain complete target trajectories. However, this method is almost unable to maintain tracking when the target is occluded. Another method is based on three-dimensional space reconstruction. This method realizes accurate mapping and tracking of pedestrian trajectories by combining foreground segmentation with three-dimensional model projection. However, it has high computational complexity and consumes a lot of computing power, making it difficult to apply to edge devices. Another method is based on homography matrix. This method determines the relative position relationship between multiple angles and calculates the distance between matching points to realize target association. However, it is limited to the need to obtain the relative positions of all cameras in advance, which lacks applicability and flexibility in actual monitoring environments. In addition, the re-identification (ReID) method is also a common technology. This method relies on target appearance features for identity registration and matching, and has strong applicability in monitoring scenarios. However, in cross-lens tracking tasks, its dependence on single-frame registration information leads to a decrease in matching accuracy when the quality of the registration frame is poor, thereby affecting the real-time and accuracy of cross-lens matching.

[0005] Therefore, how to ensure the accuracy of cross-lens target matching while considering real-time is a technical problem that needs to be solved. SUMMARY

[0006] The application provides a same-view cross-lens real-time target tracking system and method based on visual matching, aiming at solving the problems of poor real-time performance and low matching accuracy in the current cross-lens multi-target tracking in the same view.

[0007] In order to achieve the above technical effects, the technical solutions of the application are as follows: A same-view cross-lens real-time target tracking system based on visual matching comprises: An acquisition module is configured to acquire real-time video data of multiple video sampling lenses in the same view. A target detection module is configured to perform target detection on the real-time video data to obtain a detection frame containing a detection target. A feature extraction module is configured to perform feature extraction on the detection frame to obtain feature information and store the feature information in a preset database management module. A trajectory tracking module is configured to perform target tracking on the detection target according to the feature information, integrate the identity of the detection target, output trajectory information, and store the identity and the trajectory information in the preset database management module. A database management module is configured to store the identity, feature information and trajectory information of the detection target. A cross-lens matching module is configured to perform cross-lens matching on the detection target according to the identity, feature information and trajectory information stored in the database management module to obtain a cross-lens matching result.

[0008] Preferably, the feature extraction on the detection frame to obtain the feature information and the storage of the feature information in the preset database management module comprise the following steps: The detection target is input into a preset feature extractor, and the feature information of the detection target in the detection frame is extracted by the feature extractor. The cosine similarity distance between the feature information of the detection target and the feature information stored in the database management module is calculated. It is judged whether the cosine similarity distance is higher than a preset addition threshold, if yes, the feature information of the detection target is stored in the database management module, and if no, the feature information of the detection target is discarded.

[0009] Preferably, the cosine similarity distance is calculated according to the following formula:

[0010] Wherein, A is the feature information of the detection target in the detection frame, and B is the feature information stored in the database management module.

[0011] Preferably, the target tracking of the detection target according to the feature information, the integration of the identity of the detection target, and the output of the track information comprise the following steps: predicting the detection target to obtain a prediction box of the detection target; matching the prediction box with the detection box to obtain a matching result; integrating the identity of the detection target according to the matching result and outputting track information containing the position of the detection box of the matching success.

[0012] Preferably, the detection box matched with the prediction box comprises a high-confidence detection box with a confidence exceeding an upper threshold and a low-confidence detection box with a confidence lower than a lower threshold.

[0013] Preferably, the matching of the prediction box with the detection box comprises: first matching: respectively calculating the intersection-over-union distance and the feature cosine distance between the prediction box and the high-confidence detection box, taking the smaller one of the intersection-over-union distance and the feature cosine distance, if the smaller one is the intersection-over-union distance, judging whether the intersection-over-union distance is greater than a first preset distance threshold, if yes, determining that the first matching is successful, if no, determining that the first matching fails; if the smaller one is the feature cosine distance, judging whether the feature cosine distance is less than a second preset distance threshold, if yes, determining that the first matching is successful, if no, determining that the first matching fails.

[0014] second matching: calculating a second intersection-over-union distance between the prediction box matched unsuccessfully in the first matching and the low-confidence detection box, judging whether the second intersection-over-union distance is greater than a third preset distance threshold, if yes, determining that the second matching is successful, if no, determining that the second matching fails; third matching: calculating a third intersection-over-union distance between the prediction track box matched unsuccessfully in the first and second matchings and the high-confidence detection box matched unsuccessfully in the first matching, judging whether the third intersection-over-union distance is greater than a fourth preset distance threshold, if yes, determining that the third matching is successful, if no, determining that the third matching fails.

[0015] Preferably, the integration of the identity of the detection target according to the matching result and the output of the track information containing the position of the detection box of the matching success comprise: if any one of the three matchings is successful, outputting the track information containing the position of the detection box of the matching success, marking the detection target in the detection box of the matching success as the same identity, and storing the identity, the feature information of the detection target, and the track information into a preset database management module.

[0016] Preferably, the cross-lens matching of the detection target according to the identity, feature information and trajectory information stored in the database management module comprises: obtaining the detection frame of each lens current frame and the feature information of the detection target in the detection frame; matching the feature information of the detection target with the feature information stored in the database management module; if the matching is successful, the same identity is uniformly assigned to the detection target with the matching success under multiple perspectives, and the feature information and trajectory information stored in the database management module are updated; if the matching fails, the next step is performed; re-matching the trajectory information of the detection target with the matching failure with the trajectory information stored in the database management module; if the re-matching is successful, the same identity is uniformly assigned to the detection target with the re-matching success under multiple perspectives, and the feature information and trajectory information stored in the database management module are updated; if the re-matching fails, a new identity is assigned to the detection target with the re-matching failure, and new trajectory information is established.

[0017] Preferably, the matching of the feature information of the detection target with the feature information stored in the database management module comprises: calculating the cosine similarity distance of the feature information of the detection target and the feature information stored in the database management module; judging whether the cosine similarity distance is greater than a preset update threshold; if yes, the matching is successful; if no, the matching fails.

[0018] The application also provides a same-view-field cross-lens real-time target tracking method based on visual matching, which comprises the following steps: obtaining real-time video data of multiple video sampling lenses in the same view field; performing target detection on the real-time video data to obtain a detection frame containing a detection target; performing feature extraction on the detection frame to obtain feature information; performing target tracking on the detection target according to the feature information, integrating the identity of the detection target, outputting trajectory information, and storing the identity and the trajectory information in a preset database management module; storing the identity, feature information and trajectory information of the detection target by using the database management module; performing cross-lens matching of the detection target according to the identity, feature information and trajectory information stored in the database management module to obtain a cross-lens matching result.

[0019] Compared with the prior art, the technical scheme of the application has the following beneficial effects: The application provides a same-view cross-lens real-time target tracking system and method based on visual matching, which comprises an acquisition module, a target detection module, a feature extraction module, a trajectory tracking module, a database management module and a cross-lens matching module; the acquisition module can simultaneously collect real-time video data of multiple lenses in the same view, thereby providing a data basis for multi-target detection and tracking; the target detection module detects the real-time video data to obtain a detection frame, thereby ensuring that multiple targets in the picture can be accurately identified; the feature extraction module extracts features of the detected targets and stores them in the database management module, so that the targets have distinguishable feature expressions; the trajectory tracking module tracks the targets in combination with target feature information and generates identity labels and trajectory information, thereby ensuring the identity continuity and trajectory integrity of the targets in the same lens; the database management module uniformly stores the identity labels, feature information and trajectory information of the targets, thereby providing reliable data support for cross-lens target matching; and the cross-lens matching module matches the targets between multiple lenses based on the information in the database, thereby realizing cross-lens tracking of multiple lenses in the same view and effectively improving the real-time performance and accuracy of cross-lens target tracking. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A structure diagram of the same-view cross-lens real-time target tracking system based on visual matching in the embodiment of the application; Figure 2 A framework diagram of the same-view cross-lens real-time target tracking system based on visual matching in the embodiment of the application; Figure 3 An indoor scene schematic diagram in the embodiment of the application; Figure 4 A FastReID knowledge distillation module schematic diagram in the embodiment of the application; Figure 5 A Bytetrack combined with ReID feature tracking target flowchart in the embodiment of the application; Figure 6 A database management module schematic diagram in the embodiment of the application; Figure 7 A cross-lens ID matching flowchart in the embodiment of the application; Figure 8 A same-view cross-lens real-time target tracking method flowchart based on visual matching in the embodiment of the application. DETAILED DESCRIPTION

[0021] The accompanying drawings are only used for illustrative description and cannot be understood as a limitation of the patent; For those skilled in the art, it is understandable that some well-known descriptions in the drawings can be omitted; The position relationship described in the drawings is only used for illustrative description and cannot be understood as a limitation of the patent; In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments.

[0022] Embodiment 1 The present embodiment provides a visual matching-based same-view cross-lens real-time target tracking system, as shown in Figure 1 The system comprises: An acquisition module for acquiring real-time video data of multiple lenses in the same view; A target detection module for target detection on the real-time video data to obtain a detection frame containing a detected target; In the present embodiment, the target detection module is a detector, which uses YOLOv8s for pedestrian target detection, and combines an efficient multi-scale attention EMA module to enhance the detection capability of YOLOv8s in dealing with large pedestrian flow scenarios (such as parks and squares).

[0023] A feature extraction module for feature extraction on the detected target to obtain feature information and store the feature information in a preset database management module, comprising the following steps: The detected target is input into a preset feature extractor, which extracts the feature information of the detected target in the detection frame; The cosine similarity distance of the feature information of the detected target and the single-lens feature information stored in the database management module is calculated; it is judged whether the cosine similarity distance is higher than a preset addition threshold, if yes, the feature information of the detected target is stored in the database management module; if not, the feature information of the detected target is discarded.

[0024] The cosine similarity distance is calculated according to the following formula:

[0025] In the formula, A is the feature information of the detected target in the detection frame, and B is the single-lens feature information stored in the database management module.

[0026] In the present embodiment, the feature extractor is executed by a pedestrian re-identification algorithm. The system feature extraction and re-identification algorithm use the knowledge distillation module of FastReID. Considering that complex deep models have relatively slow inference speed and are not easy to deploy in edge computing hardware and artificial intelligence chips, the knowledge distillation module of the lightweight model FastReID is selected, which can realize higher precision and efficiency in the re-identification task.

[0027] Specifically, the detector will send the recognized target into the trained feature extractor, FastReID encodes the target into a ReID feature vector, and the target features under different perspectives are independently extracted and stored in the feature table of the database as ReID feature vectors, which are used to guide the tracker to track multiple targets and match multiple targets across cameras. Among them, the target features extracted by a single lens will be pre-cleaned and integrated using cosine similarity distance, and the feature vectors higher than the set threshold will be attributed to the same ID; otherwise, those that do not exceed the set threshold will be excluded, so as to ensure that all target features in the database feature table are of value.

[0028] a trajectory tracking module, configured to track the detection target according to the feature information, integrate the identity of the detection target, output trajectory information, and store the identity and the trajectory information in a preset database management module; The tracking of the detection target according to the feature information, the integration of the identity of the detection target, and the output of the trajectory information include the following steps: predicting the detection target to obtain a prediction box of the detection target; matching the prediction box with the detection box to obtain a matching result; integrating the identity of the detection target according to the matching result and outputting trajectory information containing the position of the detection box with a successful match.

[0029] The detection box matched with the prediction box includes a high-confidence detection box with a confidence exceeding an upper threshold and a low-confidence detection box with a confidence lower than a lower threshold.

[0030] The matching of the prediction box with the detection box includes: First matching: the intersection-over-union distance and the feature cosine distance between the prediction box and the high-confidence detection box are calculated respectively, the smaller one of the intersection-over-union distance and the feature cosine distance is taken, if the smaller one is the intersection-over-union distance, it is judged whether the intersection-over-union distance is greater than a first preset distance threshold, if yes, it is determined that the first matching is successful, if no, it is determined that the first matching fails; if the smaller one is the feature cosine distance, it is judged whether the feature cosine distance is less than a second preset distance threshold, if yes, it is determined that the first matching is successful, if no, it is determined that the first matching fails.

[0031] Second matching: a second intersection-over-union distance between the prediction box that fails in the first matching and the low-confidence detection box is calculated, it is judged whether the second intersection-over-union distance is greater than a third preset distance threshold, if yes, it is determined that the second matching is successful; if no, it is determined that the second matching fails. Third matching: a third intersection-over-union distance between the predicted trajectory box that fails in the first two matchings and the high-confidence detection box that fails in the first matching is calculated, and it is determined whether the third intersection-over-union distance is greater than a fourth preset distance threshold, if yes, it is determined that the third matching is successful; if no, it is determined that the third matching fails.

[0032] According to the matching result, the identity of the detection target is integrated, and the trajectory information containing the position of the matching successful detection box is output, including: If any one of the three matchings is successful, the trajectory information containing the position of the matching successful detection box is output, the detection target in the matching successful detection box is marked as the same identity, and the identity, the feature information and the trajectory information of the detection target are stored in the preset database management module.

[0033] In this embodiment, the trajectory tracking module is a tracker, the tracker combines the feature information extracted by FastReID into the ByteTrack tracking algorithm, and through the assistance of the feature information, the problem of matching failure of the high-confidence box caused by long-time occlusion of the target and missing of the feature information in some scenes is solved. At the same time, the frequency of target ID jump in the tracking process is reduced, and the robustness of the tracker is enhanced.

[0034] The specific working process of the tracker is that the high-confidence detection box output by the detector in the last step is paired with the predicted box generated by the Kalman filter, the intersection-over-union distance between the two boxes is calculated as the basis for matching, and the Hungarian algorithm is applied to complete the matching. The matching successful one will update the position and motion state of the target, and keep the ID of the current target.

[0035] The database management module is used for storing the identity, feature information and trajectory information of the detection target. Among them, for the relatively mature system architecture of cross-lens multi-target tracking, it only pays attention to the current frame information in the matching stage, and the past historical target information will not be reserved, so the DM (database management) module is added in the traditional architecture in this embodiment. It can reserve multi-frame historical information of each view angle, including target features, trajectories, target surrounding backgrounds, IDs and other information. By using the reserved target related information other than the current frame, the information can be used as a matching basis in the matching stage, and the information interaction of the same target in different views is enhanced.

[0036] The cross-lens matching module is used for cross-lens matching of the detection target according to the identity, feature information and trajectory information stored in the database management module, to obtain a cross-lens matching result, including: Obtaining the detection box of the current frame of each lens and the feature information of the detection target in the detection box; The feature information of the detected target is matched with the single-lens feature information stored in the database management module; if the match is successful, the same identity identifier is uniformly assigned to the detected target under multiple views, and the feature information and trajectory information stored in the database management module are updated; if the match fails, proceed to the next step. The trajectory information of the detected targets that failed to match is matched again with the trajectory information stored in the database management module. If the match is successful again, the same identity identifier is uniformly assigned to the detected targets that were matched again from multiple perspectives, and the feature information and trajectory information stored in the database management module are updated. If the match fails again, a new identity identifier is assigned to the detected targets that failed to match again, and new trajectory information is established.

[0037] The matching of the feature information of the detected target with the feature information stored in the database management module includes: Calculate the cosine similarity distance between the feature information of the detected target and the feature information stored in the database management module; Determine whether the cosine similarity distance is greater than a preset update threshold. If yes, the match is successful; otherwise, the match fails.

[0038] The core task of cross-camera target real-time matching is to associate the same target ID in different viewpoints and identify whether a new target appears in the current frame. In a global view, cross-camera target matching is essentially tracking the same target in different viewpoints, and it has trajectory information; however, when a new target appears in a certain viewpoint, because it is a completely new individual, it is necessary to establish a new identity and trajectory information.

[0039] To address the challenge of cross-camera multi-target matching, this embodiment proposes a visual matching method combining Track-ReID. It integrates FastReID with trajectory information and utilizes target information stored in the DM module to match the ReID information of targets in the current frame with feature information stored in the database for each single camera. Targets that do not match are re-matched by combining historical trajectory information from each viewpoint. Successfully matched targets are assigned the same ID across multiple viewpoints. Target detection boxes that fail to match are treated as newly appearing targets, and new trajectories are created based on the detection box information and updated in the database, while their IDs are updated incrementally. ID matching occurs in every real-time frame, while the data in the DM module is updated every 10 frames. This ensures that the system achieves cross-camera target ID association matching in the vast majority of scenarios.

[0040] Example 2 This embodiment uses an indoor scene as an example to illustrate a real-time target tracking system for cross-camera shots within the same field of view based on visual matching. The system framework is shown below. Figure 2 .

[0041] Acquisition Module: Two cameras, A and B, with opposing viewpoints are deployed in an indoor scene. Both cameras have a resolution of 1920×1080, a viewing height of approximately 3 meters above the ground, and an angle between the camera and the ground between 30° and 45°. The scene is relatively simple, as illustrated in the diagram. Figure 3 The test targets must include at least four people in the two monitoring perspectives within the same field of view. During specific time periods, indoor scenes may exhibit dense pedestrian traffic and interactions between different pedestrians causing occlusion, effectively testing the system's real-time tracking and correlation performance in scenarios with high target density. After data collection, low-quality video clips, such as blurry shots, are filtered out. The dataset is constructed according to the structure and standards of the multi-target tracking public dataset MOT17. Furthermore, based on the requirements of cross-camera multi-target tracking systems, calibration ensures temporal consistency of the data acquired from both perspectives. Specifically, each pair of video sequences must have the same start and end times, and the ID of the same target in different perspectives must remain consistent and unique within the dataset.

[0042] Boundary boxes were used to annotate the visible regions of targets in the video sequence using VATIC, in the format (xtl, ytl, xbr, ybr), where (xtl, ytl) and (xbr, ybr) are the two diagonal points of the annotated box. Frame-by-frame validation was performed using LabelMe to ensure the accuracy of the annotations and that the bounding boxes are the smallest boxes that can enclose the targets. To meet the training requirements of this system, the complete dataset was converted to COCO format.

[0043] Object detection module: YOLOv8s is used as the detection model. First, the 640×640 input size of the detection model needs to be modified to fit the 1088×608 input size used by the subsequent tracker. Simultaneously, an efficient multi-scale attention (EMA) module is introduced into its backbone network C2f to enhance the model's detection capability in scenarios with significant object movement (such as parks and squares). Retraining on the dataset for the pedestrian category further improves the detection accuracy of pedestrian targets. The results can also serve as the basis for the subsequent feature extractor and tracking module. Training is performed for 100 iterations on the dataset with a batch size of 4, using stochastic gradient descent as the optimizer, and setting the initial learning rate to 2×10⁻⁶. -3 The momentum is 0.9, and the weight decay is 5 × 10. -4 EIoU is used as the loss function for model training, and the calculation formula is as follows:

[0044] In the formula, and These are the center points of the predicted bounding box and the ground truth bounding box, respectively; for and The Euclidean distance between them; The diagonal distance between the minimum bounding rectangles of the predicted bounding box and the ground truth bounding box; , , , These are the width and length of the predicted bounding box and the ground truth bounding box, respectively; , The width and length of the minimum bounding rectangle that covers the predicted bounding box and the ground truth bounding box.

[0045] Feature Extraction Module: Pedestrian re-identification serves as the feature extraction module for cross-camera multi-target tracking, enhancing the tracker's target tracking performance and cross-camera multi-target association matching. In this embodiment, the re-identification algorithm used is FastReID. Since complex, deep models require time-consuming inference and are not easily deployed on edge computing hardware and AI chips, the knowledge distillation module of the lightweight FastReID model is used to achieve higher accuracy and efficiency in the re-identification task.

[0046] The training process of the FastReID knowledge distillation module is as follows: Figure 4 As shown, a pre-trained student model and a teacher model are first applied to the ReID dataset, where the student model is relatively simple, and the teacher model is more advanced. The student model and the teacher model each output a classifier, logit. , and characteristics , The training logic aims to enable the student model to learn the classification ability of the teacher model as much as possible. A probabilistic knowledge transfer model based on Kullback-Leibler (KL) divergence is used to optimize the student model to ensure consistency in the feature space distribution between the two models. After training, the features output by the student model are... Used for subsequent distance measurement and association matching.

[0047] In this system, the ReID feature vectors extracted by the feature extractor are saved in the database and assigned different IDs. These are used in two places: first, to calculate the appearance similarity between trajectories to guide the Hungarian algorithm in the tracker for trajectory matching; and second, in cross-camera association matching, to perform target association matching from different viewpoints based on the similarity of pedestrian feature vectors. The similarity is uniformly measured using cosine distance, and its formula is...

[0048] Wherein, A is the feature information of the detected target in the detection frame, and B is the single-lens feature information stored in the database management module; Let be the feature vector of the target being detected within the detection box. This refers to the single-lens feature vector stored in the database management module.

[0049] To avoid excessively increasing the computational burden on the system and affecting its operating speed, ReID feature extraction will be set before the tracking stage. That is, the ReID features of the target pedestrian in the image will be extracted before the tracker tracks the target. This allows the tracker to effectively utilize ReID feature information to improve tracking performance without increasing the computational burden too much.

[0050] Tracking Module: In the first ByteTrack match, when facing complex scenes with large areas and prolonged occlusion, the lack of target feature information may cause tracking failures for some high-confidence bounding boxes. To address this issue, this design integrates FastReID into the first ByteTrack match. By combining the feature information extracted by FastReID with the assistance of ReID information, in addition to solving the aforementioned problem, it also reduces the frequency of target ID jumps during tracking, enhancing the tracker's robustness. Details are as follows: First, in the initial Bytetrack matching, for high-confidence detection boxes, the IoU distance between them and the predicted boxes is calculated. Simultaneously, the ReID feature cosine distance is calculated, and the smaller of these distances guides the Hungarian algorithm to complete the matching. The formula is as follows:

[0051] In the formula, For the first The first goal and the first The final distance of each trajectory is the smaller of the IoU distance and the ReID cosine distance. When the cosine distance is less than the threshold and the IoU distance is greater than the threshold, the ReID feature is used as the matching basis.

[0052] like Figure 5 As shown, after the detector completes target detection in the current frame, all generated bounding boxes are divided into high-confidence boxes and low-confidence boxes according to the set thresholds. In this design, the high-confidence and low-confidence thresholds for tracking are set to 0.6 and 0.1, respectively, and the IoU thresholds for matching are set to 0.7 and 0.1, respectively. The Kalman filter predicts the next frame bounding box for each tracking trajectory. The Kalman filter consists of two stages: prediction and update. The prediction formula is as follows:

[0053]

[0054] in, Let be the target's current state vector. Here is the state transition matrix. Let be the state vector from the previous time step. To control the input matrix, To control the input vector, The noise covariance matrix generated by the process. Let be the state covariance matrix at the current moment.

[0055] In the update phase, the transformation matrix is ​​first used. The next actual state observation obtained from the detector Transform to a linear space, i.e. , The observation noise is the basis for the state update formula, which is as follows:

[0056]

[0057]

[0058] It is the Kalman filter gain matrix. When the gain is increased, the filter tends to focus on new measurements; conversely, when the gain is decreased, it relies more on historical information to avoid noise interference.

[0059] The second ByteTrack matching primarily targets low-confidence boxes, calculating the IoU between these low-confidence boxes and the predicted boxes that failed to match in the previous step. The results are: matched low-confidence boxes, unmatched low-confidence boxes, and unmatched predicted trajectory boxes. The third matching uses inactive trajectories to match the high-confidence detection boxes that failed to match in the first step. If a match is successful, the trajectory is activated; otherwise, a new target tracking trajectory is created, with the target ID incremented. This process is repeated when the next frame arrives.

[0060] Database Management Module: In relatively mature system architectures for cross-camera multi-target tracking, the matching phase only focuses on the current frame information and does not retain historical target information. This leads to the system's over-reliance on single-frame information; if the target information in the current frame is blurry, it will affect the matching effect. This patent introduces a DM (Database Management) module into the traditional architecture, such as... Figure 6 As shown, it can retain multi-frame historical information from various perspectives. By utilizing target information retained outside the current frame, it uses this information as the matching basis during the matching stage, enhancing the information interaction of the same target in different perspectives, and updating the data information table sequentially at certain frame intervals. Specifically, it can be divided into: PersonID table, Track information table, Feature table, and Match table.

[0061] Cross-camera matching module: The task of cross-camera multi-target matching is to combine target information from the current frame and historical frames to establish image associations between different shooting perspectives, and to build a model that fuses multi-view information to improve target tracking performance. This embodiment, for indoor scenes with a limited number of targets, adopts a visual matching method that combines Track-ReID with tracking trajectory information. By calling the target information stored in the DM module, and using multi-frame visual information as the matching basis, targets under different shooting perspectives are associated. At the same time, by combining historical trajectory information from multiple perspectives, new targets appearing under different perspectives, unmatched targets, and incorrectly matched targets are effectively processed.

[0062] like Figure 7 As shown, firstly, identical targets from different viewpoints are associated and matched using the FastReID algorithm. For target detection boxes in the current frame that are not associated with an ID, the cosine distance is calculated between the ReID features and the target feature information in the current database feature table. A set cosine distance threshold is used as the similarity metric. Successfully matched targets are associated with existing IDs. Targets that fail to match are matched against existing trajectory information in the database trajectory information table using their trajectory information from multiple viewpoints in the current frame. If the match still fails, the target is considered a newly appearing target, a new trajectory is created using the detection box information, the target ID is incremented, and the database information is updated. By combining re-identification and trajectory tracking, this system can stably complete cross-camera target matching tasks in most scenarios.

[0063] Since the camera lens detects and identifies targets in real time, the target information extracted in a certain frame may be insufficient but still included in the database. This can lead to incorrect matching, such as different targets being matched with the same ID in two different perspectives. To correct this situation in a timely manner, the system's matching mechanism will rematch the targets during the next database update.

[0064] This embodiment has the following effects: 1. In the target detection module, an efficient multi-scale attention EMA module is introduced into the YOLOv8 backbone network C2f to enhance the model's detection capability in scenarios with large target flow (such as parks and squares).

[0065] 2. In the trajectory tracking module, by combining the ReID target feature information extracted by ByteTrack and FastReID, the problem of matching and tracking failure due to lack of target features in complex scenarios is solved, and the robustness of the tracker is enhanced.

[0066] 3. In the cross-camera multi-target tracking system, a DM (database management) module is introduced to store multi-frame information from various viewpoints and manage it in a unified manner, thereby enhancing the stability of single-camera tracking and cross-camera matching; 4. In the cross-camera matching module, a Track-ReID visual matching method is proposed, which combines FastReID with trajectory information. By utilizing the multi-frame target information stored in the DM module, feature matching and trajectory information matching are performed on targets under multiple perspectives. This makes the system no longer rely solely on the current frame information, but combines multi-frame historical information for association matching.

[0067] Example 3 This embodiment demonstrates the evaluation metrics for the above embodiments.

[0068] (1) Target detection evaluation indicators: Intersection over Union (IoU) is the crossover ratio between the detected bounding boxes and the ground truth labeled boxes, reflecting the degree of match between the model's predictions and the actual labeled results. First, it's necessary to calculate the precision and recall at different IoU thresholds.

[0069]

[0070] mAP measures the accuracy of object detection algorithms.

[0071]

[0072] (2) Pedestrian re-identification evaluation indicators: Rank-1 accuracy represents the probability that the image with the highest similarity to the target person in the candidate pool is the correct image; it is the percentage of the first returned image that is correct. The formula for calculating rank-1 is as follows:

[0073] in, The set of all query images. To search for images The label of the first result returned in the corresponding database.

[0074] (3) Multi-target tracking and evaluation indicators

[0075] Multiple Object Tracking Accuracy (MOTA) measures the overall performance of a tracking algorithm in target localization and trajectory association. Its formula is:

[0076] MOTA calculates the matching status of all frames. For frame number, For false negatives, As a false positive example, For the number of ID switches, By calculating the number of missed detections and false detections across all frames, as well as the number of ID changes, based on the actual number of targets, the overall performance of the tracking system can be intuitively evaluated.

[0077] Identification F1 score (IDF1) measures the accuracy with which a model maintains the identity of the tracked object during tracking, characterizing whether a tracker can track an object for a long time. Its formula is:

[0078] in, To match the target number of IDs, To match the target number of ID errors, This represents the number of target IDs that were not matched in the actual tags.

[0079] (4) Evaluation index of cross-camera multi-target matching Multi-device target association score (MDA) is used to evaluate the degree of ID association between the same target in the same frame when multiple devices are tracking multiple targets simultaneously. It focuses on the performance evaluation of ID association in a multi-device environment, and its calculation method is as follows:

[0080]

[0081] in, Indicates the first frame, and k Indicates different shooting equipment. This refers to the number of correctly associated target ID pairs in a multi-device, multi-target tracking algorithm. This indicates the number of truly associated ID pairs across multiple devices. This is the number of incorrectly associated ID pairs across multiple devices. This refers to the number of ID pairs that are not correctly associated but still exist across multiple devices. The calculation within parentheses indicates the number of... j The and the first k The accuracy of multiple devices in associating the same target.

[0082] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

[0083] Example 4 like Figure 8 As shown, this embodiment also provides a real-time target tracking method for cross-camera shots within the same field of view based on visual matching, including the following steps: Acquire real-time video data from multiple video sampling cameras within the same field of view; Target detection is performed on the real-time video data to obtain a detection box containing the detected target; Feature extraction is performed on the detection box to obtain feature information; The target is tracked based on the feature information, the target's identity is integrated, trajectory information is output, and the identity and trajectory information are stored in a preset database management module. The database management module is used to store the identity, feature information, and trajectory information of the detected target; Based on the identity identifier, feature information, and trajectory information stored in the database management module, cross-camera matching is performed on the detected target to obtain the cross-camera matching result.

[0084] This embodiment first simultaneously acquires real-time video data from multiple shots within the same field of view, providing a data foundation for multi-target detection and tracking. Then, it performs target detection on the real-time video data to obtain detection boxes, ensuring accurate identification of multiple targets in the frame. Next, it extracts features from the detected targets and stores them in a database management module, giving the targets distinguishable feature representations. Then, it tracks the targets using the target feature information and generates identification and trajectory information, ensuring the continuity of target identity and the integrity of trajectory within the same shot. This embodiment uses a database management module to uniformly store target identification, feature information, and trajectory information, providing reliable data support for cross-shot target matching. Finally, it performs target matching between multiple shots based on the information in the database, achieving cross-shot tracking within the same field of view and effectively improving the real-time performance and accuracy of cross-shot target tracking.

Claims

1. A real-time target tracking system for cross-camera shots within the same field of view based on visual matching, characterized in that, include: The acquisition module is used to acquire real-time video data from multiple video sampling lenses in the same field of view; The target detection module is used to perform target detection on the real-time video data to obtain a detection box containing the detected target; The feature extraction module is used to extract features from the detection box, obtain feature information, and store the feature information into a preset database management module; The trajectory tracking module is used to track the detected target based on the feature information, integrate the identity of the detected target, output trajectory information, and store the identity and trajectory information into a preset database management module. The database management module is used to store the identity, feature information, and trajectory information of the detected target; The cross-camera matching module is used to perform cross-camera matching on the detected target based on the identity identifier, feature information, and trajectory information stored in the database management module, and obtain the cross-camera matching result.

2. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 1, characterized in that, The step of extracting features from the detection box to obtain feature information and storing the feature information in a preset database management module includes the following steps: The target to be detected is input into a preset feature extractor, which extracts the feature information of the target to be detected in the detection frame. Calculate the cosine similarity distance between the feature information of the detected target and the feature information stored in the database management module; Determine whether the cosine similarity distance is higher than a preset addition threshold. If yes, store the feature information of the detected target in the database management module; otherwise, discard the feature information of the detected target.

3. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 1, characterized in that, The cosine similarity distance The calculation formula is as follows: Wherein, A represents the feature information of the target being detected in the detection frame, and B represents the feature information stored in the database management module.

4. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 3, characterized in that, The step of tracking the detected target based on the feature information, integrating the identity of the detected target, and outputting trajectory information includes the following steps: The detection target is predicted to obtain a prediction box for the detection target; The predicted bounding box is matched with the detected bounding box to obtain the matching result; Based on the matching results, the identity identifiers of the detected targets are integrated, and trajectory information containing the locations of successfully matched detection boxes is output.

5. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 4, characterized in that, The detection boxes used to match the predicted boxes with the detection boxes include high-confidence detection boxes with a confidence level exceeding an upper threshold and low-confidence detection boxes with a confidence level below a lower threshold.

6. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 5, characterized in that, The step of matching the predicted bounding box with the detected bounding box includes: First matching: Calculate the intersection-union distance (CID) and feature cosine distance between the predicted bounding box and the high-confidence detection box, respectively. Take the smaller of the CID and feature cosine distances. If the smaller distance is the CID, determine whether the CID is greater than a first preset distance threshold. If yes, the first matching is considered successful; otherwise, the first matching is considered unsuccessful. If the smaller distance is the feature cosine distance, determine whether the feature cosine distance is less than a second preset distance threshold. If yes, the first matching is considered successful; otherwise, the first matching is considered unsuccessful. Second matching: Calculate the second intersection-union distance between the predicted bounding box that failed to match in the first matching and the low-confidence detection box, and determine whether the second intersection-union distance is greater than the third preset distance threshold. If it is, the second matching is considered successful; otherwise, the second matching is considered to have failed. Third match: Calculate the third intersection-union distance between the predicted trajectory box that failed to match in the first two matches and the high-confidence detection box that failed to match in the first match, and determine whether the third intersection-union distance is greater than the fourth preset distance threshold. If it is, the third match is considered successful; otherwise, the third match is considered to have failed.

7. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 6, characterized in that, The step involves integrating the identity identifier of the detected target based on the matching result and outputting trajectory information containing the locations of successfully matched detection boxes, including: If any one of the three matching attempts is successful, the trajectory information containing the location of the successfully matched detection box is output. The detection targets in the successfully matched detection boxes are marked as having the same identity identifier, and the identity identifier, the feature information of the detection target, and the trajectory information are stored in a preset database management module.

8. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 1, characterized in that, The step of performing cross-camera matching on the detected target based on the identity identifier, feature information, and trajectory information stored in the database management module includes: Obtain the detection bounding box of each shot in the current frame and the feature information of the detected target in the detection bounding box; The feature information of the detected target is matched with the feature information stored in the database management module. If the match is successful, the same identity identifier is uniformly assigned to the detected target under multiple views, and the feature information and trajectory information stored in the database management module are updated. If the match fails, proceed to the next step. The trajectory information of the detected targets that failed to match is matched again with the trajectory information stored in the database management module. If the match is successful again, the same identity identifier is uniformly assigned to the detected targets that were matched again from multiple perspectives, and the feature information and trajectory information stored in the database management module are updated. If the match fails again, a new identity identifier is assigned to the detected targets that failed to match again, and new trajectory information is established.

9. The real-time target tracking system for cross-camera shooting within the same field of view based on visual matching according to claim 8, characterized in that, The matching of the feature information of the detected target with the feature information stored in the database management module includes: Calculate the cosine similarity distance between the feature information of the detected target and the feature information stored in the database management module; Determine whether the cosine similarity distance is greater than a preset update threshold. If yes, the match is successful; otherwise, the match fails.

10. A real-time target tracking method for cross-camera shots within the same field of view based on visual matching, characterized in that, Includes the following steps: Acquire real-time video data from multiple video sampling cameras within the same field of view; Target detection is performed on the real-time video data to obtain a detection box containing the detected target; Feature extraction is performed on the detection box to obtain feature information; The target is tracked based on the feature information, the target's identity is integrated, trajectory information is output, and the identity and trajectory information are stored in a preset database management module. The database management module is used to store the identity, feature information, and trajectory information of the detected target; Based on the identity identifier, feature information, and trajectory information stored in the database management module, cross-camera matching is performed on the detected target to obtain the cross-camera matching result.

Images