A multi-modal target tracking method based on decision-level fusion

Abstract

The application belongs to the technical field of target tracking methods, and discloses a multi-modal target tracking method based on decision-level fusion, which comprises the following steps: acquiring visible light and infrared light video streams, inputting corresponding tracking models to obtain tracking results and performing time alignment; establishing a correlation relationship table, performing target pairing based on a time-aligned tracking result set and updating the relationship table; according to the pairing result, establishing a to-be-matched pool containing an independent target set and a to-be-judged set; performing neighborhood consistency evaluation based on the to-be-judged set to realize accurate pairing of the to-be-judged set; and performing target tracking according to the correlation relationship table and the independent target set, and updating the correlation relationship table in real time. The method significantly improves the target pairing accuracy through the neighborhood consistency evaluation mechanism, effectively solves the problem of target correlation loss caused by temporary loss of modes through the dynamic updating mechanism, and improves the tracking stability and continuity of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking methods, and more particularly, to a multi-modal target tracking method based on decision-level fusion. BACKGROUND

[0002] Target tracking is a key technology in the field of computer vision, and is widely used in video surveillance, intelligent transportation, unmanned driving, assisted driving, and human-computer interaction. With the increasing complexity of application scenarios, a single modal sensor is difficult to cope with changes in various environmental conditions, such as changes in light, occlusion, and weather changes. Therefore, multi-modal target tracking technology that uses multiple sensors to obtain complementary information has gradually become a research hotspot. In complex environments, multi-modal fusion can significantly improve the robustness and accuracy of the tracking system. Currently, the fusion strategy of multi-modal target tracking is mainly divided into three levels: data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion directly fuses at the original data level, such as pixel-level fusion of infrared images and visible light images, but this method requires high sensor calibration accuracy and is difficult to handle the heterogeneity between different sensors; feature-level fusion is performed after feature extraction, and the target description is enhanced by fusing different modal feature representations, but the fusion strategy design is complex and requires a large amount of labeled data for training; decision-level fusion fuses the decision results after each modal makes a decision, which is simple to implement, highly modular, and can fully utilize the advantages of each modal.

[0003] However, the existing decision-level fusion method usually only relies on simple spatial distance threshold and class consistency for matching in the target pairing stage. This pairing method is difficult to accurately distinguish whether it is the same target, and is prone to misjudging different targets as the same target or misjudging the same target as different targets, resulting in poor final tracking effect. Especially in a target dense scene, when multiple targets of the same class appear in close proximity, the system has difficulty accurately determining which tracking results correspond to the same entity. In addition, in the actual tracking process, some targets may be temporarily detected only in a single modality due to occlusion, light changes, or other reasons, and are judged by the system as independent targets. When these targets subsequently appear in another modality, the existing system often cannot timely establish their association with existing independent targets. In this case, the targets that should be paired are still treated as independent targets and cannot enjoy the advantages of multi-modal fusion, reducing the tracking stability and accuracy of the system. Over a long period of operation, this problem accumulates, resulting in a large number of targets in the system that can actually be paired but are not paired, seriously affecting overall performance. SUMMARY

[0004] In order to overcome the above problems of the prior art, the present application proposes a multi-modal target tracking method based on decision-level fusion, which is used to solve the above problems.

[0005] The present application provides the following technical solutions:

[0006] A multi-modal target tracking method based on decision-level fusion, comprising:

[0007] Obtaining a visible light video stream and an infrared light video stream, inputting the visible light video stream and the infrared light video stream into a visible light tracking model and an infrared light tracking model respectively to obtain corresponding tracking results, performing time alignment on the two types of tracking results to obtain a time-aligned tracking result set;

[0008] Establishing an association relationship table, performing target pairing based on the time-aligned tracking result set and updating the association relationship table;

[0009] Based on the association relationship table and the time-aligned tracking result set, dividing unpaired tracking results and establishing a to-be-matched pool, the to-be-matched pool comprising an independent target set and a to-be-judged set;

[0010] Performing neighborhood consistency evaluation based on the to-be-judged set to obtain a neighborhood consistency degree, and updating the association relationship table and the to-be-matched pool according to the neighborhood consistency degree;

[0011] According to the association relationship table and the independent target set, performing corresponding target tracking, and updating the association relationship table in real time.

[0012] Preferably, the tracking results comprise a category, a target bounding box, an identification confidence and a target number; and the time alignment is performed according to a timestamp.

[0013] Preferably, the target pairing based on the time-aligned tracking result set and the updating of the association relationship table comprise:

[0014] From the same aligned frame of the time-aligned tracking result set, obtaining tracking results of the visible light tracking model and the infrared light tracking model with the same category;

[0015] Filtering targets with a distance between target bounding boxes in the two tracking results not greater than a preset first distance threshold to form a potential pairing set;

[0016] Iterating the potential pairing set, determining potential pairings without conflicts as pairing relationships; for potential pairings with conflicts, selecting a pair with the smallest distance as the pairing relationship;

[0017] Generating a global identity number for the determined pairing relationship, associating the target numbers of the two tracking results with the global identity number, and writing into the association relationship table.

[0018] Preferably, the step of dividing and establishing the unpaired tracking results into the to-be-matched pool based on the association table and the time-aligned tracking result set comprises:

[0019] When the tracking result categories of the two tracking models are different, the two tracking results are respectively divided into independent target sets;

[0020] When the tracking result categories of the two tracking models are the same and the distance between the target bounding boxes is greater than a preset second distance threshold, the two tracking results are respectively divided into independent target sets;

[0021] When the tracking result categories of the two tracking models are the same and the distance between the target bounding boxes is greater than the first distance threshold and not greater than the second distance threshold, the two tracking results are divided into a to-be-judged set as a target pair;

[0022] Wherein, the second distance threshold is greater than the first distance threshold, and the independent target set and the to-be-judged set constitute the to-be-matched pool.

[0023] Preferably, the step of obtaining the neighborhood consistency degree based on the to-be-judged set comprises:

[0024] Selecting at least one paired target from the association table as a reference target to form a neighborhood reference set;

[0025] For each target pair in the to-be-judged set, the continuous tracking results of the respective targets of the two tracking models in the target pair in a preset time window are respectively obtained;

[0026] Based on the continuous tracking results and the neighborhood reference set, the respective neighborhood feature fingerprints of the two targets in the target pair are established;

[0027] Comparing the neighborhood feature fingerprint similarity of the two targets in the target pair as the neighborhood consistency degree; the step of updating the association table and the to-be-matched pool according to the neighborhood consistency degree comprises:

[0028] When the neighborhood consistency degree is not less than a preset neighborhood consistency degree threshold, it is determined that the two targets in the target pair are the same target, a global identity number is generated for the target pair and the corresponding relationship is written into the association table, and the target pair is removed from the to-be-judged set;

[0029] Otherwise, the target pair is separated and divided into an independent target set.

[0030] Preferably, the step of establishing the respective neighborhood feature fingerprints comprises:

[0031] For each reference target in the neighborhood reference set, the continuous tracking results thereof in the same time window are obtained;

[0032] For the target in the target pair obtained by the visible light tracking model:

[0033] For each reference target in the neighborhood reference set, the direction of the target bounding box relative to the reference target bounding box at each time-aligned position is calculated to form a direction sequence relative to the reference target; the distance between the target bounding box and the reference target bounding box at each time-aligned position is calculated to form a distance sequence relative to the reference target;

[0034] For the target obtained by the infrared light tracking model in the target pair, the same calculation steps as the target obtained by the visible light tracking model are performed;

[0035] The set of direction sequences of each target relative to all reference targets is taken as the direction feature part, and the set of distance sequences is taken as the distance feature part, and the two parts are combined to form the neighborhood feature fingerprint of the target.

[0036] Preferably, the similarity of the neighborhood feature fingerprints of the two targets in the comparison target pair includes:

[0037] For the neighborhood feature fingerprints of the two targets in the target pair, the direction feature parts and the distance feature parts thereof are processed respectively:

[0038] For the direction feature part: for each reference target, the inter-frame change rate sequences of the corresponding direction sequences of the two targets are calculated respectively, and the similarity of the two change rate sequences is compared; the direction change consistency is obtained by weighted averaging the direction change rate similarities of all reference targets;

[0039] For the distance feature part: for each reference target, the inter-frame change rate sequences of the corresponding distance sequences of the two targets are calculated respectively, and the similarity of the two change rate sequences is compared; the distance change consistency is obtained by weighted averaging the distance change rate similarities of all reference targets;

[0040] The direction change consistency and the distance change consistency are weighted averaged according to a preset weight to obtain the neighborhood feature fingerprint similarity.

[0041] Preferably, the performing corresponding target tracking according to the association relationship table and the independent target set includes:

[0042] When the tracking results of the two tracking models exist at the same time in the target tracking process, the final tracking result is determined by weighted fusion, and when only the tracking result of one tracking model exists, the tracking result of the tracking model is used for tracking;

[0043] The target in the independent target set is tracked using the corresponding tracking model.

[0044] Preferably, the real-time updating of the association table comprises: when any tracking model tracks a new target, pairing the new target with an independent target of another tracking model in the independent target set and updating the association table.

[0045] The application provides a multi-modal target tracking method based on decision-level fusion.

[0046] By establishing the association table and introducing the neighborhood consistency evaluation mechanism, the method no longer depends on simple spatial distance threshold and category information for target pairing, but constructs neighborhood feature fingerprints based on the relative motion characteristics between the target and the neighborhood reference target. By calculating the direction sequence and the distance sequence and analyzing the similarity of the inter-frame change rate, the system can accurately distinguish targets with different motion patterns, even if they are close in spatial position and belong to the same category. This multi-dimensional judgment mechanism based on motion characteristics significantly improves the accuracy of target pairing, effectively reduces the mispairing and missing pairing in dense scenes, and enables the system to maintain stable tracking performance in complex environments, laying a reliable foundation for multi-modal fusion.

[0047] Through the mechanism of real-time updating of the association table, when any tracking model detects a new target, the system will actively attempt to pair with an independent target of another modality, rather than simply categorizing it as a new independent target. This dynamic pairing strategy can timely capture targets that are temporarily detected in only a single modality due to occlusion or light changes, and subsequently appear in another modality, and establish an association between them. At the same time, continuous monitoring and updating of the independent target set ensures that the system can utilize multi-modal information for fusion tracking as much as possible, greatly improving the tracking continuity of the system during long-term operation, and providing a reliable guarantee for high-precision target tracking in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 Figure 1 is a flowchart of a multi-modal target tracking method based on decision-level fusion according to the application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application.

[0050] Embodiment 1

[0051] Please refer to Figure 1In this embodiment, a multi-modal target tracking method based on decision-level fusion includes:

[0052] S1, obtain a visible light video stream and an infrared light video stream, input the visible light video stream and the infrared light video stream into a visible light tracking model and an infrared light tracking model respectively to obtain corresponding tracking results, perform time alignment on the two types of tracking results to obtain a time-aligned tracking result set;

[0053] The tracking results include a category, a target bounding box, an identification confidence, and a target number; and the time alignment is performed according to a timestamp.

[0054] In this embodiment, it should be noted that the visible light video stream and the infrared light video stream are basic data inputs for multi-modal target tracking. The visible light video stream is usually collected by a normal RGB camera and can provide rich color and texture information in good lighting conditions; and the infrared light video stream is collected by an infrared thermal imaging camera and can capture the thermal radiation characteristics of the target in low-light or night environments. The two different modal video streams complement each other and can improve the robustness of the tracking system in complex environments.

[0055] It should be noted that the visible light tracking model and the infrared light tracking model are target tracking algorithms designed for their respective modal characteristics. The visible light tracking model usually uses the appearance characteristics of the target, such as color, texture, and shape, for tracking; and the infrared light tracking model mainly uses the thermal radiation characteristics of the target for tracking. The two models can be a target detection and tracking network based on deep learning, or a traditional tracking algorithm such as a correlation filter or mean shift.

[0056] It should be noted that each element in the tracking results has a clear technical meaning: the category represents the type of the tracked target, such as pedestrians, vehicles, etc.; the target bounding box refers to the coordinates of the rectangular box surrounding the target, which is usually composed of the upper-left corner coordinates, width, and height; the identification confidence represents the confidence level of the model for the tracking result, which is usually between 0 and 1; and the target number is a unique identifier assigned by the tracking system to each target to maintain the consistency of the target during tracking. Time alignment is performed according to the timestamp, and the specific method can be to find the frames with the closest timestamps in the two modalities to pair them and form a time-aligned tracking result set.

[0057] S2, establish an association relationship table, and perform target pairing based on the time-aligned tracking result set and update the association relationship table;

[0058] The target pairing based on the time-aligned tracking result set and the updating of the association relationship table include:

[0059] From the same aligned frame of the time-aligned tracking result set, obtain the tracking results of the visible light tracking model and the infrared light tracking model with the same category;

[0060] Screening the target whose distance between the target bounding boxes in the two tracking results is not greater than a preset first distance threshold value, to form a potential pairing set;

[0061] Traversing the potential pairing set, determining the potential pairing without conflict as a pairing relationship, and for the potential pairing with conflict, selecting the pair with the minimum distance to determine as the pairing relationship;

[0062] Generating a global identity number for the determined pairing relationship, associating the target numbers of the two tracking results with the global identity number, and writing into an association relationship table.

[0063] In the embodiment, it should be noted that the association relationship table is used to store the corresponding relationship between the tracking results of the visible light tracking model and the infrared light tracking model. The role is to establish and maintain the mapping relationship between the tracking results of the two modalities, and to provide a basis for subsequent decision-level fusion.

[0064] It should be noted that target pairing is the process of matching the tracking results from two different modalities in the same frame and determining which tracking results correspond to the same entity target. Target pairing is first based on target category for screening, and only targets with the same category will enter the subsequent pairing process, which can effectively reduce the possibility of false pairing. The same category mainly refers to the consistent semantic category of the target, such as being a pedestrian or being a vehicle, etc.

[0065] It should be noted that the distance is usually calculated by the Euclidean distance between the upper left corner coordinates of the bounding boxes, and the first distance threshold value is a preset parameter for judging whether the two target bounding boxes are close enough to be considered as different modalities of the same target. The threshold value needs to consider factors such as camera calibration error, target size and motion speed, and a suitable value can usually be determined through experiments.

[0066] It should be noted that the potential pairing set is all possible pairing combinations that meet the same category and the distance between the bounding boxes is less than the first distance threshold value. When constructing the potential pairing set, it may occur that a visible light target forms a potential pairing with multiple infrared light targets, or an infrared light target forms a potential pairing with multiple visible light targets, which results in pairing conflict. When handling the conflict, the pairing relationship with the minimum distance is preferred, which is simple and efficient and suitable for real-time processing requirements.

[0067] It should be noted that a new global identity number is assigned each time a new pairing relationship is determined. When updating the association relationship table, the visible light target number, the infrared light target number and the corresponding global identity number are written into the table to form a mapping relationship between the three.

[0068] S3, based on the association table and the time alignment tracking result set, the unpaired tracking results are divided and a to-be-matched pool is established, the to-be-matched pool includes an independent target set and a to-be-judged set;

[0069] The based on the association table and the time alignment tracking result set, the unpaired tracking results are divided and a to-be-matched pool is established includes:

[0070] When the tracking result categories of the two tracking models are different, the two tracking results are respectively divided into the independent target set;

[0071] When the tracking result categories of the two tracking models are the same and the distance between the target bounding boxes is greater than a preset second distance threshold, the two tracking results are respectively divided into the independent target set;

[0072] When the tracking result categories of the two tracking models are the same and the distance between the target bounding boxes is greater than the first distance threshold and not greater than the second distance threshold, the two tracking results are divided into the to-be-judged set as a target pair;

[0073] Wherein, the second distance threshold is greater than the first distance threshold, and the independent target set and the to-be-judged set constitute the to-be-matched pool.

[0074] In the embodiment, it needs to be explained that the unpaired tracking result refers to the tracking result that fails to establish a corresponding relationship with the tracking result of another modality in the target pairing step. These unpaired results may be caused by occlusion, sensor blind area or target characteristics not obvious in a certain modality. Reasonable division and processing of these unpaired tracking results can improve the robustness of the multi-modal fusion tracking method.

[0075] It needs to be explained that the to-be-matched pool is used to store and manage the unpaired tracking results. The to-be-matched pool is divided into two parts: the independent target set and the to-be-judged set. The targets in the independent target set are considered to be visible only in a single modality, and the tracking model of the corresponding modality will be used for tracking; and the target pairs in the to-be-judged set need further analysis to determine whether they represent the same entity target.

[0076] It needs to be explained that when the two tracking results are the same but the distance between the target bounding boxes exceeds the second distance threshold, although they are the same, the spatial position difference is too large, and they are considered as two independent targets, and are respectively divided into the independent target set. The second distance threshold can consider different input device calibration errors and different model tracking errors, and is set on the basis of actual observation and statistical analysis.

[0077] It should be noted that when two tracking results are of the same category and the distance between the target bounding boxes is between the first distance threshold and the second distance threshold, it cannot be directly determined whether they are the same target based on the category and the distance. In this case, the two tracking results are classified into the to-be-judged set as a target pair, and are waiting for subsequent neighborhood consistency evaluation to further determine. This processing manner helps to reduce misjudgment and missed judgment.

[0078] S4, neighborhood consistency evaluation is performed based on the to-be-judged set to obtain a neighborhood consistency degree, and the association relationship table and the to-be-matched pool are updated according to the neighborhood consistency degree;

[0079] The neighborhood consistency evaluation based on the to-be-judged set comprises:

[0080] At least one paired target is selected from the association relationship table as a reference target to form a neighborhood reference set;

[0081] For each target pair in the to-be-judged set, the continuous tracking results of the respective targets of the two tracking model pairs in the target pair within a preset time window are obtained respectively;

[0082] Based on the continuous tracking results and the neighborhood reference set, a respective neighborhood feature fingerprint is established for each of the two targets in the target pair;

[0083] The neighborhood feature fingerprint similarity of the two targets in the target pair is compared as the neighborhood consistency degree; and the association relationship table and the to-be-matched pool are updated according to the neighborhood consistency degree, comprising:

[0084] When the neighborhood consistency degree is not less than a preset neighborhood consistency degree threshold, it is determined that the two targets in the target pair are the same target, a global identity number is generated for the target pair, and the corresponding relationship is written into the association relationship table, and the target pair is removed from the to-be-judged set;

[0085] Otherwise, the target pair is separated and classified into an independent target set.

[0086] The neighborhood feature fingerprint comprises:

[0087] For each reference target in the neighborhood reference set, the continuous tracking results thereof within the same time window are obtained;

[0088] For the target in the target pair obtained by the visible light tracking model:

[0089] For each reference target in the neighborhood reference set, the direction of the target bounding box relative to the reference target bounding box at each time alignment position is calculated to form a direction sequence relative to the reference target; and the distance between the target bounding box and the reference target bounding box at each time alignment position is calculated to form a distance sequence relative to the reference target;

[0090] For the target pair obtained through the infrared light tracking model, the same calculation steps as the target obtained through the visible light tracking model are performed.

[0091] The direction sequence set of each target relative to all reference targets is taken as a direction feature part, and the distance sequence set is taken as a distance feature part, and the two parts are combined to form a neighborhood feature fingerprint of the target.

[0092] The comparison of the neighborhood feature fingerprints of the two targets in the target pair includes:

[0093] For the neighborhood feature fingerprints of the two targets in the target pair, the direction feature parts and the distance feature parts thereof are processed respectively:

[0094] For the direction feature part: for each reference target, the inter-frame change rate sequences of the corresponding direction sequences of the two targets are calculated respectively, and the similarity of the two change rate sequences is compared; the direction change consistency is obtained by weighted averaging the direction change rate similarities of all reference targets.

[0095] For the distance feature part: for each reference target, the inter-frame change rate sequences of the corresponding distance sequences of the two targets are calculated respectively, and the similarity of the two change rate sequences is compared; the distance change consistency is obtained by weighted averaging the distance change rate similarities of all reference targets.

[0096] The direction change consistency and the distance change consistency are weighted averaged according to a preset weight to obtain the neighborhood feature fingerprint similarity.

[0097] In this embodiment, it should be noted that the neighborhood consistency evaluation is used to solve the target association problem that cannot be determined by relying on spatial distance and category. The relative motion pattern between the target and the surrounding known paired target (neighborhood) is used for judgment. If the two tracking results correspond to the same entity, their motion modes relative to the surrounding targets should be similar.

[0098] It should be noted that the neighborhood reference set is composed of targets with determined pairing relationship in the association relationship table, which provides a stable reference coordinate system for analyzing the relative motion of the target to be determined. At least one paired target is selected as a reference in order to establish a minimum reference system, and in actual applications, multiple reference targets with reasonable spatial distribution are usually selected to improve the robustness of the evaluation. The more the number of neighborhood reference targets, the richer the feature fingerprint constructed, and the stronger the anti-interference ability.

[0099] It should be noted that the preset time window refers to a continuous time sequence, usually containing 5-20 frames of data. The neighborhood feature fingerprint is a multi-dimensional feature vector describing the relative motion characteristics of the target. By calculating the direction and distance sequence between the target and the reference target, the relative motion information of the target in the two-dimensional plane is captured. There are various implementation methods for calculating the direction and distance, which can adopt the method based on the coordinates of the upper left corner of the target bounding box, that is, the direction angle is obtained by calculating the included angle between the line connecting the two points and the horizontal axis; the distance calculation adopts the Euclidean distance between the coordinates of the upper left corner of the two target bounding boxes. This calculation method based on the upper left corner is simple and efficient, and can meet the needs of most application scenarios. The direction sequence reflects the angle change of the target relative to the reference target, which can effectively describe the shape feature of the motion trajectory; the distance sequence reflects the approach or away relationship between the target and the reference target, which can describe the motion amplitude feature. The combination of the two features greatly enhances the distinguishing ability of the feature fingerprint.

[0100] It should be noted that the inter-frame change rate sequence refers to the difference between adjacent frames in direction or distance, which eliminates the influence of absolute position and only focuses on the relative change pattern. The feature fingerprint has strong robustness to camera calibration errors and target initial position differences. The similarity of the two change rate sequences is usually calculated by cosine similarity or correlation coefficient, which can effectively measure the morphological consistency of the two sequences rather than the absolute value proximity.

[0101] It should be noted that the direction change consistency and the distance change consistency measure the similarity of the motion patterns of the two targets from different angles. In the case of multiple neighborhood reference targets, the similarities obtained from each reference target need to be weighted and averaged to obtain the final consistency. The weighting coefficient can be set based on the distance between the target and the reference target, and the closer the reference target, the greater the weight. Weighted averaging of the two consistency degrees can balance the advantages of different features and improve the accuracy of evaluation. The setting of the weight coefficient is usually determined based on experimental analysis, and may need to be adjusted in different application scenarios to adapt to specific target motion characteristics and sensor characteristics.

[0102] It should be noted that the neighborhood consistency threshold is the decision limit for determining whether two tracking results correspond to the same entity. The setting of the threshold needs to balance the accuracy and recall rate, and a too high threshold will cause missed effective pairing, and a too low threshold may introduce false pairing. In actual application, it can be adjusted according to the specific needs of the scene, and a higher threshold can be set for safety-critical applications, and a moderate threshold can be used for ordinary monitoring scenarios.

[0103] S5, performing corresponding target tracking according to the association relationship table and the independent target set, and updating the association relationship table in real time.

[0104] The corresponding target tracking according to the association table and the independent target set comprises:

[0105] When the tracking results of the two tracking models exist simultaneously in the target tracking process, the final tracking result is determined by weighted fusion, and when only the tracking result of one tracking model exists, the tracking result of the tracking model is used for tracking;

[0106] The target in the independent target set is tracked by using the corresponding tracking model.

[0107] The real-time updating of the association table comprises: when any tracking model tracks a new target, the new target is paired with an independent target of another tracking model in the independent target set, and the association table is updated.

[0108] In the embodiment, it should be noted that when the two tracking models simultaneously generate tracking results for the same target, a more reliable comprehensive result is generated by weighted fusion. The weighted fusion generally considers factors such as recognition confidence of each mode, environmental conditions and historical tracking stability to allocate weights. For example, in a strong light environment, the weight of the visible light mode can be higher; and in the night or fog, the weight of the infrared light mode can be greater. Such an adaptive weighting strategy can maintain stable tracking performance in different environmental conditions.

[0109] It should be noted that when the target in the association table is temporarily detected only in one mode, this can be caused by target occlusion, sensor blind area or target characteristics not obvious in a certain mode. At this time, the existing single-mode tracking result is directly used, which maximizes the stability of tracking and reduces the loss probability. For the targets in the independent target set, these targets exist temporarily only in a single model. For such targets, the tracking model of the corresponding mode is directly used for tracking. This processing method not only guarantees the comprehensiveness of the tracked target, but also avoids errors that may be caused by forced fusion.

[0110] It should be noted that when any tracking model detects a new target, the new target will first be paired with an independent target of another mode. This pairing process can use the same method as the initial target pairing to realize dynamic association. This dynamic updating mechanism can maximize the matching rate between targets of different models. More targets can be tracked by fusion, improving the accuracy and robustness of target tracking.

[0111] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, and the division of the units is only one of the ways, and in actual implementation, other division ways can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0112] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be included in the protection scope of the present application.

[0113] Finally: the above description is only the preferred embodiment of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multimodal target tracking method based on decision-level fusion, characterized in that, include: Acquire visible light video streams and infrared light video streams, input them into the visible light tracking model and the infrared light tracking model respectively to obtain the corresponding tracking results, perform time alignment on these two types of tracking results to obtain a time-aligned tracking result set; Establish an association table, perform target pairing based on the time-aligned tracking results set, and update the association table; Based on the relationship table and the time-aligned tracking result set, the unpaired tracking results are divided and a matching pool is established. The matching pool includes an independent target set and a set to be judged. The neighborhood consistency degree is obtained by evaluating the neighborhood consistency of the set to be judged, and the association table and the matching pool are updated according to the neighborhood consistency degree. Select at least one paired target from the association table as a reference target to form a neighborhood reference set; For each target pair in the set to be judged, obtain the continuous tracking results of the two tracking models for their respective targets within a preset time window; Based on continuous tracking results and neighborhood reference sets, neighborhood feature fingerprints are established for each of the two targets in the target pair. For each reference target in the neighborhood reference set, obtain its continuous tracking results within the same time window; For the target obtained by the visible light tracking model in the target pair: For each reference target in the neighborhood reference set, calculate the direction of the target bounding box relative to the reference target bounding box at each time alignment position, forming a direction sequence relative to the reference target; Calculate the distance between the target bounding box and the reference target bounding box at each time alignment position to form a distance sequence relative to the reference target; For targets obtained through the infrared light tracing model in a target pair, perform the same calculation steps as for targets obtained through the visible light tracing model; The set of orientation sequences of each target relative to all reference targets is used as the orientation feature part, and the set of distance sequences is used as the distance feature part. The two parts are combined to form the neighborhood feature fingerprint of the target. The similarity of neighborhood feature fingerprints between two targets in a target pair is used as the neighborhood consistency score; updating the association table and the matching pool based on the neighborhood consistency score includes: When the neighborhood consistency is not less than the preset neighborhood consistency threshold, the two targets in the target pair are determined to be the same target, a global identity number is generated for them, the corresponding relationship is written into the association table, and they are removed from the set to be determined. Otherwise, separate the target pair and assign it to an independent target set; Execute corresponding target tracking based on the relationship table and the set of independent targets, and update the relationship table in real time.

2. The multimodal target tracking method based on decision-level fusion according to claim 1, characterized in that, The tracking results include category, target bounding box, identification confidence level, and target number; the time alignment is based on timestamps.

3. The multimodal target tracking method based on decision-level fusion according to claim 2, characterized in that, The process of matching targets and updating the association table based on the time-aligned tracking results set includes: From the same aligned frame of the time-aligned tracking result set, obtain the tracking results of the visible light tracking model and the infrared light tracking model of the same category; Targets whose distance between the bounding boxes of two tracking results is no greater than a preset first distance threshold are selected to form a potential pairing set; Traverse the set of potential pairings and identify the potential pairings that do not conflict as pairings; for potential pairings that conflict, select the pair with the smallest distance to identify the pairing. Generate a global identity number for the established pairing relationship, associate the target number of the two tracking results with the global identity number, and write it into the association table.

4. The multimodal target tracking method based on decision-level fusion according to claim 3, characterized in that, The process of dividing unpaired tracking results and establishing a matching pool based on the relationship table and time-aligned tracking result set includes: When the tracking results of two tracking models are of different categories, the two tracking results are assigned to independent target sets respectively; When the tracking results of two tracking models are of the same category and the distance between the target bounding boxes is greater than a preset second distance threshold, the two tracking results are respectively assigned to independent target sets; When the tracking results of two tracking models are of the same category and the distance between the target bounding boxes is greater than the first distance threshold but not greater than the second distance threshold, the two tracking results are classified as a target pair and included in the set to be determined. Among them, the second distance threshold is greater than the first distance threshold, and the independent target set and the set to be judged form the matching pool.

5. The multimodal target tracking method based on decision-level fusion according to claim 1, characterized in that, The neighborhood feature fingerprint similarity between the two targets in the comparison target pair includes: For the neighborhood feature fingerprints of the two targets in the target pair, their orientation feature parts and distance feature parts are processed separately: For the orientation feature part: For each reference target, calculate the inter-frame change rate sequence of the orientation sequences of the two targets respectively, and compare the similarity of the two change rate sequences; perform a weighted average of the orientation change rate similarity of all reference targets to obtain the orientation change consistency. For the distance feature part: For each reference target, calculate the inter-frame change rate sequence of the distance sequences corresponding to the two targets, and compare the similarity of the two change rate sequences; perform a weighted average of the distance change rate similarity of all reference targets to obtain the distance change consistency. The similarity of neighborhood feature fingerprints is obtained by weighting the consistency of direction change and the consistency of distance change according to preset weights.

6. The multimodal target tracking method based on decision-level fusion according to claim 1, characterized in that, The step of performing corresponding target tracking based on the relationship table and the set of independent targets includes: When the tracking results of two tracking models exist simultaneously during the target tracking process, the final tracking result is determined by weighted fusion. When only the tracking result of one tracking model exists, the tracking result of that tracking model is used for tracking. The corresponding tracking model is used to track the targets in the independent target set.

7. The multimodal target tracking method based on decision-level fusion according to claim 1, characterized in that, The real-time update of the association table includes: when any tracking model tracks a new target, it is paired with an independent target of another tracking model in the independent target set, and the association table is updated.

Images