Passive multi-band sensor array system and multi-target tracking method

Abstract

The present application relates to a kind of passive multi-band sensor array system and multi-target tracking method, wherein the system includes: multi-band sensor group, ambient illuminance sensor, synchronization and control unit and preprocessing unit;Multi-band sensor group includes visible light, near infrared and long-wave infrared sensor integrated in unified shell by coaxial light path and all do not include active emitter;Synchronization and control unit generates synchronization signal, controls multi-band sensor group and ambient illuminance sensor to carry out synchronous data acquisition;Preprocessing unit receives raw image data, ambient illuminance data and synchronization signal, outputs structured multi-modal data package, provides for the multi-target tracking engine of back end, and the tracking state of multiple targets is output by multi-target tracking engine.The present application can realize stable, continuous, high-precision tracking to multiple targets in a variety of complex scenes, effectively solve the problem that existing multi-target tracking scheme environment adaptability is poor, depends on active emission, robustness is insufficient.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a passive multi-band sensor array system and a multi-target tracking method. Background Technology

[0002] In fields such as photography, security monitoring, and autonomous driving, multi-target tracking technology plays a crucial role as a core topic in computer vision. Its core task is to continuously detect targets in image sequences and maintain the consistency of their identification.

[0003] Traditional multi-target tracking solutions typically rely on a single visual sensor, such as a visible light camera. However, such solutions are highly susceptible to changes in ambient lighting, and their performance degrades drastically under complex conditions such as nighttime, foggy weather, and strong backlighting. To improve robustness, existing technologies have incorporated multi-band sensing techniques and information fusion schemes. However, for applications with strict "passive" requirements (i.e., none of the sensors contain any active transmitters), such as long-term field biological observation, covert security monitoring, or edge devices with extremely limited energy resources, existing technologies face more severe and unique challenges:

[0004] First, at the system architecture level, existing high-performance solutions mostly rely on active sensors (such as lidar or active near-infrared illuminators). This not only has problems such as high power consumption, easy exposure, high cost, and potential interference with biological systems, but also directly violates the fundamental constraint of "passive" design, making it completely unusable in such application scenarios.

[0005] Secondly, regarding energy consumption, even when using passive sensors, existing solutions typically require multiple sensors (especially high-power long-wave infrared sensors) to operate at full power continuously to avoid performance degradation. In "passive" deployment environments that often rely on batteries or solar power, this continuous high-power operation mode will drastically shorten the device's battery life, making it difficult to meet the application requirements for long-term operation and significantly reducing its practicality.

[0006] Secondly, at the algorithm level, existing technologies lack effective mechanisms to compensate for the inherent disadvantages of passive environments, such as weak signals and high noise. Their information fusion strategies are mostly static or heuristic, unable to dynamically adjust the weights of data in each band according to real-time ambient lighting conditions. This results in poor fusion performance and insufficient tracking stability during periods of changing lighting. In the data association stage, they often rely too heavily on single-modal features, making them prone to identity switching or tracking loss in scenarios with dense targets and mutual occlusion. In passive environments where the signal-to-noise ratio is already low, this problem is further amplified.

[0007] Therefore, under the strict technical constraint of "passive", existing multi-target tracking schemes have the following problems: relying on active perception makes the scheme directly unusable; the continuous full-power operation mode is fundamentally contradictory to the limited energy supply; the information fusion and data association mechanism has poor flexibility and insufficient environmental adaptability, and cannot effectively cope with the degradation of perception capabilities under "passive" conditions. Summary of the Invention

[0008] The present invention aims to provide a passive multi-band sensor array system and a multi-target tracking method to overcome the shortcomings of the prior art. The technical problem to be solved by the present invention is achieved through the following technical solution.

[0009] In a first aspect, embodiments of the present invention provide a passive multi-band sensor array system for multi-target tracking, comprising: a multi-band sensor group, an ambient light sensor, a synchronization and control unit, and a preprocessing unit;

[0010] The multi-band sensor group includes at least one visible light sensor, one near-infrared sensor, and one long-wave infrared sensor. All sensors in the multi-band sensor group are integrated in a unified housing via a coaxial optical path and none of them contain an active transmitter.

[0011] The synchronization and control unit is used to generate a synchronization signal and control the multi-band sensor group and the ambient light sensor to perform synchronized data acquisition.

[0012] The preprocessing unit is used to receive raw image data from the multi-band sensor group, ambient illumination data from the ambient illumination sensor, and synchronization signals from the synchronization and control unit, and output a structured multimodal data packet to the back-end multi-target tracking engine. The multi-target tracking engine processes the structured multimodal data packet and outputs the tracking status of multiple targets. The structured multimodal data packet includes: spatiotemporally registered multi-band images, foreground mask images, ambient illumination values ​​synchronized with the images, and corresponding timestamp information.

[0013] Secondly, embodiments of the present invention provide a multi-target tracking method based on a passive multi-band sensor array, applied to a multi-target tracking engine, wherein the multi-target tracking engine uses the aforementioned system to collect the structured multimodal data packets; the method includes:

[0014] The structured multimodal data packet is received, which includes at least a spatiotemporally registered multiband image, a foreground mask, an ambient illumination value synchronized with the image, and corresponding timestamp information.

[0015] Based on the foreground mask, potential regions in the image are located, and target detection is performed within the potential regions to obtain multiple candidate target bounding boxes;

[0016] For each candidate target bounding box, a discriminative feature vector incorporating multi-band information is extracted. The extraction process of the discriminative feature vector includes:

[0017] Visible light feature vectors, near-infrared feature vectors, and long-wave infrared feature vectors are extracted from the corresponding regions of the spatiotemporally registered visible light image, near-infrared image, and long-wave infrared image, respectively.

[0018] Based on the current ambient illuminance value, the confidence weights of the visible light feature vector, the near-infrared feature vector, and the long-wave infrared feature vector are dynamically calculated.

[0019] Based on the confidence weights, the visible light feature vector, the near-infrared feature vector, and the long-wave infrared feature vector are weighted and fused to generate the discriminative feature vector;

[0020] Based on the discriminative feature vector, timestamp sequence, and active trajectory set, data association is performed, wherein the timestamp sequence includes timestamp information of the current frame and historical timestamps, and the data association process includes:

[0021] A multimodal association cost matrix is ​​constructed that integrates appearance similarity cost and motion consistency cost. The appearance similarity cost is determined based on the cosine distance between the discriminative feature vector corresponding to the bounding box of the candidate target in the current frame and the feature vector stored in each active trajectory in the active trajectory set. The motion consistency cost is determined based on the Mahalanobis distance between the expected position and the actual detection position of the bounding box of the candidate target in the current frame. The expected position of each active trajectory in the current frame is predicted based on the active trajectory set and historical timestamps.

[0022] Solve the multimodal association cost matrix, and perform optimal matching between the candidate target bounding boxes of the current frame and the active trajectory set;

[0023] In response to the completion of data association, based on the matching status of the candidate target bounding box of the current frame with the active trajectory set, the active trajectory set is updated, and the current status of all active trajectories in the current frame is output. The current status includes the tracking bounding box, identity identifier and tracking confidence of each active trajectory.

[0024] The technical solution of this invention adopts a passive multi-band sensing design, eliminating the dependence on active transmitters and solving the problems of applicability and concealment. A dynamic weighted fusion strategy based on ambient illumination enables intelligent and flexible adjustment of the fusion mechanism, overcoming the poor environmental adaptability of traditional static fusion. By constructing a multimodal association cost matrix that integrates motion and appearance costs, the limitations of single-feature association are overcome, significantly improving the robustness and accuracy of data association in complex scenarios. Combined with foreground mask pre-screening and a multimodal discrimination mechanism, computational overhead is significantly reduced while ensuring tracking accuracy. The overall multi-target tracking solution forms an intelligent closed loop of perception, decision-making, and execution, significantly improving the level of automated tracking and solving the problem of reliable tracking in variable outdoor environments. Attached Figure Description

[0025] Figure 1 This is an interactive architecture diagram of the passive multi-band sensor array system and the multi-target tracking engine provided in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of a multi-target tracking method provided in an embodiment of the present invention;

[0027] Figure 3 A schematic diagram illustrating the construction of a multimodal correlation cost matrix provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram illustrating the updating of the active trajectory set provided in an embodiment of the present invention. Detailed Implementation

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] This invention provides a passive multi-band sensor array system for multi-target tracking, such as... Figure 1 As shown, the passive multi-band sensor array system 10 includes: a multi-band sensor group 11, an ambient light sensor 12, a synchronization and control unit 13, and a preprocessing unit 14; the multi-band sensor group 11 includes at least one visible light sensor 111, one near-infrared sensor 112, and one long-wave infrared sensor 113. All sensors in the multi-band sensor group 11 are integrated in a unified housing through a coaxial optical path, and none of them contain an active transmitter;

[0031] The synchronization and control unit 13 is used to generate a synchronization signal and control the multi-band sensor group 11 and the ambient light sensor 12 to perform synchronous data acquisition.

[0032] The preprocessing unit 14 is used to receive raw image data from the multi-band sensor group 11, ambient illumination data from the ambient illumination sensor 12, and synchronization signals from the synchronization and control unit 13, and output a structured multimodal data packet to the back-end multi-target tracking engine 20. The multi-target tracking engine 20 processes the structured multimodal data packet and outputs the tracking status of multiple targets. The structured multimodal data packet includes: spatiotemporally registered multi-band image, foreground mask, ambient illumination value synchronized with the image, and corresponding timestamp information.

[0033] The passive multi-band sensor array system 10 provided in this embodiment of the invention includes a multi-band sensor group 11, an ambient illuminance sensor 12, a synchronization and control unit 13 connected to the multi-band sensor group 11 and the ambient illuminance sensor 12, and a preprocessing unit 14 connected to the multi-band sensor group 11, the ambient illuminance sensor 12, and the synchronization and control unit 13.

[0034] The multi-band sensor group 11 includes a visible light sensor 111, a near-infrared sensor 112, and a long-wave infrared sensor 113, which serve as image sensors. All image sensors are passive, receiving energy from the external environment without active transmitters. This passive design, lacking active transmitters, is the physical basis and prerequisite for this solution. The passive design enables completely passive, covert detection, significantly reducing power consumption and extending battery life. Furthermore, the passive design ensures that the performance of all image sensors is strictly dependent on the natural environment, making intelligent scheduling based on ambient light levels possible.

[0035] Visible light sensor 111, near-infrared sensor 112, and long-wave infrared sensor 113 cover the visible light, near-infrared, and long-wave infrared bands, respectively. To eliminate parallax between multiple sensors and achieve pixel-level precise registration between multi-band images, supporting multi-modal fusion and stable tracking, all image sensors are precisely integrated into a unified housing via a coaxial optical path. This ensures optical axis consistency, excellent mechanical structural rigidity, and improves long-term stability and reliability under complex environments such as vibration and temperature differences. Furthermore, the image sensors are housed within the unified housing and connected to an optical window on the housing via a coaxial optical path for accurate imaging and protection.

[0036] As a preferred embodiment, the visible light sensor 111, the near-infrared sensor 112, and the long-wave infrared sensor 113 share the same beam splitter prism. This optical component can efficiently separate the incident beam into three different sensors, which is one of the most compact, efficient, and low-cost reasonable design solutions for achieving multi-band coaxial integration.

[0037] The ambient light sensor 12, as a visual sensing unit, is placed on, for example, a uniform housing surface to sense the true ambient light intensity in situ without obstruction, providing key decision inputs for the dynamic scheduling of the working mode of the image sensor in the multi-band sensor group 11 and the optimization of the overall system power consumption.

[0038] After generating a synchronization signal, the synchronization and control unit 13 sends the synchronization signal to the multi-band sensor group 11 and the ambient light sensor 12, and controls the multi-band sensor group 11 and the ambient light sensor 12 to perform synchronous data acquisition, so as to ensure that the multi-band image data and the ambient light data have strictly consistent timestamps, providing a basis for the preprocessing unit 14 to generate structured multimodal data packets.

[0039] The preprocessing unit 14 is connected to the multi-band sensor group 11, the ambient light sensor 12, and the synchronization and control unit 13. It receives raw image data from the multi-band sensor group 11, ambient light data from the ambient light sensor 12, and synchronization signals from the synchronization and control unit 13. It processes the raw image data, ambient light data, and synchronization signals, and outputs a structured multimodal data packet. The output structured multimodal data packet includes: a spatiotemporally registered multi-band image, a foreground mask, ambient light values ​​synchronized with the image, and corresponding timestamp information.

[0040] The preprocessing unit 14 provides structured multimodal data packets to the backend multi-target tracking engine 20, which is deployed on the backend server and processes the data packets to achieve tracking and status analysis of multiple targets.

[0041] The following describes a multi-target tracking scheme based on a passive multi-band sensor array system and a multi-target tracking engine, using a specific scenario. A passive multi-band sensor array system 10 is deployed near deer habitats (such as forest clearings or water source slopes). The core of this system is a multi-band sensor group 11, which includes three core sensors: a 5-megapixel color visible light sensor 111, a near-infrared sensor 112 with a resolution of 1280x1024, and a long-wave infrared sensor 113 with a resolution of 640x512. The three sensors are integrated into a unified housing via a coaxial optical path design and share the same optical lens and optical path through a beam splitter assembly, ensuring that the images of each band have a completely consistent field of view, fundamentally eliminating parallax. The multi-band sensor group 11 is not equipped with any supplementary lighting or laser emitters, avoiding interference with external biological activities. An ambient light sensor 12 is installed in a suitable location within the unified housing to perceive ambient light intensity in real time without obstruction. The synchronization and control unit 13 (which can be integrated into the device's internal motherboard) generates synchronization signals according to a preset cycle (e.g., 30 times per second), controlling the three sensors and the ambient light sensor 12 to acquire data at the same microsecond-level time point, ensuring that the acquired data is completely consistent in time. The preprocessing unit 14 (e.g., an embedded processing chip) receives image streams from the three sensors, ambient light data from the ambient light sensor 12, and synchronization signals from the synchronization and control unit 13 in real time. The preprocessing unit 14 processes the above data (including spatiotemporal registration, foreground detection, and light synchronization) every 33 milliseconds (i.e., 30 frames / second), and encapsulates it into a structured multimodal data packet. This data packet is transmitted to the backend server in real time through a network interface. The backend server runs the multi-target tracking engine 20, which receives and processes the serialized multimodal data packet, analyzes the trajectory of each individual in the deer herd, and achieves target tracking.

[0042] The passive multi-band sensor array system provided in this invention integrates visible light sensors, near-infrared sensors, and long-wave infrared sensors through a coaxial optical path, achieving pixel-level spatiotemporal registration of multi-band images and providing a data foundation for multimodal fusion tracking. By designing a synchronous acquisition mechanism, strict synchronization between ambient illumination data and image data is achieved. Furthermore, the introduction of an ambient illumination sensor provides a decision-making basis for adaptively scheduling the image sensor's operating mode based on ambient light, optimizing overall system power consumption, and dynamically adjusting fusion weights. Through passive design and unified shell packaging, the system's reliability and durability in complex outdoor environments are improved while avoiding interference with biological organisms. The generation of structured multimodal data packets by a preprocessing unit significantly reduces the data processing complexity of the multi-target tracking engine and improves overall operating efficiency. Moreover, this system, in conjunction with a multi-target tracking engine, can achieve stable, continuous, and high-precision tracking of multiple targets in various complex scenarios, effectively solving the problems of poor environmental adaptability, reliance on active transmission, rigid fusion mechanisms, and insufficient robustness in existing multi-target tracking solutions.

[0043] In embodiments of the present invention, such as Figure 1 As shown, the ambient illuminance data collected by the ambient illuminance sensor 12 is provided to the synchronization and control unit 13. The synchronization and control unit 13 dynamically configures the standby sensor and auxiliary sensor in the multi-band sensor group 11 based on the ambient illuminance data collected by the ambient illuminance sensor 12, and controls the auxiliary sensor to be in a low-power standby state by default.

[0044] The synchronization and control unit 13 dynamically configures the operating modes of the image sensors in the multi-band sensor group 11 based on the ambient light data collected by the ambient light sensor 12: setting the image sensor adapted to the ambient light data as a standby sensor, and setting at least one of the remaining image sensors as a low-power standby auxiliary sensor. The standby sensor undertakes the main sensing responsibility, providing a high-quality image data stream in the current environment. The auxiliary sensor in the low-power standby state is a backup sensor, used to quickly switch to full-power operation based on the instructions of the synchronization and control unit 13 when there is a need for shooting (such as when a target to be tracked is identified), so as to cooperate with the standby sensor to carry out multi-band image acquisition. By reasonably configuring the standby sensor and auxiliary sensor based on the ambient light, the multi-band sensor group 11 can be transformed from a full-power static operating mode to a dynamic mode of intelligent scheduling on demand, thereby significantly optimizing the overall power consumption and extending the continuous working time of the equipment in the field while ensuring the system's sensing performance.

[0045] As an example, the synchronization and control unit 13 performs intelligent scheduling based on the ambient illuminance (e.g., lux value) read by the ambient illuminance sensor 12. In high-illuminance environments (e.g., during daytime, ambient illuminance > 50 lux): the visible light sensor 111 is configured as a standby sensor, providing images with rich colors and clear details; the near-infrared sensor 112 and the long-wave infrared sensor 113 serve as auxiliary sensors, defaulting to a low-power standby state and only being activated when data fusion is required. In low-to-medium illuminance environments (e.g., during dusk or dawn, 10 lux < ambient illuminance ≤ 50 lux): the signal-to-noise ratio of the visible light sensor 111 decreases. At this time, the near-infrared sensor 112, due to its high sensitivity to weak light, is switched to a standby sensor; the long-wave infrared sensor 113 continues to work as an auxiliary sensor, providing thermal radiation information for fusion; the visible light sensor 111 can switch to standby or low-power mode. In extremely low light or no-light environments (such as nighttime, with ambient light ≤10 lux): the long-wave infrared sensor 113 becomes the standby sensor, completely independent of light, and can clearly present the target outline through the perception of thermal radiation; the visible light sensor 111 is completely dormant, and the near-infrared sensor 112 serves as an auxiliary sensor. This dynamic scheduling strategy ensures that the system has the most suitable sensor as the sensing subject under any lighting conditions, guaranteeing all-weather, all-time monitoring capabilities while minimizing the overall power consumption of the system. It is especially suitable for long-term monitoring scenarios in the field where no one is on duty and where power is supplied by limited energy sources such as solar energy.

[0046] The following reference Figure 1 This paper describes the schemes for generating spatiotemporally registered multi-band images, generating foreground mask maps, and outputting structured multimodal data packets according to embodiments of the present invention. The spatiotemporally registered multi-band images are pixel-aligned images generated by the preprocessing unit 14, which uses the pre-calibrated intrinsic and extrinsic parameters of each sensor in the multi-band sensor group 11 to uniformly reproject the images acquired by the near-infrared sensor 112 and the long-wave infrared sensor 113 onto the image plane coordinate system of the visible light sensor 111.

[0047] Spatiotemporal registration refers to the process of geometrically aligning images acquired by sensors of different wavelengths. The preprocessing unit 14 uses the precisely measured parameters of each sensor (intrinsic parameters: such as focal length and distortion; extrinsic parameters: relative position and attitude) to uniformly transform the images captured by the near-infrared sensor 112 and the long-wave infrared sensor 113 onto the imaging plane of the visible light sensor 111 through digital reprojection technology, ultimately generating a set of multi-band images with perfectly aligned pixel positions, providing a precise data foundation for subsequent pixel-level feature fusion and target tracking.

[0048] In a specific example, each sensor is finely calibrated using a specially designed calibration board to obtain its intrinsic parameters (such as focal length and distortion coefficients) and extrinsic parameters (position and attitude rotation angles). After receiving the original image, the preprocessing unit 14 initiates a reprojection algorithm based on these parameters, mapping each pixel in the near-infrared and long-wave infrared images to the image coordinate system of the visible light sensor 111. In the structured multimodal data packet received by the backend multi-target tracking engine 20, the three images are pixel-aligned. At this point, the near-infrared reflection signal and thermal radiation signal of any target (such as the head of a deer) at a specific pixel position (x, y) in the visible light image will precisely correspond to the same coordinates (x, y) in the registered image, thus laying a solid foundation for subsequent feature-level fusion and stable tracking.

[0049] Before outputting the structured multimodal data packet, the preprocessing unit 14 executes a preliminary foreground detection algorithm to process the current frame image acquired by the guard sensor and generate a binarized foreground mask map; the foreground mask map indicates all potential areas in the image that have changed.

[0050] Before generating the final multimodal data packet, the preprocessing unit 14 performs a crucial preliminary processing step: foreground detection. The preprocessing unit 14 runs a foreground detection algorithm on image frames acquired by the monitored sensor (i.e., the master sensor dynamically selected based on ambient light). This algorithm effectively distinguishes between stationary backgrounds and moving foreground objects in the scene by comparing the current frame with an established background model. The result of image processing using the foreground detection algorithm is the generation of a binarized foreground mask. In this foreground mask, pixels identified as background by the algorithm are set to 0 (black), while all pixels identified as foreground (i.e., potential areas of change) are set to 1 (white). This foreground mask clearly marks the outlines and regions of all potentially interesting moving objects (such as deer, pedestrians, and vehicles) in the image with a minimal amount of data. Foreground detection provides crucial preprocessing results for the backend multi-object tracking engine 20. The multi-target tracking engine 20 no longer needs to perform calculations on the entire image. Instead, it can prioritize focusing on the foreground regions indicated by the foreground mask for subsequent target recognition, feature extraction, and data association. This greatly reduces the amount of computation and improves the processing efficiency and real-time performance of the entire tracking system.

[0051] In a specific example, when the long-wave infrared sensor 113 is used as a monitored sensor, the preprocessing unit 14 utilizes the raw thermal imaging data it acquires to run a lightweight frame difference or background subtraction algorithm to generate a foreground mask. Due to the characteristics of thermal imaging, there is usually a significant temperature difference between isothermal targets such as deer and their surrounding environment, making it easier and more reliable to separate moving targets from stationary backgrounds in thermal images. After algorithm processing, a binary foreground mask is generated: the moving area occupied by the moving target (such as a deer) is marked as white (pixel value 255), while the stationary background areas such as forests and grasslands are marked as black (pixel value 0). This foreground mask is incorporated into a structured multimodal data packet and transmitted to the backend multi-target tracking engine 20. The multi-target tracking engine 20 does not need to perform intensive calculations on the entire image and can immediately focus on the foreground region to perform target detection and feature extraction. This processing mechanism significantly reduces computational complexity and greatly improves the real-time processing capability of the system.

[0052] Optionally, the structured multimodal data packet also includes sensor metadata, which includes at least the exposure time, gain, and chip temperature of the long-wave infrared sensor 113 when each sensor in the multi-band sensor group 11 acquires images.

[0053] The structured multimodal data packet also includes sensor metadata. This sensor metadata is not redundant information, but rather important auxiliary data that provides key context for the backend multi-target tracking engine 20, thereby improving overall performance and reliability. The sensor metadata includes at least: the exposure time and gain of each sensor in the multi-band sensor group 11, used to evaluate image quality and provide a basis for algorithm optimization; and the chip temperature of the long-wave infrared sensor 113, used to perform accurate non-uniformity correction of thermal imaging data and compensate for thermal drift effects, which is a key parameter to ensure the quality of long-wave infrared images.

[0054] In a specific example, the structured multimodal data packet includes a machine-readable metadata header file (such as a JSON format file) used to record key parameters during image acquisition. These recorded parameters serve as sensor metadata. The sensor metadata provides crucial support for data processing and analysis by the backend multi-target tracking engine 20. For instance, the multi-target tracking engine 20 can perform non-uniformity correction (NUC) on thermal imaging data based on the chip temperature of the long-wave infrared sensor 113, effectively compensating for thermal drift effects and improving image quality and temperature measurement accuracy. All provided exposure and gain parameters offer a complete context for the raw data, supporting system performance retrospective analysis. For example, it can study the impact of different imaging parameters on target recognition rate, providing a basis for algorithm optimization and sensor control strategy adjustment.

[0055] When generating a structured multimodal data packet, the preprocessing unit 14 organizes the spatiotemporally registered multi-band image, foreground mask, ambient illumination value synchronized with the image, corresponding timestamp information, and sensor metadata according to a predetermined structural logic to form a complete data packet. This data packet typically contains the following data blocks: image data block (stores the registered multi-band core image data), foreground mask data block (stores the binarized foreground mask for quickly locating the region of interest), ambient data block (stores the synchronized ambient illumination value), timestamp (serves as a unique time identifier for all data), and sensor metadata block (containing imaging parameters such as exposure time and gain, and state parameters such as long-wave infrared chip temperature, providing key context for backend processing).

[0056] This invention provides a multi-target tracking method based on a passive multi-band sensor array, applied to a multi-target tracking engine. The multi-target tracking engine uses the aforementioned passive multi-band sensor array system to acquire structured multimodal data packets; such as... Figure 2 As shown, the method includes the following steps:

[0057] Step 201: Receive structured multimodal data packets. The structured multimodal data packets include at least the spatiotemporally registered multiband image, the foreground mask, the ambient illumination value synchronized with the image, and the corresponding timestamp information.

[0058] This step is the data preparation and input stage for the multi-target tracking engine. It provides a data foundation for the implementation of complex tracking algorithms by receiving a standardized data packet that has been preprocessed by the front end, has strict spatiotemporal consistency, and rich information dimensions.

[0059] The spatiotemporally registered multi-band images are geometrically aligned images from sensors in different bands (visible, near-infrared, and long-wave infrared). Spatiotemporal registration eliminates parallax, ensuring that the same target is in the exact same pixel position in images across different bands. This is a prerequisite for pixel-level or feature-level fusion to associate different attributes of the target, such as color, texture, and thermal radiation. The foreground mask is a binary image (usually white foreground and black for background) that marks all potential areas of movement or change in the image, providing attention regions. This allows the multi-target tracking engine to directly focus on the attention regions for target detection and feature extraction, greatly reducing computational complexity. The ambient illumination value synchronized with the image is a precise numerical value (usually in lux) representing the intensity of ambient light at the moment of image acquisition. Based on the ambient illumination value, the weights of each band feature during fusion can be dynamically adjusted (e.g., giving higher weights to visible light during the day and relying on thermal infrared features at night). The corresponding timestamp information records the absolute or relative time of data acquisition, providing a temporal context. This is a necessary parameter for calculating the target's velocity and acceleration, and for motion prediction, while also ensuring the continuity of multi-frame data over time, which is fundamental to maintaining the target's identity. The structured multimodal data packet also includes sensor metadata, which includes at least the exposure time, gain, and chip temperature of each sensor during image acquisition. This sensor metadata provides key parameters for image processing, quality enhancement, and algorithm optimization.

[0060] As a concrete example, the multi-target tracking engine receives and parses structured multimodal data packets, which include: registered visible light images, registered near-infrared images, registered long-wave infrared images, binary foreground masks, ambient illumination values, timestamps, and sensor metadata (such as exposure time and gain). The multi-target tracking engine loads the multi-band images into the GPU (Graphics Processing Unit) memory and loads parameters such as ambient illumination and timestamps into memory variables. In subsequent processing, the multi-target tracking engine will call the sensor metadata for image quality assessment and enhancement, adaptive adjustment of algorithm parameters, etc.

[0061] Step 202: Locate potential regions in the image based on the foreground mask map, and perform target detection within the potential regions to obtain multiple candidate target bounding boxes.

[0062] The multi-object tracking engine utilizes a binarized foreground mask to quickly locate motion regions (potential regions) in an image, significantly narrowing the scope of subsequent image processing. Target detection is performed within the potential regions indicated by the foreground mask, obtaining multiple initial detection boxes and their detection confidence scores. Redundant detection boxes are then removed using post-processing algorithms such as Non-Maximum Suppression (NMS), ultimately yielding multiple deduplicated candidate target bounding boxes. This avoids redundant calculations across the entire image, significantly improving real-time processing capabilities. During this process, image quality can be evaluated based on exposure time and gain parameters in the sensor metadata. If an excessively high sensor gain value is detected, it can be inferred that the image in that band has significant noise, allowing for adaptive adjustments to the confidence threshold or post-processing parameters in the detection algorithm, thereby improving detection reliability.

[0063] As a concrete example, connected component analysis is performed on the foreground mask image to find all connected white pixel regions (blobs), each corresponding to a potential moving target. Assuming three such regions are found, three coarse candidate regions can be identified. These three candidate regions, along with their corresponding three-band image patches, are then fed into a lightweight target detection neural network model (such as a variant of YOLO or Faster R-CNN). This network model can incorporate sensor metadata during processing; for example, it can use chip temperature information from a long-wave infrared sensor to perform non-uniformity correction (NUC) preprocessing on the thermal image to improve image quality and detection accuracy. The network model regresses multiple initial detection boxes and detection confidence scores. Subsequently, a non-maximum suppression (NMS) post-processing algorithm is used to merge and remove redundant detection boxes with high overlap, retaining the optimal detection box with the highest detection confidence. Assuming the processed output includes two high-confidence deer bounding boxes, this represents the final two candidate target bounding boxes.

[0064] Step 203: Extract a discriminative feature vector that integrates multi-band information for each candidate target bounding box.

[0065] After determining the candidate bounding boxes for the current frame, a discriminative feature vector is extracted for each candidate bounding box. The process of extracting the discriminative feature vector includes:

[0066] Visible light feature vectors, near-infrared feature vectors, and long-wave infrared feature vectors are extracted from the corresponding regions of the spatiotemporally registered visible light image, near-infrared image, and long-wave infrared image, respectively.

[0067] Based on the current ambient illuminance value, the confidence weights of the visible light feature vector, near-infrared feature vector, and long-wave infrared feature vector are dynamically calculated.

[0068] Based on confidence weights, the visible light feature vector, near-infrared feature vector, and long-wave infrared feature vector are weighted and fused to generate a discriminative feature vector.

[0069] Extracting discriminative feature vectors is a core step in achieving intelligent fusion and robust tracking, aiming to generate a unified feature representation that is insensitive to environmental changes and highly discriminative of target identity. In this process, sensor metadata provides crucial support for feature extraction and fusion: on the one hand, thermal imaging features can be calibrated based on the chip temperature of long-wave infrared sensors to eliminate the effects of thermal drift; on the other hand, the reliability of features in each band can be evaluated based on the exposure and gain parameters of each sensor, allowing for fine-tuning of fusion weights and achieving more refined adaptive fusion. This process mainly includes the following three levels:

[0070] 1. Parallel Extraction of Multi-Band Features. Feature vectors are extracted from corresponding regions (i.e., regions covered by the bounding box of the same candidate target) of the spatiotemporally registered visible light, near-infrared, and long-wave infrared images to leverage the complementarity of information from different bands: visible light images provide rich texture and color information, near-infrared images enhance structural information under low light conditions, and long-wave infrared images provide thermal radiation features. During feature extraction, sensor metadata can be used to preprocess the original images; for example, chip temperature parameters can be used to correct non-uniformity in long-wave infrared images, improving feature quality.

[0071] 2. Dynamic Calculation of Confidence Weights. Based on the current ambient illumination value, the confidence weights of each spectral band feature vector are dynamically calculated. For example, the visible light feature vector is assigned the highest weight under sufficient light, the weight of the near-infrared feature vector is increased in low-light environments, and the long-wave infrared feature vector is primarily relied upon at night or in complete darkness. This intelligently adjusts the level of confidence in each spectral band data according to ambient lighting conditions. In this process, sensor metadata (such as the exposure time and gain of each sensor) can be further incorporated for auxiliary decision-making: if a visible light sensor has extremely high gain even during the day, it indicates that its image quality has actually degraded, and its weight can be appropriately reduced to further improve the rationality of the weight allocation.

[0072] 3. Weighted Fusion and Generation. Based on the calculated confidence weights, the feature vectors of the three bands are weighted and fused to generate a unified and highly discriminative feature vector. This achieves the fusion of multi-source heterogeneous information into a compact and efficient feature representation, providing a direct basis for subsequent data association and target identity matching.

[0073] In a specific example, based on the region corresponding to the bounding box of the candidate target, corresponding image patches are cropped from visible light, near-infrared, and long-wave infrared images, respectively. Before feature extraction, the image patches are preprocessed using sensor metadata: for example, non-uniformity correction (NUC) is performed on the long-wave infrared image patches by calling the corresponding correction parameters according to the chip temperature of the long-wave infrared sensor; and image quality is evaluated based on the exposure and gain parameters of each sensor. The three cropped image patches are input into three feature extraction networks, respectively, to extract three feature vectors (each with a dimension of 256): visible light feature vector (feat_vis), near-infrared feature vector (feat_nir), and long-wave infrared feature vector (feat_lwir). Each feature vector is L2 normalized to make its magnitude 1, which facilitates the subsequent cosine distance calculation. The system queries the current ambient illuminance value (e.g., 125.7 lux) and calculates the confidence weights of each band feature vector according to a preset weight allocation strategy function (e.g., w_vis = 0.75, w_nir = 0.15, w_lwir = 0.10). Furthermore, the weights can be fine-tuned based on sensor metadata: if the visible light image gain is too high, w_vis is appropriately reduced; if the long-wave infrared image quality is significantly improved after NUC correction, w_lwir is appropriately increased. Then, a weighted feature fusion is performed: fused_feature = (w_vis * feat_vis) + (w_nir * feat_nir) + (w_lwir * feat_lwir). The fused feature vector fused_feature is then L2 normalized again to generate a final, fused, and normalized discriminative feature vector.

[0074] Step 204: Based on the discriminative feature vector, timestamp sequence and active trajectory set, perform data association. The timestamp sequence includes the timestamp information of the current frame and historical timestamps.

[0075] The data association process includes:

[0076] A multimodal association cost matrix is ​​constructed that integrates appearance similarity cost and motion consistency cost. The appearance similarity cost is determined based on the cosine distance between the discriminative feature vector corresponding to the bounding box of the candidate target in the current frame and the feature vector stored in each active trajectory in the active trajectory set. The motion consistency cost is determined based on the Mahalanobis distance between the expected position and the actual detection position of the bounding box of the candidate target in the current frame. The expected position of each active trajectory in the current frame is predicted based on the active trajectory set and historical timestamps.

[0077] Solve the multimodal association cost matrix and perform optimal matching between the candidate target bounding boxes and the active trajectory set in the current frame.

[0078] After determining the discriminative feature vectors of candidate target bounding boxes, data association is performed based on these vectors, timestamp sequences, and active trajectory sets. The data association process aims to optimally match the candidate target bounding boxes of the current frame with existing tracking trajectories to maintain the continuity of target identity. During this process, sensor metadata can be used to enhance the reliability of the association: for example, when calculating the appearance similarity cost, sensor status (such as lower reliability of image features under high gain) can be considered, dynamically adjusting the weight of appearance cost in the total cost to improve the robustness of the association decision. The data association process includes the following two key steps:

[0079] 1. Construct a multimodal association cost matrix

[0080] This step comprehensively evaluates motion consistency and appearance similarity, calculating the matching cost between each candidate target bounding box and each existing active trajectory. The core process is as follows: Figure 3 As shown:

[0081] (1) Calculation of motion consistency cost

[0082] The expected position of the current frame is inferred by using the Kalman filter of each active trajectory based on its historical state (position, velocity) and timestamp sequence; the Mahalanobis distance between the actual detected position of each candidate target bounding box and the predicted position of the active trajectory is calculated. The Mahalanobis distance takes into account the uncertainty of the predicted state (covariance matrix), and the smaller the value, the higher the consistency of the motion trajectory.

[0083] (2) Calculation of appearance similarity cost

[0084] The appearance similarity cost is determined by calculating the cosine distance between the discriminative feature vector of the candidate target bounding box in the current frame and the appearance feature models maintained by each trajectory in the active trajectory set. The appearance feature models are calculated from the historical appearance feature queues maintained by each trajectory (e.g., feature vectors from the most recent 20 frames), such as their average feature vector. A smaller cosine distance value indicates a higher appearance similarity between the candidate target bounding box and the trajectory. During the calculation, the reliability of the feature vector can be evaluated by incorporating sensor metadata (e.g., exposure, gain) used to generate it, and the cosine distance can be weighted with confidence to further improve the accuracy of appearance matching.

[0085] (3) The motion consistency cost and appearance similarity cost are fused in a weighted manner: Cost_i,j = α*M_i,j (motion consistency cost) + β*C_i,j (appearance similarity cost), where α and β are hyperparameters that balance the weights of the two types of costs and can be dynamically adjusted according to sensor metadata. For example, when the chip temperature of the long-wave infrared sensor is stable and the image quality is high, the weight of appearance cost (β) can be increased; when the exposure time of all sensors is short (which may introduce motion blur), the weight of motion cost (α) can be appropriately increased. Finally, a comprehensive cost matrix (multimodal association cost matrix) with the dimension of [number of candidate target bounding boxes × number of trajectories] is generated, and the matrix element values ​​represent the comprehensive mismatch between candidate target bounding box i and trajectory j.

[0086] 2. Solve for the multimodal correlation cost matrix

[0087] This step is based on solving the multimodal association cost matrix to find the optimal matching scheme that minimizes the overall association cost. For example, optimization algorithms (such as the Hungarian algorithm or the global nearest neighbor algorithm) are used to solve the multimodal association cost matrix and output the optimal matching relationship between the candidate target bounding box and the trajectory.

[0088] It should be noted that the multimodal association cost matrix is ​​a global matrix that integrates motion and appearance modalities. Its rows and columns represent all candidate bounding boxes and all active trajectories in the current frame, respectively. Each element in this matrix represents the total matching cost considering both motion and appearance modalities. Motion consistency cost and appearance similarity cost are also constructed for multiple candidate bounding boxes in the current frame. Motion consistency cost: based on Kalman filter prediction, calculates the Mahalanobis distance between the center point of each candidate bounding box and the predicted position of each trajectory. Appearance similarity cost: calculates the cosine distance between the discriminative feature vector of each candidate bounding box and the appearance feature model maintained by each trajectory. Throughout the construction and solution process, sensor metadata, as important contextual information, is used to enhance the accuracy of cost calculation and the reliability of association results.

[0089] In a specific example, the process of calculating the motion consistency cost is as follows: For each active trajectory tj, predict its position state Spred_j in the current frame T; for each candidate target bounding box di, calculate the Mahalanobis distance between its center point position and the predicted state Spred_j. The larger the Mahalanobis distance, the less likely the candidate target bounding box di originates from the active trajectory tj. The process of calculating the appearance similarity cost is as follows: For each active trajectory tj, it maintains an appearance feature queue (e.g., storing feature vectors from the most recent 20 frames), and calculates the trajectory average feature based on the appearance feature queue; for each candidate target bounding box di, calculate the cosine distance between its discriminative feature vector and the trajectory average feature, where a smaller cosine distance indicates greater similarity. During this process, the sensor metadata recorded when the discriminative feature vector was generated is queried. If the vector originates from a high-gain, long-exposure image, it may contain more noise, and the cosine distance value can be appropriately penalized and weighted to reduce its reliability. The two costs are weighted and merged to obtain the final multimodal association cost matrix. The multimodal association cost matrix is ​​solved to output the optimal matching relationship between the candidate target bounding box and the trajectory. The matching results will be recorded along with the metadata of the sensors used, providing data support for subsequent trajectory management, system performance analysis, and parameter optimization.

[0090] By introducing sensor metadata, the system constructs an adaptive closed loop from perception to optimization: non-uniformity correction (NUC) is performed using chip temperature, image quality is evaluated based on exposure and gain, and the fusion and association strategies are dynamically adjusted accordingly. This ability to "introspect" about its own state and environment enables a continuous and adaptive optimization process, significantly improving tracking accuracy and robustness. This highly adaptive intelligent mechanism represents a significant advantage in the current field of multimodal tracking. Step 205: In response to the completion of data association, based on the matching of the candidate target bounding box and the active trajectory set in the current frame, the active trajectory set is updated, and the current state of all active trajectories in the current frame is output. The current state includes the tracking bounding box, identity, and tracking confidence level corresponding to each active trajectory.

[0091] After data association is completed, the trajectory update and management phase begins. In this phase, the multi-target tracking engine performs the following operations based on the matching results:

[0092] Update: For a trajectory that successfully matches a candidate target bounding box, update its state using information from that bounding box (such as position and feature vectors) (e.g., update the Kalman filter and incorporate the new features into its appearance feature model). Simultaneously, update the trajectory's tracking confidence based on the quality of the match and the detection confidence of the candidate target bounding box.

[0093] Creation: For a candidate target bounding box that does not match any trajectory, if its detection confidence is higher than a predetermined threshold, a new trajectory is created for it and a new identity (ID) is assigned. The initial tracking confidence of the new trajectory can be directly inherited from the detection confidence of the candidate target bounding box, or determined according to a weighted combination of its detection confidence and a preset initial value.

[0094] Prediction and Labeling: For trajectories that do not match any candidate target bounding boxes, no updates are made, and their positions are predicted by their Kalman filters based on historical states. At the same time, the number of unmatches is incremented by one, and the tracking confidence is exponentially decayed according to a preset decay factor.

[0095] Through the above updates and management, the current status of all active trajectories in the current frame is output. The current status includes the tracking bounding box, identity identifier, and dynamically calculated tracking confidence for each trajectory. This tracking confidence comprehensively reflects the overall assessment of the reliability of the current status of each trajectory.

[0096] The multi-target tracking method provided in this invention adopts a passive multi-band sensing design, completely eliminating the dependence on active transmitters and solving the problems of applicability and concealment. A dynamic weight fusion strategy based on ambient illumination enables intelligent and flexible adjustment of the fusion mechanism, overcoming the poor environmental adaptability of traditional static fusion. By constructing a multimodal association cost matrix that fuses motion and appearance costs, the limitations of single-feature association are overcome, significantly improving the robustness and accuracy of data association in complex scenarios. Combined with foreground mask pre-screening and a multimodal discrimination mechanism, computational overhead is significantly reduced while ensuring tracking accuracy. The overall multi-target tracking scheme forms an intelligent closed loop of perception, decision-making, and execution, significantly improving the level of automated tracking and solving the problem of reliable tracking in variable outdoor environments.

[0097] The following describes the process of dynamically calculating the confidence weights of the visible light feature vector, near-infrared feature vector, and long-wave infrared feature vector based on ambient illuminance values. When calculating the confidence weights for each band of features: In response to an ambient illuminance value greater than the daytime threshold, a first weight is assigned to the visible light feature vector, and a second weight is assigned to the near-infrared and long-wave infrared feature vectors; in response to an ambient illuminance value greater than the nighttime threshold and less than or equal to the daytime threshold, a first weight is assigned to the near-infrared feature vector, a second weight to the long-wave infrared feature vector, and a third weight to the visible light feature vector; in response to an ambient illuminance value less than or equal to the nighttime threshold, a first weight is assigned to the long-wave infrared feature vector, a second weight to the near-infrared feature vector, and a third weight to the visible light feature vector; wherein, the first weight is greater than the second weight, and the second weight is greater than the third weight.

[0098] In the dynamic calculation of the confidence weights mentioned above, the importance of each sensor's feature vector in the fusion decision is dynamically allocated based on the reliability of the imaging quality of each sensor under different lighting conditions, thereby improving the system's perception robustness in all-weather scenarios. Specifically, the ambient illumination is divided into three intervals, and weight allocation principles are formulated for each interval:

[0099] In high-illuminance scenarios (daytime), when the ambient illuminance value is higher than the daytime threshold, the visible light sensor has the optimal imaging conditions and can provide the richest information in terms of color and texture. Therefore, the visible light feature vector is assigned the highest first weight, making it the dominant feature. The near-infrared feature vector and the long-wave infrared feature vector are used as auxiliary features and are assigned relatively low second weights. Furthermore, the values ​​of the second weights corresponding to the near-infrared feature vector and the long-wave infrared feature vector can be different, with the second weight corresponding to the near-infrared feature vector being slightly larger.

[0100] In low-to-medium illumination scenarios (dusk or dawn), when the ambient illuminance value falls between the nighttime and daytime thresholds, the signal-to-noise ratio of visible light images decreases, reducing reliability. Near-infrared sensors, due to their sensitivity to low light, offer relatively superior image quality; therefore, the near-infrared feature vector is given the highest weight. Long-wave infrared feature vectors provide stable thermal radiation information and are assigned the second weight. Visible light features, due to their lowest utility, are assigned the lowest weight, the third.

[0101] In extremely low-light scenarios (nighttime), when the ambient illuminance is less than or equal to the nighttime threshold, visible light becomes essentially ineffective, and near-infrared light has limited effectiveness. Long-wave infrared sensors, completely independent of ambient light, become the most reliable sensing source due to their thermal imaging characteristics; therefore, the long-wave infrared feature vector is assigned the first weight. The near-infrared feature vector, as a possible auxiliary, is assigned the second weight, while the visible light feature vector is assigned a third weight or is negligible. This relationship of first weight > second weight > third weight (the third weight is usually very small) ensures that, in any environment, the system always prioritizes the most reliable data source, achieving a smooth transition of the most reliable features, which is crucial for adaptive fusion.

[0102] In a specific example, for a scene with ample midday sunlight, since the visible light image quality is highest during the day and best distinguishes the individual characteristics of different deer (such as antler shape and body spots), the visible light feature is assigned the highest weight, such as 0.7, the near-infrared feature is assigned a weight of 0.2, and the long-wave infrared feature is assigned a weight of 0.1. For a moonless scene at night, since the long-wave infrared sensor image is the clearest and most stable, it is the main source of features. Near-infrared features can provide some contour information if there is faint starlight, while visible light features are completely useless. Therefore, the long-wave infrared feature is assigned the highest weight, such as 0.7, the near-infrared feature is assigned a weight of 0.3, and the visible light feature is assigned a weight of 0.

[0103] In an optional embodiment, before calculating the appearance similarity cost based on the discriminative feature vector, the method further includes: calculating the thermal radiation statistical features within the bounding box of each candidate target based on the long-wave infrared image, wherein the thermal radiation statistical features include the average temperature, temperature variance, and heat distribution histogram; and concatenating the thermal radiation statistical features with the discriminative feature vector to determine a comprehensive feature vector for calculating the appearance similarity cost.

[0104] Based on the weighted fusion of visible light feature vectors, near-infrared feature vectors, and long-wave infrared feature vectors to generate a discriminative feature vector, the thermal radiation statistical features within the bounding box of each candidate target in the long-wave infrared image can be specifically analyzed. Introducing thermal radiation statistical features provides a physical attribute-based discriminative dimension that complements visual features. Thermal radiation statistical features (such as average temperature, temperature variance, and heat distribution histogram) directly reflect the physical properties of the target (such as material or biological body temperature), providing crucial complementary information for distinguishing visually similar targets with different physical properties (e.g., a cold car turned off versus a hot car that has just stopped). This feature dimension is less affected by changes in ambient lighting or visual camouflage, thus effectively enhancing the ability to distinguish targets under conditions of target occlusion, similar appearance, and poor lighting. Adding thermal radiation statistical features to the discriminative feature vector to determine the comprehensive feature vector used to calculate the appearance similarity cost can improve the accuracy of appearance matching and the overall tracking robustness.

[0105] In a specific example, for a candidate target bounding box, the average temperature (e.g., 31.2°C), temperature variance, and heat distribution histogram of its internal pixels are calculated to determine its thermal radiation statistical characteristics. This feature effectively reflects the target's physical properties and internal state. For example, in deer herd monitoring, adult male deer exhibit subtle but distinguishable differences in head thermal radiation patterns compared to female deer due to the unique blood vessel distribution in their antlers. This thermal radiation statistical feature (e.g., a 4-dimensional vector) is concatenated with a discriminative feature vector (e.g., a 256-dimensional vector) to form a new comprehensive feature vector. This comprehensive feature vector introduces a biological feature discriminative dimension based on physical attributes into appearance similarity comparisons, enabling the capture of subtle pattern differences that are difficult to distinguish using traditional visual features (such as capturing subtle differences in thermal radiation patterns between the antler region of adult male deer and the head of female deer). Therefore, in the data association stage, even if multiple targets look highly similar in the visible or near-infrared bands, their unique thermal radiation characteristics are sufficient to clearly distinguish them in the feature space (e.g., two deer look extremely similar in visible or near-infrared images (e.g., the same body size and fur color), but their different body temperatures or unique thermal distribution patterns are sufficient to clearly distinguish them), thereby significantly reducing identity switching and improving tracking accuracy.

[0106] The following describes the process of updating the active trajectory set based on the matching of candidate target bounding boxes with the active trajectory set in the current frame. If a candidate target bounding box successfully matches a target active trajectory, the state of the target active trajectory is updated, and the identity of the target active trajectory is maintained; or, if a candidate target bounding box does not successfully match any active trajectory and the detection confidence of the candidate target bounding box is higher than a predetermined threshold, a new active trajectory is created for the candidate target bounding box and a new identity is assigned.

[0107] Based on the number of consecutive unmatched frames corresponding to each active trajectory, the active trajectories that are not matched by the current frame are judged and processed.

[0108] Output the current state of all active trajectories in the current frame.

[0109] After obtaining the data association results based on data association, based on Figure 4 The diagram illustrates several branching processes for updating the active trajectory set. For the trajectory update branch, if a candidate target bounding box in the current frame successfully matches an existing trajectory, the trajectory's state is updated using the new information from that candidate bounding box (such as position, velocity, and appearance features), maintaining its identity and ensuring that the same target is seen in the video. For the new trajectory creation branch, if a candidate target bounding box in the current frame fails to match any existing trajectory, the detection confidence of the candidate target bounding box is obtained. If the detection confidence is high (indicating it is likely a real target rather than noise), a new trajectory is created for it and a new unique identity is assigned to handle situations where a new target enters the field of view.

[0110] For active trajectories not matched in the current frame, the following operations are performed during status determination and processing:

[0111] In response to the first active trajectory failing to successfully match any candidate target bounding box within M consecutive frames, the first active trajectory is marked as temporarily lost, and position prediction continues.

[0112] If the second active trajectory fails to match any candidate target bounding box within N consecutive frames, the second active trajectory is deemed invalid and its identity is deleted. The value of N is greater than M.

[0113] Specifically, for the first active trajectory marked as temporarily lost, if a new candidate target bounding box appears near the predicted position in a subsequent frame, and the motion vector of the new candidate target bounding box is consistent with the predicted vector of the first active trajectory, it is determined that the first active trajectory has successfully matched the new candidate target bounding box and the identity of the first active trajectory is restored.

[0114] See also Figure 4For active trajectories that fail to match any candidate target bounding box in the current frame, the number of consecutive unmatched frames needs to be counted, and state management is performed based on this. If the first active trajectory fails to match any candidate target bounding box within M consecutive frames (including the current frame), the first active trajectory is marked as temporarily lost, and position prediction continues. If the second active trajectory fails to match any candidate target bounding box within N consecutive frames (including the current frame, and the value of N is greater than M, such as M being 5 and N being 30), the second active trajectory is determined to be invalid and its identity is deleted.

[0115] After completing all the above state updates, output the latest state of all active trajectories in the current frame, including the tracking bounding box, identity identifier, and dynamically calculated tracking confidence of the trajectory.

[0116] It's important to note that for the first active trajectory marked as temporarily lost, its prediction process continues. This involves continuously predicting the most likely location of the target in each subsequent frame, based on its motion state (position, velocity) before disappearing, and the uncertainty of that prediction. If a new candidate target bounding box appears near the predicted location in a subsequent frame (a dynamic region defined by metrics such as Mahalanobis distance, considering uncertainty), the recovery verification process is initiated. The core criterion for this process is the consistency of motion vectors: the motion vector of the new candidate target bounding box is calculated and compared with the predicted vector of the temporarily lost trajectory for similarity. This comparison covers two dimensions: direction (whether they are in the same direction or have a very small angle) and magnitude (whether the velocity magnitudes are similar). If both are highly consistent in direction and magnitude, the similarity verification is considered successful. When both spatial proximity and motion consistency are satisfied, the newly appearing candidate target is determined with extremely high confidence to be the previously lost target. The new candidate target bounding box is then associated with this temporarily lost trajectory. This trajectory will maintain and use its original identity identifier, ensuring the continuity of the target's identity. This ensures the continuity and stability of the identity link even after the target experiences brief obscuration or disappearance.

[0117] Among these, the dual verification (direction + amplitude) of motion vector consistency judgment is crucial to preventing mismatches, which is especially important in scenarios with densely intersecting targets and represents a subtle, non-obvious design. The trajectory's state machine (activated, temporarily lost, deleted) is linked to the resource scheduling of the entire system. For example, once a trajectory is marked as "temporarily lost," it can not only continue to be predicted but also trigger the system to increase its attention to that area (e.g., fine-tuning the scanning frequency of the monitoring sensor) or decrease its tracking confidence, reflecting a system-level management strategy.

[0118] The following example illustrates the management of multi-target tracking trajectories. This example simulates the tracking of a deer herd in a forest clearing, fully demonstrating the lifecycle of a trajectory from creation and recovery from temporary loss to deletion.

[0119] 1. New target emergence trajectory creation

[0120] If a newly appearing deer is detected, and its candidate target bounding box (detection box) does not match any existing trajectory, and the detection confidence (0.9) is higher than the predetermined creation threshold (0.7), then the new trajectory creation process is initiated.

[0121] Identity assignment: Assign a free, unique identity to the target, such as ID167.

[0122] Motion model initialization: Based on the position information of the detection box, a Kalman filter is initialized. The filter's state vector includes position, velocity, and acceleration, and is given a large initial error covariance matrix to represent uncertainty.

[0123] Appearance feature model initialization: Extract the discriminative feature vector corresponding to the detection box and store it in the appearance feature queue dedicated to ID167 as the initial sample of the appearance feature model.

[0124] Track status initialization: Set the status of track ID167 to active and initialize the relevant counters (e.g., set the unupdated counter to zero and the continuous matching counter to 1).

[0125] At this point, Little Deer ID167 was officially initialized as a tracking target, and continuous tracking began.

[0126] 2. The target disappears briefly and then reappears.

[0127] At frame T+10, a male deer (ID123) enters a dense thicket and goes undetected; its status is marked as occluded, and the counter is not updated. Based on the speed of ID123 before disappearing, its possible location in subsequent frames is predicted. As the number of predicted frames increases, the uncertainty of the predicted location becomes increasingly greater. At this point, updating its appearance feature queue is paused.

[0128] In frame T+15, an unmatched bounding box appears near the predicted location of ID123 (Mahaviran distance less than the threshold). The similarity (including direction and magnitude) between the bounding box's motion vector and the trajectory prediction vector is calculated. If they are highly similar, the association is considered successful. The trajectory ID123 state is restored to Active, and the unupdated counter is reset. The Kalman filter is updated using the new bounding box's location information, and its feature vector is added to the appearance feature queue. This achieves continuous preservation of the target's identity.

[0129] 3. Deletion of the target's trajectory after departure

[0130] After deer ID133 leaves the monitoring area, its trajectory fails to match for N consecutive frames (e.g., N=30). The unupdated counter accumulates to the deletion threshold, at which point trajectory ID133 is removed from the active list, and its identity is released to the available ID pool (optionally, its complete tracking history is stored in the database). This mechanism ensures effective recycling of system resources and long-term operational stability.

[0131] The above process enables automated and efficient management of the lifecycle of a large number of trajectories, effectively ensuring the accuracy and continuity of multi-target tracking in complex scenarios.

[0132] The passive multi-band sensor array system and multi-target tracking method of this invention have been described above. The core innovation of this invention lies in achieving a system-level technical effect of '1+1 is far greater than 2' through the deep collaborative design of hardware architecture, preprocessing pipeline, and tracking algorithm. This effect is not a simple stacking of sensors, processors, and algorithms, but rather stems from the organic integration and mutual enhancement of technical elements at each level.

[0133] Hardware provides the foundation for algorithms: the coaxial optical path and hardware synchronization triggering mechanism ensure the inherent spatiotemporal registration of multi-band images. This provides the physical possibility and accuracy guarantee for backend algorithms to perform pixel-level fusion and multi-source consistency verification (IoU calculation). Without this hardware foundation, all subsequent advanced algorithms will lose their foundation or their effectiveness will be greatly reduced.

[0134] Preprocessing reduces the burden on the algorithm and empowers it: The structured multimodal data packets generated by the preprocessing unit not only contain the registered image but also integrate rich contextual information such as foreground mask maps, synchronous illumination, and sensor metadata. This design moves a large amount of tedious preprocessing work (such as registration and foreground detection) forward, allowing the backend tracking engine to focus on high-level semantic decisions (such as data association and trajectory management), greatly improving the real-time performance of the entire system. At the same time, sensor metadata (such as chip temperature and gain) provides the algorithm with a sense of 'health status', enabling it to perform adaptive correction and weighting (such as non-uniformity correction NUC), a key quality loop that traditional solutions neglect.

[0135] The algorithm optimizes system resource allocation by employing reverse optimization: trajectory state management based on tracking confidence (activation, temporary loss, deletion) and dynamic scheduling of sensor operating modes (attendant / assistant) according to ambient illumination, forming an intelligent closed loop of perception-decision-execution. The system can intelligently allocate valuable computing and sensing resources based on tracking results and environmental conditions, thereby maintaining high-precision tracking performance with extremely low power consumption and achieving synergistic optimization of the traditionally contradictory goals of power consumption and performance.

[0136] In summary, this invention generates a synergistic gain effect through the above-mentioned multi-level, cross-module tight coupling and collaborative optimization, and its overall performance far exceeds that of traditional multi-sensor tracking systems with discrete designs.

[0137] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0138] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments described in this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0139] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0140] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0141] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatial relative descriptions used herein will be interpreted accordingly.

[0142] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A passive multi-band sensor array system for multi-target tracking, characterized in that, include: Multi-band sensor array, ambient light sensor, synchronization and control unit, and preprocessing unit; The multi-band sensor group includes at least one visible light sensor, one near-infrared sensor, and one long-wave infrared sensor. All sensors in the multi-band sensor group are integrated in a unified housing via a coaxial optical path and none of them contain an active transmitter. The synchronization and control unit is used to generate a synchronization signal and control the multi-band sensor group and the ambient light sensor to perform synchronized data acquisition. The synchronization and control unit also dynamically configures the standby sensor and auxiliary sensor in the multi-band sensor group based on the ambient illuminance data collected by the ambient illuminance sensor, and controls the auxiliary sensor to be in a low-power standby state by default. The logic for configuring the monitoring sensor and auxiliary sensor in the synchronization and control unit is as follows: When the ambient illuminance value is greater than the daily threshold, the visible light sensor is configured as the standby sensor, and the near-infrared sensor and the long-wave infrared sensor are configured as the auxiliary sensors. When the ambient illuminance value is greater than the overnight threshold and less than or equal to the daytime threshold, the near-infrared sensor is configured as the standby sensor, and the long-wave infrared sensor is configured as the auxiliary sensor. When the ambient illuminance value is less than or equal to the nighttime threshold, the long-wave infrared sensor is configured as the guard sensor. The preprocessing unit is used to receive raw image data from the multi-band sensor group, ambient illumination data from the ambient illumination sensor, and synchronization signal from the synchronization and control unit, and output a structured multimodal data packet to the back-end multi-target tracking engine, which processes the structured multimodal data packet and outputs the tracking status of multiple targets. Before outputting the structured multimodal data packet, the preprocessing unit performs a foreground detection algorithm on the current frame image acquired by the guard sensor to generate a binarized foreground mask image. The structured multimodal data packet includes: spatiotemporally registered multiband images, foreground mask images, ambient illumination values ​​synchronized with the images, and corresponding timestamp information.

2. The system according to claim 1, characterized in that, The spatiotemporally registered multi-band image is a pixel-aligned image generated by the preprocessing unit, which uses the pre-calibrated intrinsic and extrinsic parameters of each sensor in the multi-band sensor group to uniformly reproject the images acquired by the near-infrared sensor and the long-wave infrared sensor onto the image plane coordinate system of the visible light sensor.

3. The system according to claim 1 or 2, characterized in that, Before outputting the structured multimodal data packet, the preprocessing unit executes a preliminary foreground detection algorithm to process the current frame image acquired by the guard sensor and generate a binarized foreground mask image. The monitored sensor is the sensor in the multi-band sensor group that maintains image acquisition, and the foreground mask icon indicates all potential areas in the image that have changed.

4. The system according to claim 1, characterized in that, The structured multimodal data packet also includes sensor metadata, which includes at least the exposure time and gain of each sensor in the multi-band sensor group when acquiring images, as well as the chip temperature of the long-wave infrared sensor.

5. A multi-target tracking method based on a passive multi-band sensor array, applied to a multi-target tracking engine, characterized in that, The multi-target tracking engine uses the system described in any one of claims 1-4 to collect the structured multimodal data packets; the method includes: The structured multimodal data packet is received, which includes at least a spatiotemporally registered multiband image, a foreground mask, an ambient illumination value synchronized with the image, and corresponding timestamp information. Based on the foreground mask, potential regions in the image are located, and target detection is performed within the potential regions to obtain multiple candidate target bounding boxes; For each candidate target bounding box, a discriminative feature vector incorporating multi-band information is extracted. The extraction process of the discriminative feature vector includes: Visible light feature vectors, near-infrared feature vectors, and long-wave infrared feature vectors are extracted from the corresponding regions of the spatiotemporally registered visible light image, near-infrared image, and long-wave infrared image, respectively. Based on the current ambient illuminance value, the confidence weights of the visible light feature vector, the near-infrared feature vector, and the long-wave infrared feature vector are dynamically calculated. Based on the confidence weights, the visible light feature vector, the near-infrared feature vector, and the long-wave infrared feature vector are weighted and fused to generate the discriminative feature vector; Based on the discriminative feature vector, timestamp sequence, and active trajectory set, data association is performed, wherein the timestamp sequence includes timestamp information of the current frame and historical timestamps, and the data association process includes: A multimodal association cost matrix is ​​constructed that integrates appearance similarity cost and motion consistency cost. The appearance similarity cost is determined based on the cosine distance between the discriminative feature vector corresponding to the bounding box of the candidate target in the current frame and the feature vector stored in each active trajectory in the active trajectory set. The motion consistency cost is determined based on the Mahalanobis distance between the expected position and the actual detection position of the bounding box of the candidate target in the current frame. The expected position of each active trajectory in the current frame is predicted based on the active trajectory set and historical timestamps. Solve the multimodal association cost matrix, and perform optimal matching between the candidate target bounding boxes of the current frame and the active trajectory set; In response to the completion of data association, based on the matching status of the candidate target bounding box of the current frame with the active trajectory set, the active trajectory set is updated, and the current status of all active trajectories in the current frame is output. The current status includes the tracking bounding box, identity identifier and tracking confidence of each active trajectory.

6. The method according to claim 5, characterized in that, The step of dynamically calculating the confidence weights of the visible light feature vector, the near-infrared feature vector, and the long-wave infrared feature vector based on the current ambient illuminance value includes: In response to the ambient illuminance value being greater than the daytime threshold, a first weight is assigned to the visible light feature vector, and a second weight is assigned to the near-infrared feature vector and the long-wave infrared feature vector. In response to the ambient illuminance value being greater than the nighttime threshold and less than or equal to the daytime threshold, a first weight is assigned to the near-infrared feature vector, a second weight is assigned to the long-wave infrared feature vector, and a third weight is assigned to the visible light feature vector. In response to the ambient illuminance value being less than or equal to a nighttime threshold, a first weight is assigned to the long-wave infrared feature vector, a second weight is assigned to the near-infrared feature vector, and a third weight is assigned to the visible light feature vector; Wherein, the first weight is greater than the second weight, and the second weight is greater than the third weight.

7. The method according to claim 5, characterized in that, The method further includes: Based on the long-wave infrared image, the thermal radiation statistical characteristics within the bounding box of each candidate target are calculated. The thermal radiation statistical characteristics include average temperature, temperature variance, and heat distribution histogram. The statistical features of thermal radiation are concatenated with the discriminative feature vector to determine a comprehensive feature vector used to calculate the cost of appearance similarity.

8. The method according to claim 5, characterized in that, The process of updating the active trajectory set based on the matching status of the candidate target bounding boxes in the current frame with the active trajectory set includes: In response to a successful match between the candidate target bounding box and the target active trajectory, the state of the target active trajectory is updated, and the identity of the target active trajectory is maintained; or, in response to a failure to successfully match the candidate target bounding box with any active trajectory and the detection confidence of the candidate target bounding box being higher than a predetermined threshold, a new active trajectory is created for the candidate target bounding box and a new identity is assigned. Based on the number of consecutive unmatched frames corresponding to each active trajectory, the active trajectories that are not matched by the current frame are judged and processed. Output the current state of all active trajectories in the current frame.

9. The method according to claim 8, characterized in that, The process of determining and processing the status of active trajectories not matched by the current frame based on the number of consecutive unmatched frames corresponding to each active trajectory includes: In response to the first active trajectory failing to successfully match any candidate target bounding box within M consecutive frames, the first active trajectory is marked as temporarily lost, and position prediction continues. In response to the second active trajectory failing to successfully match any candidate target bounding box within N consecutive frames, the second active trajectory is determined to be invalid and its identity is deleted, where the value of N is greater than M. Specifically, for the first active trajectory marked as temporarily lost, if a new candidate target bounding box appears near the predicted position in a subsequent frame, and the motion vector of the new candidate target bounding box is consistent with the predicted vector of the first active trajectory, it is determined that the first active trajectory has successfully matched the new candidate target bounding box and the identity of the first active trajectory is restored.

Images