Passive multi-view dynamic wave band tracking method and system

Abstract

The invention relates to a passive multi-view dynamic wave band tracking method and system, the method is applied to a main processor, the main processor interacts with a sensor array and an environment illumination sensor, and the sensor array comprises a visible light sensor, a near-infrared sensor and a passive long-wave infrared sensor. Sensors in the sensor array do not comprise active emitters and are integrated in a unified shell through a coaxial light path, the visible light sensor and the near-infrared sensor share the same beam splitter prism, and the method comprises the steps of dynamically configuring an on-duty sensor and a low-power-consumption standby auxiliary sensor according to environmental illumination, awakening the auxiliary sensor when a target to be tracked is detected, and controlling the infrared sensor to be in a near-infrared state; the guarding sensor and the auxiliary sensor are controlled to carry out synchronous image acquisition on a to-be-tracked target to obtain multiple paths of images; processing the multiple paths of images to obtain a bounding box of the to-be-tracked target and the tracking confidence of the bounding box, and updating the tracking state according to the tracking confidence; and adjusting the working mode of the sensor array according to the tracking state and the latest environment illumination. According to the invention, intelligent resource scheduling can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a passive multi-view dynamic band tracking method and system. BACKGROUND

[0002] In the fields of photography, security monitoring, and autonomous driving, multi-band image acquisition devices (such as multi-view cameras) play a crucial role. These devices typically integrate sensors of different bands, such as visible light, near-infrared (NIR), and long-wave infrared (LWIR), to obtain clear and reliable image information under various complex lighting environments.

[0003] Existing multi-band image fusion schemes in the prior art rely heavily on continuous full-power operation of sensors to achieve high-quality fused images. However, such schemes have a fundamental limitation: they usually assume that the system can use active emission sources (such as near-infrared fill light, laser radar, etc.) without restriction or ignore the power consumption pressure brought by continuous operation of multiple sensors. For passive systems, i.e., systems where all sensors do not contain any active emitters and rely only on natural light or thermal radiation in the environment for passive imaging, the above traditional schemes face severe challenges. First, in terms of information acquisition, the passive constraint means that in extremely low-illumination environments (such as night, heavy fog, dense forest), the system cannot enhance the signal through active fill light, and the signal-to-noise ratio of visible light and near-infrared sensors drops sharply, making traditional detection and tracking algorithms easily fail. Second, in terms of energy consumption, passive systems are often deployed in edge devices that are extremely sensitive to power consumption (such as drones, portable monitoring devices, and long-term surveillance sensor nodes), and if the high-power sensors such as long-wave infrared are required to operate continuously at full power, it will severely restrict the endurance of the device, which is unacceptable in practical applications.

[0004] Therefore, under the strict technical constraint of no active emission source, the existing schemes that rely on continuous full-power operation of multiple sensors are not applicable because their power consumption cannot meet the endurance requirements of edge devices. At the same time, under the premise of no active fill light, reliable detection and tracking in low light become a unique technical challenge. The present application is proposed to overcome this unique challenge under this specific technical constraint. SUMMARY

[0005] The present application aims to provide a passive multi-view dynamic band tracking method and system to solve the deficiencies in the prior art. The technical problems to be solved by the present application are solved by the following technical solutions.

[0006] In a first aspect, the present application provides a passive multi-sensor dynamic waveband tracking method, applied to a main processor, the main processor interacting with a multi-waveband sensor array and an ambient illuminance sensor, the multi-waveband sensor array comprising a visible light sensor, a near-infrared sensor and a passive long-wave infrared sensor belonging to image sensors, all the image sensors not containing active emitters and integrated in a unified housing through coaxial optical paths, the visible light sensor sharing a first light splitting prism with the near-infrared sensor, the method comprising:

[0007] According to the ambient illuminance collected by the ambient illuminance sensor, a guard sensor in the multi-waveband sensor array and an auxiliary sensor in a low-power standby state are dynamically configured, the guard sensor being an image sensor adapted to the ambient illuminance;

[0008] In response to identifying a to-be-tracked target whose detection confidence meets a preset condition in a guard image collected by the guard sensor, a trigger signal is generated, and the auxiliary sensor is controlled to switch to a full-power working state according to the trigger signal;

[0009] The guard sensor and the auxiliary sensor are controlled to synchronously collect images of the to-be-tracked target at the same time, and multiple images associated with the to-be-tracked target are acquired;

[0010] The multiple images are processed to acquire a target bounding box of a current frame corresponding to the to-be-tracked target and a tracking confidence corresponding to the target bounding box, and a target tracking state of the to-be-tracked target is updated according to the tracking confidence corresponding to the target bounding box;

[0011] According to the target tracking state and the latest ambient illuminance collected by the ambient illuminance sensor, a working mode of the multi-waveband sensor array is dynamically adjusted.

[0012] In a second aspect, the present application provides a passive multi-sensor dynamic waveband tracking system, comprising a main processor, a multi-waveband sensor array and an ambient illuminance sensor interacting with the main processor, the multi-waveband sensor array comprising a visible light sensor, a near-infrared sensor and a passive long-wave infrared sensor belonging to image sensors, all the image sensors not containing active emitters and integrated in a unified housing through coaxial optical paths, the visible light sensor sharing a first light splitting prism with the near-infrared sensor;

[0013] The main processor comprises a central processing unit, an image signal processor and a tracking engine;

[0014] The ambient illuminance sensor collects ambient illuminance;

[0015] The central processing unit dynamically configures the on-duty sensors and the auxiliary sensors in low-power standby mode in the multi-band sensor array according to the ambient illuminance.

[0016] The image signal processor receives the monitored image acquired by the monitored sensor and provides the monitored image to the tracking engine;

[0017] The tracking engine responds to the identification of a target to be tracked in the monitored image whose detection confidence meets a preset condition, and reports it to the central processing unit. The central processing unit generates a trigger signal and controls the auxiliary sensor to switch to full power operation based on the trigger signal.

[0018] The central processing unit controls the monitoring sensor and the auxiliary sensor to synchronously acquire images of the target to be tracked at the same time.

[0019] The image signal processor receives multiple images associated with the target to be tracked and provides them to the tracking engine;

[0020] The tracking engine processes the multiple images, obtains the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box, and provides the tracking confidence to the central processing unit;

[0021] The central processing unit updates the target tracking status of the target to be tracked based on the tracking confidence level, and dynamically adjusts the operating mode of the multi-band sensor array based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor.

[0022] The technical solution provided by this invention dynamically configures a standby sensor and a low-power standby auxiliary sensor based on the ambient illuminance collected by the ambient illuminance sensor. During periods without a target to track, a single sensor remains on standby, avoiding the energy consumption caused by multiple sensors operating at full power, thus significantly reducing static power consumption. Based on a detection confidence trigger mechanism, when a target to be tracked is identified, the auxiliary sensor is quickly switched to full-power operation, achieving a seamless switch from low-power standby mode to high-performance tracking mode, ensuring continuous and stable tracking of dynamic targets. During the multi-sensor collaborative operation phase, each sensor is controlled to synchronously acquire images, and multiple images are fused to generate a target bounding box for the target to be tracked and calculate its tracking confidence. The target tracking status is updated based on the tracking confidence, and the operating mode of the multi-band sensor array is dynamically adjusted based on the latest target tracking status and the latest ambient illuminance, forming a real-time resource scheduling closed loop based on tracking status and environmental perception. This allows for adaptive resource allocation according to task requirements and environmental changes, achieving an optimal balance between power consumption, computing power, and performance while ensuring tracking reliability.

[0023] Furthermore, by selecting different sensor combinations based on ambient illuminance and controlling the auxiliary sensor to switch to full-power operation when a target is detected, the complementary advantages of sensors in different bands can be effectively integrated, overcoming the perception limitations of a single sensor under different environmental conditions. This ensures that robust and accurate tracking results can be output under any lighting and environmental conditions, improving all-weather operation capability and environmental adaptability. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the passive multi-view dynamic band tracking method according to an embodiment of the present invention;

[0025] Figure 2 Implementation process for determining the target bounding box in embodiments of the present invention Figure 1 ;

[0026] Figure 3 Implementation process for determining the target bounding box in embodiments of the present invention Figure 2 ;

[0027] Figure 4 This is a schematic diagram of a passive multi-view dynamic band tracking system according to an embodiment of the present invention. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] This invention provides a passive multi-view dynamic band tracking method applied to a main processor. The main processor interacts with a multi-band sensor array and an ambient light sensor. The multi-band sensor array includes visible light sensors, near-infrared sensors, and passive long-wave infrared sensors, which are image sensors. None of the image sensors contain active transmitters; they passively receive energy from the external environment for imaging. This passive design, without active transmitters, is the physical basis and prerequisite for this solution. The passive design enables completely passive, covert detection, significantly reducing power consumption and extending battery life. Furthermore, the passive design ensures that the performance of all image sensors is strictly dependent on the natural environment, making intelligent scheduling based on ambient light possible. Based on this, the main processor dynamically and flexibly configures the state of each image sensor (e.g., low-power standby state and full-power operating state) according to the ambient light, achieving on-demand allocation of computing and sensing resources. This dynamic configuration mechanism is an intelligent decision-making mechanism based on the aforementioned passive design, ultimately achieving the core performance goal of maximizing efficiency.

[0030] Visible light sensors, near-infrared sensors, and passive long-wave infrared sensors operate in the visible light, near-infrared, and long-wave infrared bands, respectively. To eliminate parallax and achieve pixel-level precise registration between multi-band images, supporting multimodal fusion and stable tracking, all image sensors are precisely integrated into a unified housing via a coaxial optical path. This ensures optical axis consistency, excellent mechanical structural rigidity, and improves long-term stability and reliability under complex environments such as vibration and temperature differences. Given that the visible light and near-infrared spectra are adjacent and have highly similar optical characteristics, a shared first beam-splitting prism design allows for efficient separation of the two signals using a single optical component. This is the most compact, efficient, and low-cost design solution for multi-band coaxial integration. An ambient light sensor, serving as a visual perception unit, is used to sense ambient light intensity in situ, providing crucial decision input for the dynamic scheduling and power consumption optimization of the multi-band sensor array.

[0031] It is important to note that the image sensor is housed inside a unified housing and connected to an optical window on the housing via a coaxial optical path for accurate imaging and protection; the ambient light sensor is placed, for example, on the surface of the housing to sense the true ambient light intensity in situ without obstruction.

[0032] like Figure 1 As shown, the passive multi-view dynamic band tracking method of this invention includes the following steps:

[0033] Step 101: Based on the ambient illuminance collected by the ambient illuminance sensor, dynamically configure the standby sensor and the auxiliary sensor in the multi-band sensor array that is in low-power standby mode. The standby sensor is an image sensor adapted to the ambient illuminance.

[0034] An ambient illuminance sensor is used to monitor ambient illuminance (i.e., ambient brightness) and reports the monitoring data to the main processor. Ambient illuminance sensors typically operate in a low-power periodic sampling mode, reducing power consumption while maintaining ambient illuminance levels for the main processor. Specifically, the sensor can report data promptly after monitoring ambient illuminance, sampling and reporting based on fixed low-power periods to provide the main processor with continuous ambient illuminance trend data; alternatively, it can report to the main processor when it detects a change in ambient illuminance exceeding a threshold, enabling the main processor to respond quickly to sudden environmental changes.

[0035] After receiving the ambient illuminance reported by the ambient illuminance sensor, the main processor dynamically configures the operating modes of the image sensors in the multi-band sensor array based on the range of ambient illuminance: image sensors adapted to the ambient illuminance are designated as standby sensors, and at least one of the remaining image sensors is designated as low-power standby auxiliary sensors. The standby sensor assumes the primary sensing responsibility, providing a high-quality image data stream for the current environment. The auxiliary sensors in low-power standby mode are backup sensors, used to quickly switch to full-power operation under the control of the main processor to cooperate with the standby sensor in image acquisition. By rationally configuring the standby and auxiliary sensors based on ambient illuminance, the multi-band sensor array can be transformed from a full-power static operating mode to an on-demand intelligent scheduling sensing system, thereby achieving a balance between performance, power consumption, and operational reliability.

[0036] Step 102: In response to identifying a target to be tracked in the monitored image acquired by the monitored sensor whose detection confidence meets the preset conditions, a trigger signal is generated, and the auxiliary sensor is controlled to switch to full power operation state according to the trigger signal.

[0037] The monitored sensor reports the acquired monitored images to the main processor. After acquiring the monitored images, the main processor identifies whether they contain a target to be tracked that meets preset confidence criteria. If no qualified target is identified, the current configuration of the multi-band sensor array is maintained. If a qualified target is identified, the main processor generates a trigger signal, which controls the auxiliary sensor to switch to full-power operation, enabling the auxiliary sensor to cooperate with the monitored sensor in image acquisition. By controlling the auxiliary sensor to switch to full-power operation, multi-band collaborative sensing capabilities can be built, improving target tracking accuracy and environmental adaptability.

[0038] Step 103: Control the monitoring sensor and auxiliary sensor to synchronously acquire images of the target to be tracked at the same time, and obtain multiple images associated with the target to be tracked.

[0039] After the auxiliary sensor switches to full-power operation, the main processor sends synchronization commands to both the monitoring sensor and the auxiliary sensor. This commands the monitoring sensor and the auxiliary sensor to simultaneously acquire images of the target being tracked at the same time, obtaining multiple images that are strictly aligned in time and associated with the target. Synchronization is the control method, and simultaneous acquisition is the result requirement, ensuring strict temporal alignment of the multiple images.

[0040] Since multiple image sensors in the embodiments of the present invention are integrated in a unified housing through a coaxial optical path, the multiple images provided by the monitoring sensor and the auxiliary sensor are naturally registered in space. Synchronous acquisition ensures strict temporal alignment of the multiple images, thereby providing highly reliable data that is completely consistent in the spatiotemporal dimension for subsequent multi-band image fusion.

[0041] Step 104: Process the multiple images to obtain the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence of the target bounding box, and update the target tracking status of the target to be tracked according to the tracking confidence of the target bounding box.

[0042] The multiple images of the target being tracked, provided collaboratively by the stationary sensor and auxiliary sensors, correspond to the same time point, eliminating motion blur and image misalignment caused by asynchronous acquisition times. The number of images corresponding to the multiple images is equal to the number of image sensors working collaboratively. For example, if one auxiliary sensor is switched to full power, combined with the original stationary sensor, a total of two images are acquired; or if two auxiliary sensors are switched to full power, combined with the original stationary sensor, a total of three images are acquired. Therefore, the acquired multiple images constitute a complete set of images, strictly registered in time and space, acquired and provided by multiple image sensors of different bands at the same synchronous acquisition time.

[0043] By processing multiple images associated with the target to be tracked, the target bounding box of the current frame corresponding to the target is obtained, as well as the tracking confidence score corresponding to the target bounding box of the current frame. The current frame refers to the moment when the processing is performed based on the latest synchronously acquired multiple images. This moment represents the latest perceived state of the target to be tracked. The current frame is a data frame concept containing image data and a corresponding timestamp, containing both image information and implicit time information. The target bounding box is a rectangular region defining the spatial range of the target to be tracked in the image, indicating the spatial range of the target in the two-dimensional image; the tracking confidence score is a quantitative evaluation value of the accuracy of the target bounding box. By obtaining the target bounding box, the target to be tracked can be accurately located; by obtaining the tracking confidence score of the target bounding box, the reliability of tracking can be evaluated.

[0044] After obtaining the tracking confidence score corresponding to the target bounding box, the target tracking state is updated based on this confidence score. The target tracking state is a comprehensive assessment of the target's existence, stability, and tracking quality. Essentially, it's a time-series-based, memory-based state that integrates historical information and current observations; it's not an isolated, moment-specific state. Updating the target tracking state based on the target bounding box's tracking confidence score allows for dynamic adjustment of subsequent tracking strategies, ensuring effective target tracking.

[0045] Step 105: Based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor, dynamically adjust the working mode of the multi-band sensor array.

[0046] The target tracking status indicates the detectability of the target (e.g., stable, occluded, lost), while the ambient illumination indicates the visibility conditions of the environment. Combining these two dimensions of information allows for dynamic optimization and scheduling of the operating modes of a multi-band sensor array. For example, it enables intelligent switching of the image sensor's operating modes and parameter configurations: under stable target and high-illuminance conditions, high-precision optical modes are prioritized; when occlusion or low-illuminance conditions occur, it automatically switches to anti-interference modes such as infrared; and in extreme scenarios where the target is lost, a wide-area survey mode is activated to reacquire the target.

[0047] The aforementioned intelligent perception and scheduling strategy significantly improves the target maintenance capability in complex environments, enabling stable tracking and continuous perception in all weather and all scenarios, and effectively ensuring the reliability of tracking tasks.

[0048] The above-described embodiments of the present invention dynamically configure the standby sensor and the low-power standby auxiliary sensor based on the ambient illuminance collected by the ambient illuminance sensor. During periods without a target to track, a single sensor remains on standby, avoiding the energy consumption caused by multiple sensors operating at full power, thus significantly reducing static power consumption. Based on a detection confidence trigger mechanism, when a target to be tracked is identified, the auxiliary sensor is quickly switched to full-power operation, achieving a seamless switch from low-power standby mode to high-performance tracking mode, ensuring continuous and stable tracking of dynamic targets. During the multi-sensor collaborative operation phase, each sensor is controlled to synchronously acquire images, and multiple images are fused to generate a target bounding box for the target to be tracked and calculate its tracking confidence. The target tracking status is updated based on the tracking confidence, and the operating mode of the multi-band sensor array is dynamically adjusted based on the latest target tracking status and the latest ambient illuminance, forming a real-time resource scheduling closed loop based on tracking status and environmental perception. This allows for adaptive resource allocation according to task requirements and environmental changes, achieving an optimal balance between power consumption, computing power, and performance while ensuring tracking reliability.

[0049] Furthermore, by selecting different sensor combinations based on ambient illuminance and controlling the auxiliary sensor to switch to full-power operation when a target is detected, the complementary advantages of sensors in different bands can be effectively integrated, overcoming the perception limitations of a single sensor under different environmental conditions. This ensures that robust and accurate tracking results can be output under any lighting and environmental conditions, improving all-weather operation capability and environmental adaptability.

[0050] The following describes the process of dynamically configuring the on-duty sensor and the low-power standby auxiliary sensor based on the ambient illuminance collected by the ambient illuminance sensor.

[0051] In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the daytime threshold, the visible light sensor is configured as the standby sensor, and the near-infrared sensor and the passive long-wave infrared sensor are configured as auxiliary sensors.

[0052] In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the nighttime threshold and less than or equal to the daytime threshold, the near-infrared sensor is configured as a standby sensor, and the passive long-wave infrared sensor is configured as an auxiliary sensor.

[0053] In response to an ambient illuminance level less than or equal to a nighttime threshold collected by an ambient illuminance sensor, the passive long-wave infrared sensor is configured as a standby sensor, and the near-infrared sensor is configured as an auxiliary sensor.

[0054] In this embodiment, based on the real-time monitoring of illumination conditions by the ambient illuminance sensor, and combined with preset daytime and nighttime thresholds (the two thresholds are scientific parameters calibrated and optimized based on the performance critical points of each image sensor), the primary and secondary roles of different image sensors are assigned to dynamically configure the standby sensors adapted to the ambient illuminance and the auxiliary sensors in a low-power standby state.

[0055] The characteristics of each image sensor in the multi-band sensor array are as follows: Visible light sensors offer high resolution, providing rich detail, texture, and color information, making them the core sensor for precise target identification and behavior analysis; near-infrared sensors possess certain night vision and penetration capabilities, are less affected by adverse weather conditions, and provide transitional observation support in low-light environments; passive long-wave infrared sensors are completely independent of ambient light, directly sensing the target's thermal radiation characteristics and achieving stable detection under extreme conditions such as complete darkness and strong obstruction. These three types of sensors form a perception closed loop through mode switching and fusion mechanisms, jointly ensuring the continuity of tracking tasks in complex environments.

[0056] When the ambient illuminance is higher than the daytime threshold, the current environment is in strong light mode. The visible light sensor is used as the main sensor (standby sensor), which can provide clear details and output high-quality images by relying on its high resolution advantage. At the same time, the near-infrared sensor and the passive long-wave infrared sensor are used as auxiliary sensors (backup sensors) to deal with sudden occlusion or backlight compensation.

[0057] When the ambient illuminance is greater than the nighttime threshold and less than or equal to the daytime threshold, the current environment is in a low-light mode (such as dusk or dawn). The near-infrared sensor is configured as a standby sensor to maintain observation by utilizing its night vision enhancement capability. In addition, the passive long-wave infrared sensor is used as an auxiliary sensor to cope with deeper light attenuation based on the passive long-wave infrared sensor.

[0058] When the ambient illuminance is less than or equal to the nighttime threshold, the current environment is in an extremely low light mode (such as night). The passive long-wave infrared sensor is used as the standby sensor to achieve no light source detection through thermal radiation detection, and the near-infrared sensor is used as an auxiliary sensor to meet the local supplementary lighting needs.

[0059] In the above implementation scheme, a high-efficiency architecture of single main sensor on standby + low-power standby auxiliary sensor is constructed based on intelligent scheduling of the on-duty sensor and auxiliary sensor based on ambient illumination. By dynamically selecting the image sensor adapted to the current environment as the on-duty sensor and combining it with the auxiliary sensor in low-power standby state, low static power consumption is ensured during periods without target tracking. At the same time, the complementary advantages of multiple sensors can be quickly activated when a target is detected or when tracking challenges are faced through the on-demand wake-up mechanism. This overcomes the perception limitations of a single sensor under different environmental conditions and ensures that robust and accurate tracking results can be output under any lighting conditions, achieving reliable tracking at all times and all-weather environmental adaptation.

[0060] In an optional embodiment, the scheme for identifying whether a target to be tracked exists in the monitored image and generating a trigger signal based on the identified target to be tracked includes the following steps:

[0061] Obtain candidate image regions that contain potential objects from the monitored image;

[0062] The candidate image region is input into a lightweight object detection neural network model, which is a pruned or quantized network model suitable for the sensor band.

[0063] Obtain the target score and category probability output by a lightweight object detection neural network model. The target score represents the probability that a candidate image region contains a potential object, and the category probability represents the probability that a potential object belongs to a specific category.

[0064] The detection confidence is determined by multiplying the target score by the maximum value of the category probability;

[0065] In response to a detection confidence level greater than a confidence threshold, a potential object is identified as a target to be tracked, and a trigger signal is generated.

[0066] For the monitored images reported by the monitored sensors, a series of preprocessing operations are performed on the original monitored images to improve image quality and provide a clearer, more stable, and less noisy input source for subsequent candidate region identification. After image preprocessing, the preprocessed image is processed based on a candidate region identification algorithm to accurately locate regions that may contain potential objects. Furthermore, by identifying candidate image regions, traditional full-image scanning can be replaced, significantly reducing subsequent computational load. After identifying and acquiring candidate image regions (e.g., through cropping), the candidate image regions are input into a lightweight target detection neural network model. This model is specifically trained and optimized for the current monitored sensor's wavelength bands (visible light / near-infrared / long-wave infrared), exhibiting high perceptual feature matching accuracy. The model undergoes deep optimization techniques such as pruning and quantization, aiming to complete the task with minimal computational overhead and power consumption.

[0067] After processing candidate image regions using a lightweight object detection neural network model, the model outputs two key metrics: target score and class probability. The target score, for example, is a value between 0 and 1, representing the probability that the candidate image region contains any potential object. The class probability, for example, is a vector representing the probability distribution of the potential object belonging to various specific categories (such as people, vehicles, and animals). The target score is then multiplied by the maximum value of the class probability to obtain a detection confidence score. This calculation logic integrates information from two dimensions: a region must be both highly likely to contain the target (requiring a high target score) and clearly belong to a known category (high maximum class probability) for a high detection confidence score.

[0068] After obtaining the detection confidence level based on the above logic, the relationship between the detection confidence level and the confidence level threshold is determined. If the detection confidence level is greater than the confidence level threshold, the potential object is identified as the target to be tracked, and a trigger signal is generated. If the detection confidence level is less than or equal to the confidence level threshold, the potential object is identified as an irrelevant target, and the image sensor remains in a low-power standby state. The trigger signal acts as a switch to control the auxiliary sensor to switch to full-power operation. Based on this signal, the multi-sensor collaborative acquisition and image fusion stage can begin.

[0069] In the above scheme, by using region extraction and a lightweight network, intelligent analysis is performed only on specific regions, which can reduce the amount of computation and reduce static power consumption. By adopting an event-driven mechanism based on detection confidence judgment, when the target to be tracked is identified, it can seamlessly switch from low-power monitoring to high-performance tracking, which can significantly improve the energy efficiency ratio.

[0070] The following examples, based on three typical lighting scenarios, illustrate the implementation process of identifying the target to be tracked based on detection confidence and triggering sensor collaborative operation.

[0071] Example 1: Daytime mode (ambient illuminance greater than daytime threshold)

[0072] The stationary sensor is a visible light sensor, while the auxiliary sensors include a near-infrared sensor and a passive long-wave infrared sensor. In this lighting scenario, the illumination is sufficient, and the target has rich color and texture features, but there may be interference such as shadows and strong light reflections.

[0073] The process for calculating the detection confidence level is as follows:

[0074] 1. Acquire image data

[0075] The visible light sensor acquires a frame of high-definition color image. The motion detection module senses the motion changes in the image and identifies a fast-moving region, such as (200, 300) with a size of 100x50 pixels, and then crops out this region.

[0076] 2. Input to a lightweight neural network model

[0077] The cropped image patches are resized and then fed into a lightweight YOLO model optimized for visible light images. This model is trained using a large number of daytime visible light images (people, vehicles), with 3-channel (RGB) images as input. The model is pruned and quantized to focus on the features of daytime targets (such as color, texture, and shape).

[0078] 3. Obtain the output of the lightweight neural network model

[0079] The lightweight neural network model processes the image patch and outputs the most reliable prediction box and its parameters: for example, the targetability score is 0.95 (the model determines that there is an object in the region), and the category probability [pedestrian: 0.75, car: 0.20, other: 0.05] (the model judges that the object is most likely to be a pedestrian).

[0080] 4. Calculate the detection confidence level

[0081] The maximum category probability is 0.75 (the probability of "pedestrian"), and the target score is 0.95. Therefore, the detection confidence is 0.95 * 0.75 = 0.7125.

[0082] Assuming a confidence threshold of 0.6, since 0.7125 > 0.6, a reliable target is detected, a trigger signal is generated, and the low-power standby near-infrared sensor and passive long-wave infrared sensor are activated, entering the three-sensor fusion tracking mode.

[0083] Example 2: Dusk / Dawn Mode (Ambient illumination greater than the nighttime threshold and less than or equal to the daytime threshold)

[0084] The primary sensor is a near-infrared sensor, while the auxiliary sensor is a passive long-wave infrared sensor. In this lighting scenario, the ambient light is weak, and the signal-to-noise ratio of the visible light sensor decreases. However, the near-infrared sensor can still effectively image, and its resistance to shadow interference and camouflage identification capabilities are superior.

[0085] The process for calculating the detection confidence level is as follows:

[0086] 1. Acquire image data

[0087] A near-infrared sensor acquires a grayscale image frame, and the motion detection module detects a suspicious area and crops out the image block.

[0088] 2. Input to a lightweight neural network model

[0089] The cropped grayscale image patches are input into a lightweight YOLO model optimized for the near-infrared band. This model, trained on a large amount of near-infrared image data, uses a single-channel grayscale image as input and can effectively capture and utilize the unique features of near-infrared imaging, such as the high reflectivity of vegetation, the infrared response characteristics of specific materials, and the ability to reduce shadow interference.

[0090] 3. Obtain the output of the lightweight neural network model

[0091] The lightweight neural network model processes the cropped grayscale image and outputs a targetability score of 0.88 and class probabilities [pedestrian: 0.82, vegetation: 0.15, others: 0.03] (the model eliminates most vegetation interference and confirms it is a pedestrian).

[0092] 4. Calculate the detection confidence level

[0093] The maximum category probability is 0.82 (the probability of "pedestrian"), and the target score is 0.88. Therefore, the detection confidence is 0.88 * 0.82 = 0.7216.

[0094] Assuming a confidence threshold of 0.6, since 0.7216 > 0.6, a reliable target is detected, a trigger signal is generated, and the passive long-wave infrared sensor in low-power standby mode is activated, entering the dual-sensor fusion tracking mode.

[0095] Example 3: Night Mode (Ambient Illumination less than or equal to Night Threshold)

[0096] The stationary sensor uses a passive long-wave infrared sensor, while the auxiliary sensor is a near-infrared sensor. In this scenario, the environment is completely dark with no visible light, requiring imaging and detection to rely on the difference in thermal radiation between the target and the background.

[0097] The process for calculating the detection confidence level is as follows:

[0098] 1. Acquire image data

[0099] A passive long-wave infrared sensor outputs a frame of thermal radiation image (usually represented as a grayscale image, where high pixel values ​​represent high radiation intensity / high temperature areas and low pixel values ​​represent low radiation intensity / low temperature areas); based on differences in radiation intensity (such as threshold segmentation or background subtraction), significant high radiation intensity areas (heat source areas) are identified in the image and cropped out.

[0100] 2. Input to a lightweight neural network model

[0101] The cropped image patch is input into a lightweight YOLO model optimized for thermal imaging. This model takes single-channel thermal radiation intensity data (or grayscale data) as input and is trained on a thermal imaging dataset. It can effectively identify the shape features of thermal targets (such as the thermal outline of the human body and the thermal signal of a vehicle engine) and is insensitive to visible light features such as color and texture.

[0102] 3. Obtain the output of the lightweight neural network model

[0103] The lightweight neural network model processes the input image patch and outputs a targetability score of 0.98 (hot targets stand out against a cold background and usually have a high targetability score), and outputs class probabilities [pedestrian: 0.90, vehicle: 0.08, animal: 0.02] (the model determines that it is a pedestrian based on the shape of the thermal contour and effectively distinguishes the hot animal).

[0104] 4. Calculate the detection confidence level

[0105] The maximum category probability is 0.90 (the probability of "pedestrian"), and the target score is 0.98. Therefore, the detection confidence is 0.98 * 0.90 = 0.882.

[0106] Assuming a confidence threshold of 0.6, since 0.882 > 0.6, a reliable target is detected, a trigger signal is generated, the low-power standby near-infrared sensor is activated, and the system enters the dual-sensor fusion tracking mode.

[0107] The three embodiments described above employ an adaptive sensor scheduling strategy linked to ambient illumination to dynamically configure the on-duty and auxiliary sensors. Combined with intelligent detection confidence calculation based on a lightweight band-specific model, they achieve maximum reduction of static power consumption during periods without a target and rapid activation of multimodal sensing capabilities after target detection, thus constructing a highly intelligent tracking system that can seamlessly adapt to all illumination conditions.

[0108] The process of processing multiple images and obtaining the target bounding box of the current frame corresponding to the target to be tracked is described below. This process includes the following steps:

[0109] Spatiotemporal registration is performed on the acquired multi-channel images;

[0110] Extract features from each image stream, and then fuse the extracted features to generate a comprehensive feature.

[0111] Based on comprehensive features, the target to be tracked is identified and located, and the target bounding box of the current frame corresponding to the target to be tracked is output.

[0112] After acquiring multiple images, each image is preprocessed to standardize the data and improve the accuracy and robustness of subsequent registration and fusion. Preprocessing operations include, but are not limited to, noise suppression, radiometric correction (intensity normalization), geometric distortion correction, and image enhancement. After preprocessing, spatiotemporal registration is performed on the multiple images.

[0113] The multi-band sensor array in this embodiment adopts a passive coaxial design and supports simultaneous data acquisition from multiple image sensors. The coaxial optical path design enables natural pixel-level or sub-pixel-level registration of the visible light sensor, near-infrared sensor, and passive long-wave infrared sensor in space, greatly simplifying the image registration algorithm; subsequent fine geometric distortion correction is all that is needed. The synchronous acquisition mechanism ensures that all band images strictly correspond to the same moment, fundamentally eliminating the time misalignment problem caused by target motion. Based on this, the spatiotemporal registration of multiple images includes the following two processes: 1. Spatial registration (pixel alignment): Based on the initial alignment provided by the coaxial optical path, non-reference images (such as near-infrared images) are resampled using pre-calibrated geometric transformation parameters (such as small translation or rotation matrices) to achieve sub-pixel-level precise alignment with the reference image (such as visible light images). This process simultaneously completes distortion correction of each sensor lens to eliminate optical errors. 2. Time registration (frame alignment): Relying on the synchronous acquisition mechanism to assign a unified timestamp to multiple images, the images from each sensor are already time-aligned during acquisition. Therefore, no software algorithm is needed for time interpolation or compensation; the synchronously acquired multiple images are processed as a unified data packet.

[0114] After registration, complementary features are extracted from each image stream and fused to generate a comprehensive feature. This comprehensive feature is then used to identify and locate the target to be tracked, determining the target bounding box for the current frame. During feature extraction, complementary features are extracted from each registered image stream. For example, rich texture and color features are extracted from visible light images; stable structural features in low-light environments are extracted from near-infrared images; and light-insensitive thermal radiation features are extracted from long-wave infrared images. During feature fusion, these heterogeneous features from different bands are fused at the feature level to generate a more comprehensive feature that is more robust to environmental interference (such as shadows, camouflage, and strong light). After obtaining the comprehensive feature, it is input into the recognition module. Based on the fused multi-band information, the target to be tracked is accurately identified and located, and the most accurate target bounding box in the current frame is output. This implementation scheme, by fusing complementary features provided by multi-band sensors to construct a comprehensive feature, significantly improves the accuracy and stability of target recognition and localization, ultimately outputting a more confident and accurate target bounding box. The process of determining the target bounding box is illustrated below with a specific example.

[0115] Example 1: Daytime mode (three-way fusion of visible light sensor, near-infrared sensor, and passive long-wave infrared sensor)

[0116] In bright sunlight during the day, targets may be in complex backgrounds such as shadows, strong light reflections, or dense vegetation. Therefore, a single sensor cannot reliably locate them, requiring the use of multiple sensors. The characteristics of the three types of sensors mentioned above are as follows: Visible light sensors: provide high-resolution images with rich color and texture details, but their performance is easily affected by lighting conditions, and they are prone to information loss or overexposure in shadow and bright light areas. Near-infrared sensors: have high responsiveness to vegetation and some man-made materials, have a certain ability to penetrate fog and haze, can mitigate the effects of shadows, and provide texture and structural information complementary to visible light sensors. Passive long-wave infrared sensors: directly reflect the thermal radiation characteristics of objects; their imaging is completely independent of ambient lighting conditions. Live targets (such as people and vehicles) usually have significant thermal contrast and stand out in thermal images, but their spatial resolution is usually low and they have fewer texture details.

[0117] Figure 2 This demonstrates the implementation process for determining the target bounding box. Figure 1 .

[0118] After completing spatiotemporal registration of multiple images from three sensors, feature extraction is performed on each image. Specifically: the visible light image (provided by the visible light sensor) is input to branch CNN (Convolutional Neural Network) 1 to extract feature map F1, which contains rich details such as edges, corners, colors, and textures; the near-infrared image (provided by the near-infrared sensor) is input to branch CNN2 with the same structure but independent weights to extract feature map F2. This feature map focuses on characterizing structural information less affected by shadows, vegetation distribution, and the material composition identification features of specific man-made materials (such as military camouflage), effectively complementing the visible light features; the infrared thermal image (provided by the passive long-wave infrared sensor) is input to branch CNN3 to extract feature map F3, which reflects the salience and approximate shape features of the thermal target area.

[0119] In the feature fusion stage, a weighted concatenation strategy is adopted to integrate multi-source features. Feature maps F1, F2, and F3 are stacked along the channel dimension (if each feature map has 256 channels, the stacked feature map has 768 channels), and the concatenated features are fused through a 1×1 convolutional layer. This operation is equivalent to learning an adaptive weighted combination mechanism, enabling the network to autonomously and dynamically adjust the contribution of each sensor feature according to the characteristics of the image region. For example, long-wave infrared features are given higher weight in the salient areas of hot targets, while visible light features are emphasized in areas where texture structure needs to be discerned.

[0120] Finally, a comprehensive feature map F is generated by fusion. fused This integrated feature map, as a representation of comprehensive features, combines the complementary advantages of multiple band sensors, carrying far more information and exhibiting greater robustness than image features provided by a single sensor. The integrated feature map will be fed into subsequent task networks to ultimately output a more accurate and robust target bounding box.

[0121] In the feature fusion stage, the fusion strategy based on channel concatenation and adaptive weighting is implemented as follows:

[0122] First, the feature maps F1 (visible light details), F2 (near-infrared structure), and F3 (thermal infrared saliency) from the three branches are concatenated along the channel dimension to obtain preliminary fused features. If each feature map has 256 channels, the concatenated feature map will have 768 channels. Then, a 1×1 convolutional layer is used to perform a fusion operation on this concatenated feature map, the calculation of which can be expressed as:

[0123] F fused =Conv1×1(Concat(F1,F2,F3))

[0124] The aforementioned 1×1 convolutional layer essentially learns a cross-modal adaptive weighting mechanism. Through training, it automatically learns a set of optimal fusion weights, thereby dynamically adjusting the contribution of each modality feature in different image regions. For example, in regions with obvious hot targets, feature map F3 can be assigned higher weights; in regions where fine texture or color discrimination is required, feature map F1 tends to be given higher importance; for targets that are partially occluded or in complex backgrounds (such as vegetation), the penetrability of feature map F2 and the saliency of feature map F3 can be combined to jointly assist in recognition.

[0125] Finally, the comprehensive feature map F obtained by feature fusion is... fused The data will be fed into a subsequent shared detection head network (typically composed of an RPN (Region Proposal Network) and a classification / regression subnetwork). Based on the rich multimodal features provided, this network outputs a series of detection results (including the presence confidence, class probability, and precise bounding box coordinates for each candidate target). These detection results are more accurate and robust than those from a single sensor source, providing a reliable basis for ultimately determining and outputting a high-precision target bounding box for the target to be tracked.

[0126] The core advantage of the above fusion method lies in its ability to integrate the complementary characteristics of different sensors: visible light provides high-resolution texture and color information; near-infrared provides anti-shadow capabilities and material composition identification features; and long-wave infrared provides target saliency based on thermal radiation and independent of illumination.

[0127] For example, when facing a target camouflaged in camouflage and concealed in the shadows of a forest, a visible light sensor may lose sight of the target due to the camouflage and shadows; a near-infrared sensor can indicate abnormal areas based on the difference in material reflection between vegetation and camouflage clothing; and a passive long-wave infrared sensor can clearly present the thermal outline of a human body. The proposed fusion method can comprehensively utilize all these clues to achieve accurate detection and localization of the target.

[0128] Example 2: Night Mode (Two-way fusion of near-infrared sensor and passive long-wave infrared sensor)

[0129] In scenarios with no or extremely low ambient light at night, this solution primarily relies on the collaborative detection of near-infrared sensors and passive long-wave infrared sensors. When no active lighting is available (this solution is a passive design), the near-infrared sensor can image using weak starlight or moonlight, but its signal-to-noise ratio is significantly lower than during the day, limiting feature extraction. The passive long-wave infrared sensor, on the other hand, relies entirely on the object's own thermal radiation for imaging, unaffected by ambient light. Against the cold background at night, heat sources such as people and vehicle engines stand out prominently.

[0130] Figure 3This demonstrates the implementation process for determining the target bounding box. Figure 2 .

[0131] After completing the spatiotemporal registration of the two images, feature extraction is performed separately: the near-infrared image (provided by the near-infrared sensor) is input to the CNN2 branch, and feature map F2 is extracted. This feature has a low signal-to-noise ratio and weak features at night, but may still contain some contour or texture information; the infrared thermal image (provided by the passive long-wave infrared sensor) is input to the CNN3 branch, and feature map F3 is extracted. Because it has a clear thermal target shape, it serves as the dominant feature source in this scene.

[0132] The feature fusion stage adopts an attention-based fusion strategy, which is suitable for the dominant mode (thermal infrared) + auxiliary mode (near infrared) architecture in this scenario.

[0133] 1. Generate attention map: Input the feature map F3 (dominant feature source) into a small network (such as a convolutional layer with a sigmoid activation function) to generate an attention map A with values ​​between 0 and 1, which is used to identify the importance of each position in the image. For example, high value regions (close to 1) usually correspond to significant hot targets (such as human bodies or vehicles), while low value regions (close to 0) represent the background.

[0134] 2. Feature weighting: Utilizing attention map A to weight the near-infrared features (features) of the auxiliary mode. Figure 4 Spatially weighted, the modulated features are obtained: F2_attended = A*F2. The physical meaning of this operation is to preserve or even enhance near-infrared features only in areas where thermal targets may exist (believing in the texture and contour details they may provide), while suppressing near-infrared features that may contain a lot of noise in the background areas.

[0135] 3. Feature Fusion: The modulated near-infrared features are added to and fused with the original long-wave infrared features to obtain the final comprehensive feature map: F fused =F3+F2_attended.

[0136] The core concept of this strategy can be summarized as: using thermal imaging as a guide to enhance low-light imaging. Highly reliable long-wave infrared features provide accurate prior knowledge of target location, guiding the network to focus on and extract valuable information from near-infrared features that are noisy but may contain important details.

[0137] 4. The enhanced integrated feature map is fed into the shared CNN detection head network for final confidence scoring, classification, and bounding box regression, outputting accurate object detection results (such as complete information including confidence, category, and object bounding box).

[0138] The resulting integrated feature map has two advantages: 1. It uses a clear and robust thermal target shape as the main body, ensuring the high detectability of the target in complex nighttime environments; 2. It incorporates enhanced low-light details (such as human posture and local texture of vehicles) inside the thermal target outline, which not only makes the tracking box more stable, but also improves the ability to distinguish dense or adjacent targets.

[0139] The following describes the scheme for calculating the tracking confidence corresponding to the target bounding box. Calculating the tracking confidence requires considering comprehensive features, the target bounding box in the current frame, and the independent bounding boxes associated with each target sensor. The target sensor is the image sensor that outputs the target to be tracked. The independent bounding box associated with each target sensor is determined based on the data independently provided by each target sensor. After multi-channel image registration and before feature extraction and fusion, the independent data provided by each sensor is analyzed to obtain the independent bounding boxes associated with each target sensor. The scheme for calculating the tracking confidence includes the following steps:

[0140] A response map is generated based on comprehensive features, and an appearance similarity score is generated based on the contrast between the global maximum response value in the response map and the response value of the surrounding background area.

[0141] The motion prediction error is determined based on the degree of overlap between the target bounding box in the current frame and the expected bounding box predicted based on the historical trajectory of the target to be tracked.

[0142] Calculate the overlap between each pair of independent bounding boxes associated with each target sensor, and calculate the average value to generate a multi-source consistency score;

[0143] The tracking confidence level corresponding to the target bounding box is determined based on the appearance similarity score, motion prediction error, and multi-source consistency score.

[0144] When calculating tracking confidence, it is necessary to comprehensively consider three major factors: appearance matching degree, motion coherence, and multi-sensor consistency, in order to generate a comprehensive, robust, and integrated tracking confidence.

[0145] The core idea of ​​appearance similarity scoring is to quantitatively evaluate the degree and significance of matching between the candidate region of the current frame and the historical appearance features of the target. The specific implementation process is as follows: First, a target template for matching in the current frame is generated based on the target's historical appearance model, and the search area is determined. The target's historical appearance model can be understood as a dedicated identity photo library built for the target, and the target template is a specific tool derived from the target's historical appearance model for real-time matching in the current frame. Then, within the search area, using a comprehensive feature map (a form of comprehensive feature representation), the similarity between the target template and the comprehensive feature map is calculated through correlation filtering or deep learning matching algorithms to generate a response map. The numerical distribution on this response map reflects the similarity score between each location and the target template. Analyzing the peak characteristics of the response map, the matching significance index can be obtained by calculating indicators such as the average peak correlation energy (APCE) or by directly regressing through a lightweight scoring network to quantify the contrast between the global maximum response value and its surrounding background area. Combining the global maximum response value (reflecting the degree of matching) and the contrast index (reflecting the significance of matching), an appearance similarity score is generated. The higher the score, the more reliable the matching result.

[0146] The core idea behind calculating motion prediction error is to compare the deviation between the predicted and actual positions based on the assumption of motion continuity, thereby evaluating the rationality of the motion trajectory. Specifically, the process involves using a motion predictor to predict the target's expected position (expected bounding box) in the current frame based on its motion state in previous frames. The overlap between the expected bounding box and the actual target bounding box in the current frame is then calculated, typically using the Intersection over Union (IoU) ratio. A lower IoU value indicates a larger motion prediction error. A larger error indicates a greater deviation between the actual and predicted motion of the target, which can be used to filter out abnormal tracking results.

[0147] The core idea behind calculating the multi-source consistency score is to utilize redundant information among sensors for cross-validation to assess the consistency of tracking results from different sensors. Specifically, the process involves obtaining the independent bounding boxes associated with each target sensor (such as visible light sensors, near-infrared sensors, and passive long-wave infrared sensors that output tracking results) and calculating the inter-unit overlap (IoU) between these independent bounding boxes. The average of all IoU values ​​yields the multi-source consistency score; a higher score indicates greater consistency in the judgments of all sensors regarding the target position, and stronger reliability of the tracking results.

[0148] After obtaining the appearance similarity score, motion prediction error, and multi-source consistency score, the parameters from these three different dimensions are fused into a unified, interpretable, and comprehensive tracking confidence score. The specific implementation is as follows: 1. Appropriately normalize the parameters of the three dimensions to eliminate differences in their units and numerical ranges. It is important to note that the motion prediction error (a negatively correlated parameter, the smaller the better) needs to be converted into a motion consistency score (a positively correlated parameter, the larger the better) to facilitate subsequent unified weighted fusion. 2. Assign different weights to each dimension according to the specific application scenario and perform weighted fusion (e.g., weighted averaging); for example, in high-speed motion scenarios, motion consistency can be given higher weight, while in complex backgrounds, appearance similarity can be given higher weight. 3. Generate a comprehensive tracking confidence score, which is a scalar value representing the overall evaluation of the reliability of the target bounding box in the current frame. A higher value indicates a more reliable tracking result. The process of determining the tracking confidence score is illustrated below with two examples.

[0149] Example 1: Night Mode (Near Infrared Sensor + Passive Long-Wave Infrared Sensor)

[0150] 1. Calculation of appearance similarity score

[0151] On the comprehensive feature map obtained after feature fusion, a tracking algorithm performs sliding window matching with the target template within the predicted search area to generate a response map. The maximum response value is, for example, Smax = 0.92; the mean peak correlation energy (APCE) of the response map is calculated as a contrast index, for example, APCE = 8.5. The appearance similarity score is calculated using the following formula: Appearance Similarity Score = Smax × ln(APCE + 1) = 0.92 × ln(9.5) ≈ 0.92 × 2.25 ≈ 2.07. A high Smax indicates that the algorithm found a candidate region with a high similarity to the target template within the search area. A high APCE indicates that the matching response of this candidate region is very prominent, with a significant contrast to the response values ​​of the surrounding background region, indicating a high degree of uniqueness in the matching result, rather than a generally high overall response level. The two combined constitute the appearance similarity score; the former reflects the degree of matching, and the latter reflects the significance of the matching result, effectively avoiding false detections and tracking errors. 2.07 is a very high appearance confidence score, indicating a high appearance matching degree and strong significance in the matching result.

[0152] 2. Calculation of Motion Consistency Score

[0153] For example, using a predictor such as a Kalman filter, the expected bounding box Bp of the current frame is predicted based on the target's historical motion state, and the intersection-over-union ratio (IoU) is calculated between this prediction and the target bounding box Bt (the actual tracking result) of the current frame. The measured IoU is 0.85. Therefore, the motion prediction error E = 1 - IoU = 0.15. In this example, the IoU value is high (0.85), and the error is small (0.15), indicating that the actual trajectory of the target matches the prediction very well, demonstrating high motion consistency. If the target suddenly makes a sharp turn or accelerates, the IoU will decrease, and the motion prediction error will increase. Converting the motion prediction error E into a positively correlated motion consistency score, the motion consistency score = 1 – E = IoU = 0.85. This high score indicates that the actual trajectory of the target matches the prediction model well, demonstrating high motion consistency.

[0154] 3. Calculation of Multi-Source Consistency Score

[0155] When multiple sensors are operating, each sensor outputting the target to be tracked is associated with an independent bounding box. The consistency of these independent bounding boxes is compared. For example, a near-infrared sensor independently tracks and obtains an independent bounding box B1, while a passive long-wave infrared sensor independently tracks and obtains an independent bounding box B2. The cross-union ratio (IoU) between the two is calculated. Since there are only two sensors, the average value is equal to this IoU value. For example, if the IoU is 0.95, the average value is also 0.95. A score of 0.95 is extremely high, indicating that both sensors have captured the same target and their judgments of its position are highly consistent, which is strong evidence of tracking reliability. If one sensor is briefly interfered with, its tracking result will drift, causing the IoU with the other sensor to decrease, thus lowering the average value.

[0156] 4. Weighted fusion yields the tracking confidence level.

[0157] The scores for the three categories above are normalized. Appearance similarity score normalization: 2.07 / 3 ≈ 0.69; Motion consistency score normalization: 0.85 (already within the range [0, 1], no further processing required); Multi-source consistency score normalization: 0.95 (already within the range [0, 1], no further processing required). If the three factors are given equal weights (α = β = γ = 1 / 3), then the tracking confidence score = (0.69 + 0.85 + 0.95) / 3 ≈ 0.83. This score (out of 1.0) is relatively high, indicating that the tracking result of the current frame has high reliability.

[0158] Example 2: Daytime mode (visible light sensor + near-infrared sensor + passive long-wave infrared sensor)

[0159] This embodiment demonstrates the process of calculating tracking confidence by fusing information from visible light sensors, near-infrared sensors, and passive long-wave infrared sensors in a daytime environment.

[0160] 1. Calculation of appearance similarity score

[0161] Based on the integrated feature map fused from the features provided by the three types of sensors and the target template, a response map is generated (see the above introduction, which will not be repeated here). Then, the maximum response value and the average peak correlation energy (APCE) are obtained. For example, the maximum response value Smax = 0.95, and the APCE as a contrast index = 9.2. According to the formula, the appearance similarity score is calculated as Smax × ln(APCE + 1) = 0.95 × ln(10.2) ≈ 0.95 × 2.32 ≈ 2.21. This value is relatively high, indicating that the appearance matching degree is high and the matching result is significant.

[0162] 2. Calculation of Motion Consistency Score

[0163] The expected bounding box Bp of the current frame is predicted using a Kalman filter, and the intersection-over-union (IoU) is calculated between it and the target bounding box Bt (the actual tracking result) of the current frame. The measured IoU is 0.88. Therefore, the motion prediction error E = 1 - IoU = 0.12. The motion prediction error is converted into a motion consistency score, where the motion consistency score = IoU = 0.88. This value indicates that the actual motion trajectory of the target is highly consistent with the prediction of the prediction model.

[0164] 3. Calculation of Multi-Source Consistency Score

[0165] Each of the three sensors independently acquires its own bounding box. The cross-union ratio (IoU) between all pairs of independent bounding boxes is calculated and averaged. For example, if the three IoUs are 0.90, 0.82, and 0.85, then the average IoU is (0.90 + 0.82 + 0.85) / 3 ≈ 0.86. This value indicates that the observation results of the three sensors have high consistency.

[0166] 4. Tracking confidence calculation

[0167] The scores were normalized as follows: appearance similarity score normalized: 2.21 / 3 ≈ 0.74; motion consistency score normalized: 0.88 (within the range [0, 1]); multi-source consistency score normalized: 0.86 (within the range [0, 1]). If equal-weighted fusion (each weight 1 / 3) is used, the tracking confidence value = (0.74 + 0.88 + 0.86) / 3 ≈ 0.83. This indicates that the tracking results are highly reliable under the current daytime mode and can maintain a stable tracking state.

[0168] The two embodiments described above, through multimodal feature fusion, integrate appearance, motion, and multi-source consistency scores, and generate tracking confidence through weighted fusion, effectively improve tracking reliability in complex environments.

[0169] This technical solution employs "synchronous acquisition" and "coaxial optical path." The "natural spatiotemporal registration" resulting from the combination of these two elements is the foundation for achieving high-performance multimodal fusion.

[0170] To achieve precise synchronous acquisition under the "on-demand wake-up" mechanism, this invention employs deep collaborative design at the hardware architecture level. The multi-band sensor array is not a simple stacking of sensors, but rather achieves hardware-level synchronization through a dedicated synchronization trigger circuit and exposure control logic. When the main processor issues a synchronization acquisition command, this circuit ensures that the start and end times of exposure for all woken sensors (both standby and auxiliary sensors) are strictly aligned, thereby eliminating time misalignment between multiple images caused by sequential sensor activation or differences in exposure times at the hardware level. This hardware-level synchronization guarantee is the physical foundation for subsequent high-precision multimodal spatiotemporal registration and fusion, and cannot be achieved solely through software scheduling.

[0171] Furthermore, the aforementioned "coaxial optical path" design of this invention is a necessary prerequisite and hardware guarantee for realizing the innovative algorithm feature of "multi-source consistency score". Since the visible light, near-infrared, and long-wave infrared sensors are integrated on the same optical axis through a beam splitter, they physically ensure that the observation perspective of the same scene is completely consistent at any time, achieving pixel-level spatial registration between images of different wavelengths. It is this inherent, hardware-guaranteed spatial consistency that makes the "independent bounding boxes" generated based on independent data from each sensor directly comparable. Based on this hardware, calculating the intersection-over-union (IoU) ratio between each pair of independent bounding boxes to generate the "multi-source consistency score" has clear physical meaning and extremely high reliability.

[0172] Therefore, the "coaxial optical path" hardware design of this invention and the subsequent "multi-source consistency score" algorithm feature constitute an inseparable collaborative innovation whole. The two are closely integrated and jointly ensure the accuracy and robustness of target tracking confidence assessment in complex environments.

[0173] The hardware foundation of the "coaxial optical path" and the algorithmic feedback of "multi-source consistency verification" form the first-layer enhancement loop. The coaxial optical path ensures pixel-level spatial registration between multi-band images, providing the physical possibility for the algorithm layer to perform accurate comparison (IoU calculation) of independent bounding boxes from different sensors; while the calculation result of the "multi-source consistency score" verifies the accuracy of hardware synchronization and registration, and provides a reliable confidence basis for fusion tracking. This closed loop, where the hardware provides the foundation for the algorithm and the algorithm provides verification and optimization guidance for the hardware, makes the overall reliability of the system far exceed the simple sum of improvements to a single hardware or algorithm. The following describes the scheme for updating the target tracking state based on the tracking confidence corresponding to the target bounding box, which includes the following steps:

[0174] If the tracking confidence level corresponding to the target bounding box is greater than the first threshold, and the historical tracking status of the target to be tracked has been in a stable state for more than the first duration or the historical tracking status has been in a preliminary confirmation state for more than the second duration, the target to be tracked is determined to have entered a high-confidence stable tracking state.

[0175] If the tracking confidence corresponding to the target bounding box is greater than the third threshold, less than or equal to the first threshold, and the historical tracking status of the target to be tracked has been in the preliminary confirmation state for more than the third duration, the target to be tracked is determined to have entered a stable state.

[0176] If the tracking confidence corresponding to the target bounding box is less than or equal to the second threshold, and the historical tracking state of the target to be tracked is a high-confidence stable tracking state or a stable state, the target to be tracked is determined to be downgraded to the preliminary confirmation state.

[0177] If the tracking confidence level corresponding to the target bounding box is less than or equal to the second threshold, and the historical tracking status of the target to be tracked is a preliminary confirmation status, the target to be tracked is determined to be in an ineffective state.

[0178] In response to the tracking confidence level corresponding to the target bounding box being greater than the second threshold and less than or equal to the third threshold, the historical tracking state of the target to be tracked is maintained.

[0179] Among them, the first threshold is greater than the third threshold, and the third threshold is greater than the second threshold.

[0180] When updating the target tracking status, it is necessary to comprehensively evaluate the instantaneous tracking quality of the current frame (represented by tracking confidence) and the persistence of historical tracking status to determine whether to upgrade, downgrade, or maintain the target tracking status. That is, changes in tracking status require continuous, high-quality evidence, while the judgment of tracking failure is relatively rapid and sensitive, thereby improving the reliability of tracking.

[0181] The scheme for updating the target tracking state defines multiple state levels and clarifies the transition conditions between them (first threshold > third threshold > second threshold).

[0182] The entry conditions for a high-confidence stable tracking state are the most stringent. It requires high instantaneous quality: the tracking confidence must be greater than the first threshold (the highest threshold); and historical persistence: the historical state must have remained stable for a period of time (first duration) or maintained preliminary confirmation for a period of time (second duration), with the second duration often longer than the first. This means that sufficient reliable evidence needs to be accumulated before entering the highest trust level. The preliminary confirmation state can be understood as: a potential target has been discovered, but the reliability of its tracking results is still questionable, requiring further observation for several frames (lasting for a period of time) to decide whether to convert it to a stable state or to determine failure due to poor performance. If the target is in a high-confidence stable tracking state, it indicates the highest level of confidence in the tracking results for that target.

[0183] For a steady state, the entry condition is the standard condition. Its instantaneous quality requirement is moderate: tracking confidence is greater than the third threshold (intermediate threshold) and less than or equal to the first threshold; historical persistence requirement: the historical state must be initially confirmed for a period of time (third duration). That is, the target's tracking quality is good and reliable, representing a normal tracking state.

[0184] For the preliminary confirmation state, the entry conditions are: 1. Degraded from a higher state (stable state or high-confidence stable tracking state) if the tracking confidence is less than or equal to the second threshold (minimum threshold); 2. Initial state: When a new target is first detected, it may enter this state first. The preliminary confirmation state is a buffer or warning state. If there is doubt about the tracking results of the target, but it has not been completely abandoned, this state provides a wait-and-see period. If the quality of subsequent frames improves, the target can recover; if it remains poor, it is considered invalid.

[0185] For the failure state, the entry conditions are the most sensitive. Instantaneous quality requirement: tracking confidence is less than or equal to the second threshold (minimum threshold); historical state requirement: the target is already in the preliminary confirmation buffer state. That is, in the preliminary confirmation state, if the tracking confidence of the current frame is found to be low, it is determined that the target has been lost, and the re-detection logic needs to be triggered to search for the target again in subsequent frames.

[0186] For state maintenance, the condition is: the tracking confidence is between the second threshold and the third threshold, that is, greater than the second threshold and less than or equal to the third threshold, the historical tracking state of the target to be tracked is maintained; this can be regarded as a gray area, where the tracking quality is neither good enough to upgrade nor bad enough to downgrade, so the current state is not changed and the changes in subsequent frames are observed.

[0187] In the above embodiments, the first threshold (highest) is set to ensure that the auxiliary sensor is only shut down when the appearance, motion, and multi-source consistency are all excellent; the second threshold (lowest) is set to be very sensitive to ensure that the trend of tracking failure can be quickly captured; the third threshold (middle) is a standard for "good performance". The introduction of multiple duration parameters (first, second, and third durations) requires that the state transitions be continuous, which effectively filters out instantaneous noise. This is an unconventional design and not something that would be easily conceived by those skilled in the art. This state machine control is linked to "passive" constraints. It is emphasized that it is precisely because passive sensors have low signal-to-noise ratios and high uncertainties in low light that such a complex and cautious state machine needs to be designed to manage risks and power consumption. This is a customized solution for a specific problem.

[0188] The target tracking state update scheme based on tracking confidence and time persistence described above achieves a balance between reliable tracking and sensitive loss judgment by setting high-confidence stable tracking state, stable state, preliminary confirmation state and failure state, and defining strict conversion conditions between them and three confidence thresholds and duration, thereby improving the tracking robustness and stability in complex scenarios.

[0189] The target tracking states in this application embodiment include a high-confidence stable tracking state, a stable state, a preliminary confirmation state, and a failure state. When dynamically adjusting the operating mode of the multi-band sensor array based on the target tracking state and the latest ambient illuminance collected by the ambient illuminance sensor, the following is included:

[0190] In response to the target tracking state being a high-confidence stable tracking state, the auxiliary sensor is controlled to enter a low-power standby state, while the tracking state of the monitoring sensor is maintained.

[0191] In response to the target tracking state being stable, the control sensor and some auxiliary sensors work together, with some auxiliary sensors determining the status based on the latest ambient light.

[0192] In response to the target tracking status being initially confirmed, the control sensor and all auxiliary sensors are fused together for detection, and the weight allocation of each sensor is dynamically optimized based on the latest ambient light.

[0193] In response to a target tracking failure, the auxiliary sensor is controlled to enter a low-power standby state, the monitor sensor is controlled to switch to a low frame rate wide-area survey mode, and in response to re-acquiring the target, the monitor sensor and the auxiliary sensor are dynamically adjusted based on the latest ambient light.

[0194] If the target tracking status is a high-confidence stable tracking status, that is, the target to be tracked has been firmly locked, it can enter the energy-saving precision mode. In this mode, the auxiliary sensor enters a low-power standby state, and the main sensor (the monitoring sensor, the image sensor most adapted to the current ambient light, such as a visible light sensor during the day) continues to maintain the working state. While ensuring the tracking effect, it achieves maximum energy saving and extends the device's battery life.

[0195] If the target tracking is stable, some uncertainties may arise during the tracking process (such as occasional partial occlusion or poor image quality). In such cases, the primary sensor and some auxiliary sensors need to work together, meaning a subset of auxiliary sensors are used to supplement information and effectively track the target. The selection of these auxiliary sensors is based on ambient light levels. For example, during the day when lighting is good, a passive long-wave infrared sensor is used to provide thermal feature information complementary to visible light, enhancing the ability to identify occluded or camouflaged targets. At dusk or in low light, the passive long-wave infrared sensor is used, leveraging its light-independent thermal imaging characteristics to provide crucial thermal radiation data support for the near-infrared sensor acting as the primary sensor, jointly overcoming uncertainties in low light. By appropriately allocating resources, uncertainties are mitigated, tracking stability is ensured, and resource waste from waking up all sensors is avoided.

[0196] If the target tracking status is in the preliminary confirmation state, such as the initial state in the above embodiments or the state determined due to degradation, then the full verification mode is entered. The main sensor and all auxiliary sensors perform fusion detection, collecting as much information as possible for cross-verification. During this process, the weight allocation of each sensor is dynamically optimized according to the latest ambient light. For example, the image information provided by the visible light sensor during the day has the highest weight and serves as the main judgment basis, while the information provided by the thermal imaging sensor at night has the highest weight. By scheduling various sensors, the complementary advantages of multi-band information (such as the detailed texture of visible light, the low-light structure of near-infrared light, and the thermal characteristics of infrared light) can be comprehensively utilized to verify the target's identity and status to the greatest extent, providing the most sufficient data basis for status updates.

[0197] If target tracking fails, the auxiliary sensor enters a low-power standby state to reduce power consumption, while the primary sensor switches to a low-frame-rate wide-area survey mode, meaning the primary sensor transitions from a close-monitoring mode to a surround-viewing mode. Lowering the refresh rate further saves energy, and wide-area surveying expands the field of view, making it easier to rediscover the target. Survey parameters are configured based on the latest ambient light levels to intelligently adjust them. For example, in high-light daytime conditions, the primary visible light sensor uses lower gain and higher resolution for wide-area scanning; at dusk, the primary near-infrared sensor automatically increases sensitivity (gain) and may employ specific image enhancement algorithms to optimize the probability of detection in low-light environments. If the target is recaptured based on wide-area surveying (whether it's a previous target or a new one), the primary sensor in the multi-band sensor array and the auxiliary sensor in low-power standby mode are dynamically reconfigured based on the latest ambient light dynamics. The above implementation scheme can achieve an optimal balance between tracking performance and energy saving by dynamically adjusting the working mode of the multi-band sensor array, significantly extending the equipment's battery life, while ensuring tracking reliability and environmental adaptability in different scenarios.

[0198] This invention provides a passive multi-view dynamic band tracking system, such as... ​ As shown, it includes a main processor 10, a multi-band sensor array 20 that interacts with the main processor 10, and an ambient light sensor 30. The multi-band sensor array 20 includes a visible light sensor 21, a near-infrared sensor 22, and a passive long-wave infrared sensor 23, which are image sensors. All image sensors do not contain active emitters and are integrated in a unified housing through a coaxial optical path. The visible light sensor 21 and the near-infrared sensor 22 share a first beam splitter.

[0199] The main processor 10 includes: a central processing unit 11, an image signal processor 12, and a tracking engine 13;

[0200] Ambient illuminance sensor 30 collects ambient illuminance data;

[0201] The central processing unit 11 dynamically configures the on-duty sensors and the auxiliary sensors in low-power standby mode in the multi-band sensor array 20 according to the ambient light level.

[0202] Image signal processor 12 receives the monitored images acquired by the monitored sensor and provides the monitored images to tracking engine 13;

[0203] The tracking engine 13 responds to the identification of a target to be tracked in the monitored image whose detection confidence meets the preset conditions, and reports it to the central processing unit 11. The central processing unit 11 generates a trigger signal and controls the auxiliary sensor to switch to full power operation based on the trigger signal 11.

[0204] The central processing unit 11 controls the monitoring sensor and the auxiliary sensor to synchronously acquire images of the target to be tracked at the same time.

[0205] Image signal processor 12 receives multiple images associated with the target to be tracked and provides them to tracking engine 13;

[0206] The tracking engine 13 processes multiple images, obtains the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box, and provides the tracking confidence to the central processing unit 11;

[0207] The central processing unit 11 updates the target tracking status of the target to be tracked based on the tracking confidence corresponding to the target bounding box, and dynamically adjusts the working mode of the multi-band sensor array based on the target tracking status and the latest ambient light collected by the ambient light sensor 30.

[0208] Optionally, in response to the ambient illuminance collected by the ambient illuminance sensor 30 being greater than the daytime threshold, the central processing unit 11 configures the visible light sensor 21 as a standby sensor, and configures the near-infrared sensor 22 and the passive long-wave infrared sensor 23 as auxiliary sensors.

[0209] In response to the ambient illuminance collected by the ambient illuminance sensor 30 being greater than the nighttime threshold and less than or equal to the daytime threshold, the central processing unit 11 configures the near-infrared sensor 22 as a guard sensor and the passive long-wave infrared sensor 23 as an auxiliary sensor.

[0210] In response to the ambient illuminance collected by the ambient illuminance sensor 30 being less than or equal to the nighttime threshold, the passive long-wave infrared sensor 23 is configured as a monitoring sensor, and the near-infrared sensor 22 is configured as an auxiliary sensor.

[0211] Optionally, the image signal processor 12 preprocesses the guarded image and provides it to the tracking engine 13; the tracking engine 13 acquires the preprocessed guarded image and acquires candidate image regions in the guarded image that include potential objects;

[0212] The tracking engine 13 inputs candidate image regions into a lightweight target detection neural network model, wherein the lightweight target detection neural network model is a pruned or quantized network model suitable for the conservatism sensor band;

[0213] The tracking engine 13 acquires the target score and category probability output by the lightweight object detection neural network model. The target score represents the probability that the candidate image region contains a potential object, and the category probability represents the probability that the potential object belongs to a specific category.

[0214] The tracking engine 13 multiplies the target score by the maximum value in the category probability to determine the detection confidence. In response to the detection confidence being greater than the confidence threshold, the potential object is identified as the target to be tracked. The tracking engine 13 reports the presence of the target to be tracked in the monitored image to the central processing unit 11, and the central processing unit 11 generates a trigger signal.

[0215] Optionally, the image signal processor 12 preprocesses the multiple images and provides them to the tracking engine 13, and the tracking engine 13 performs spatiotemporal registration on the preprocessed multiple images.

[0216] Tracking engine 13 extracts features from each image stream, fuses the extracted features to generate comprehensive features, and identifies and locates the target to be tracked based on the comprehensive features, and outputs the target bounding box of the current frame corresponding to the target to be tracked.

[0217] The tracking engine 13 calculates the tracking confidence corresponding to the target bounding box based on the comprehensive features, the target bounding box of the current frame, and the independent bounding boxes associated with each target sensor. The target sensor is the image sensor of the target to be tracked.

[0218] Optionally, the tracking engine 13 generates a response map based on the comprehensive features, and generates an appearance similarity score based on the contrast between the global maximum response value in the response map and the response value of the surrounding background area.

[0219] The tracking engine 13 determines the motion prediction error based on the degree of overlap between the target bounding box in the current frame and the expected bounding box predicted based on the historical trajectory of the target to be tracked.

[0220] Tracking engine 13 calculates the overlap between each pair of independent bounding boxes associated with each target sensor and calculates the average value to generate a multi-source consistency score;

[0221] The tracking engine 13 determines the tracking confidence level corresponding to the target bounding box based on the appearance similarity score, motion prediction error, and multi-source consistency score.

[0222] Optionally, in response to the tracking confidence corresponding to the target bounding box being greater than a first threshold, if the historical tracking state of the target to be tracked has been in a stable state for more than a first duration or the historical tracking state has been in a preliminary confirmation state for more than a second duration, the central processing unit 11 determines that the target to be tracked has entered a high-confidence stable tracking state.

[0223] In response to the tracking confidence level corresponding to the target bounding box being greater than the third threshold, less than or equal to the first threshold, and the historical tracking status of the target to be tracked being in the preliminary confirmation state for more than the third duration, the central processing unit 11 determines that the target to be tracked has entered a stable state.

[0224] In response to the tracking confidence level corresponding to the target bounding box being less than or equal to the second threshold, and the historical tracking state of the target to be tracked being a high-confidence stable tracking state or a stable state, the central processing unit 11 determines that the target to be tracked is downgraded to the preliminary confirmation state.

[0225] If the tracking confidence level corresponding to the target bounding box is less than or equal to the second threshold, and the historical tracking status of the target to be tracked is a preliminary confirmation status, the central processing unit 11 determines that the target to be tracked is in a failed state.

[0226] In response to the tracking confidence level corresponding to the target bounding box being greater than the second threshold and less than or equal to the third threshold, the central processing unit 11 determines that the target to be tracked maintains the historical tracking state.

[0227] Among them, the first threshold is greater than the third threshold, and the third threshold is greater than the second threshold.

[0228] Optionally, the target tracking status includes a high-confidence stable tracking status, a stable status, a preliminary confirmation status, and a failure status;

[0229] In response to the target tracking state being a high-confidence stable tracking state, the central processing unit 11 controls the auxiliary sensor to enter a low-power standby state and maintains the tracking state of the monitoring sensor;

[0230] In response to the target tracking state being stable, the central processing unit 11 controls the monitoring sensor and some auxiliary sensors to work together, with some auxiliary sensors determining the status based on the latest ambient light.

[0231] In response to the target tracking status being initially confirmed, the central processing unit 11 controls the monitoring sensor and all auxiliary sensors to perform fusion detection, and dynamically optimizes the weight allocation of each sensor according to the latest ambient light.

[0232] In response to the target tracking status being in a failed state, the central processing unit 11 controls the auxiliary sensor to enter a low-power standby state, controls the guard sensor to switch to a low frame rate wide-area survey mode, and, in response to re-acquiring the target, the central processing unit 11 dynamically adjusts the guard sensor and the auxiliary sensor based on the latest ambient light.

[0233] In this embodiment, the "preliminary confirmation state" is established to prevent frequent state transitions when the target is briefly obscured or the image quality fluctuates instantaneously, thereby avoiding system oscillations and improving stability. The "high confidence stable state" is established to implement the most aggressive energy-saving strategy and shut down all auxiliary sensors only when absolute certainty is achieved.

[0234] The aforementioned "on-demand wake-up" power control and "state machine"-driven resource scheduling form a reinforcing loop. The "on-demand wake-up" mechanism under "passive" constraints significantly reduces the system's static power consumption, while the four-state machine (high-confidence stable, stable, preliminary confirmation, failure) based on multi-dimensional confidence and time persistence intelligently manages resource allocation throughout the entire process, from "low-power monitoring" to "full-force fusion tracking" and then to "reacquisition." The state machine makes decisions based on tracking confidence and historical states, determining which sensors to wake up and what fusion weights to use, thereby achieving precise on-demand allocation of computing, sensing, and energy resources. This not only leads to a significant reduction in power consumption but also, by investing sufficient resources in key stages, improves the system's tracking accuracy and robustness under the same power budget.

[0235] This invention achieves a "synergistic gain" effect through the deep integration of hardware, algorithm, and power consumption management. The overall system performance is characterized by a significant reduction in static power consumption, while simultaneously improving tracking stability and accuracy in challenging scenarios such as target occlusion, sudden changes in lighting, and complex backgrounds, thanks to multimodal complementary fusion and intelligent resource scheduling. This simultaneous optimization of power consumption and performance—two traditionally contradictory metrics—is a prime example of the "1+1 greater than 2" technical effect of this invention, with overall performance far exceeding systems employing traditional continuous fusion schemes or simple sensor switching strategies.

[0236] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0237] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A passive multi-view dynamic band tracking method, applied to a main processor, wherein the main processor interacts with a multi-band sensor array and an ambient light sensor, the multi-band sensor array including a visible light sensor, a near-infrared sensor, and a passive long-wave infrared sensor belonging to image sensors, all image sensors do not contain active emitters and are integrated in a unified housing via a coaxial optical path, the visible light sensor and the near-infrared sensor share a first beam splitter, characterized in that, The method includes: Based on the ambient illuminance collected by the ambient illuminance sensor, the standby sensor and the auxiliary sensor in the multi-band sensor array are dynamically configured. The standby sensor is an image sensor adapted to the ambient illuminance. In response to identifying a target to be tracked in the monitored image acquired by the monitored sensor whose detection confidence meets a preset condition, a trigger signal is generated, and the auxiliary sensor is controlled to switch to full power operation state according to the trigger signal; The system controls the monitoring sensor and the auxiliary sensor to simultaneously acquire images of the target to be tracked at the same time, thereby obtaining multiple images associated with the target to be tracked. The multi-channel images are processed to obtain the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box, and the target tracking status of the target to be tracked is updated according to the tracking confidence corresponding to the target bounding box. The operating mode of the multi-band sensor array is dynamically adjusted based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor.

2. The method according to claim 1, characterized in that, The step of dynamically configuring the standby sensors and auxiliary sensors in a low-power standby state in the multi-band sensor array based on the ambient illuminance collected by the ambient illuminance sensor includes: In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the daytime threshold, the visible light sensor is configured as the standby sensor, and the near-infrared sensor and the passive long-wave infrared sensor are configured as the auxiliary sensors. In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the nighttime threshold and less than or equal to the daytime threshold, the near-infrared sensor is configured as the monitoring sensor, and the passive long-wave infrared sensor is configured as the auxiliary sensor. In response to the ambient illuminance collected by the ambient illuminance sensor being less than or equal to the nighttime threshold, the passive long-wave infrared sensor is configured as the monitoring sensor, and the near-infrared sensor is configured as the auxiliary sensor.

3. The method according to claim 1, characterized in that, The response to identifying a target to be tracked in the monitored image acquired by the monitored sensor whose detection confidence level meets a preset condition, and generating a trigger signal, includes: Obtain candidate image regions containing potential objects from the monitored image; The candidate image region is input into a lightweight target detection neural network model, wherein the lightweight target detection neural network model is a pruned or quantized network model suitable for the monitored sensor band; Obtain the target score and category probability output by the lightweight object detection neural network model, wherein the target score represents the probability that the candidate image region includes the potential object, and the category probability represents the probability that the potential object belongs to a specific category; The detection confidence is determined by multiplying the target score by the maximum value among the category probabilities. In response to the detection confidence level being greater than a confidence threshold, the potential object is determined to be a target to be tracked, and the trigger signal is generated.

4. The method according to claim 1, characterized in that, The step of processing the multi-channel images to obtain the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box includes: Spatiotemporal registration is performed on the acquired multi-channel images; Extract features from each image stream, and then fuse the extracted features to generate a comprehensive feature. Based on the comprehensive features, the target to be tracked is identified and located, and the target bounding box of the current frame corresponding to the target to be tracked is output; Based on the comprehensive features, the target bounding box of the current frame, and the independent bounding boxes associated with each target sensor, the tracking confidence corresponding to the target bounding box is calculated, and the target sensor is the image sensor that outputs the target to be tracked.

5. The method according to claim 4, characterized in that, The step of calculating the tracking confidence corresponding to the target bounding box based on the integrated features, the target bounding box of the current frame, and the independent bounding boxes associated with each target sensor includes: A response map is generated based on the comprehensive features, and an appearance similarity score is generated based on the contrast between the global maximum response value in the response map and the response value of the surrounding background area. The motion prediction error is determined based on the degree of overlap between the target bounding box in the current frame and the expected bounding box predicted based on the historical trajectory of the target to be tracked. Calculate the overlap between each pair of independent bounding boxes associated with each target sensor, and calculate the average value to generate a multi-source consistency score; The tracking confidence level corresponding to the target bounding box is determined based on the appearance similarity score, the motion prediction error, and the multi-source consistency score.

6. The method according to claim 1, characterized in that, The step of updating the target tracking state of the target to be tracked based on the tracking confidence corresponding to the target bounding box includes: In response to the tracking confidence level corresponding to the target bounding box being greater than a first threshold, if the historical tracking state of the target to be tracked has been in a stable state for more than a first duration or the historical tracking state has been in a preliminary confirmation state for more than a second duration, it is determined that the target to be tracked has entered a high-confidence stable tracking state. In response to the tracking confidence level corresponding to the target bounding box being greater than a third threshold, less than or equal to a first threshold, and the historical tracking status of the target to be tracked being in a preliminary confirmation state for more than a third duration, it is determined that the target to be tracked has entered a stable state. In response to the tracking confidence level corresponding to the target bounding box being less than or equal to the second threshold, and the historical tracking state of the target to be tracked being a high-confidence stable tracking state or a stable state, the target to be tracked is determined to be downgraded to a preliminary confirmation state. In response to the tracking confidence level corresponding to the target bounding box being less than or equal to the second threshold, and the historical tracking status of the target to be tracked being a preliminary confirmation status, the target to be tracked is determined to be in a failed state. In response to the tracking confidence level corresponding to the target bounding box being greater than a second threshold and less than or equal to a third threshold, the historical tracking state of the target to be tracked is maintained. Wherein, the first threshold is greater than the third threshold, and the third threshold is greater than the second threshold.

7. The method according to any one of claims 1 to 6, characterized in that, The target tracking status includes high-confidence stable tracking status, stable status, preliminary confirmation status, and failure status; The step of dynamically adjusting the operating mode of the multi-band sensor array based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor includes: In response to the target tracking state being a high-confidence stable tracking state, the auxiliary sensor is controlled to enter a low-power standby state, and the tracking state of the guard sensor is maintained; In response to the target tracking state being stable, the system controls the on-duty sensor and some auxiliary sensors to work together, with the auxiliary sensors determining the latest ambient illuminance based on the latest ambient light level. In response to the target tracking status being in the preliminary confirmation state, the system controls the monitoring sensor and all auxiliary sensors to perform fusion detection, and dynamically optimizes the weight allocation of each sensor based on the latest ambient light level. In response to the target tracking state being in a failed state, the auxiliary sensor is controlled to enter a low-power standby state, the guard sensor is switched to a low frame rate wide-area survey mode, and in response to re-acquiring the target, the guard sensor and the auxiliary sensor are dynamically adjusted based on the latest ambient light.

8. A passive multi-view dynamic band tracking system, comprising a main processor, a multi-band sensor array interacting with the main processor, and an ambient light sensor, wherein the multi-band sensor array includes a visible light sensor, a near-infrared sensor, and a passive long-wave infrared sensor belonging to image sensors; all image sensors do not contain active emitters and are integrated in a unified housing via a coaxial optical path; the visible light sensor and the near-infrared sensor share a first beam splitter, characterized in that: The main processor includes: a central processing unit, an image signal processor, and a tracking engine; The ambient illuminance sensor collects ambient illuminance; The central processing unit dynamically configures the on-duty sensors and the auxiliary sensors in low-power standby mode in the multi-band sensor array according to the ambient illuminance. The image signal processor receives the monitored image acquired by the monitored sensor and provides the monitored image to the tracking engine; The tracking engine responds to the identification of a target to be tracked in the monitored image whose detection confidence meets a preset condition, and reports it to the central processing unit. The central processing unit generates a trigger signal and controls the auxiliary sensor to switch to full power operation based on the trigger signal. The central processing unit controls the monitoring sensor and the auxiliary sensor to synchronously acquire images of the target to be tracked at the same time. The image signal processor receives multiple images associated with the target to be tracked and provides them to the tracking engine; The tracking engine processes the multiple images, obtains the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box, and provides the tracking confidence to the central processing unit; The central processing unit updates the target tracking status of the target to be tracked based on the tracking confidence level, and dynamically adjusts the operating mode of the multi-band sensor array based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor.

9. The system according to claim 8, characterized in that, The tracking engine performs spatiotemporal registration on the multi-channel images; The tracking engine extracts features from each image stream, fuses the extracted features to generate a comprehensive feature, and identifies and locates the target to be tracked based on the comprehensive feature, outputting the target bounding box of the current frame corresponding to the target to be tracked. The tracking engine generates a response map based on the comprehensive features, and generates an appearance similarity score based on the contrast between the global maximum response value in the response map and the response value of the surrounding background area. The tracking engine determines the motion prediction error based on the degree of overlap between the target bounding box in the current frame and the expected bounding box predicted based on the historical trajectory of the target to be tracked. The tracking engine calculates the overlap between each pair of independent bounding boxes associated with each target sensor and calculates the average value to generate a multi-source consistency score. The target sensor is the image sensor that outputs the target to be tracked. The tracking engine determines the tracking confidence level corresponding to the target bounding box based on the appearance similarity score, the motion prediction error, and the multi-source consistency score.

10. The system according to claim 8 or 9, characterized in that, The target tracking status includes high-confidence stable tracking status, stable status, preliminary confirmation status, and failure status; In response to the target tracking state being a high-confidence stable tracking state, the central processing unit controls the auxiliary sensor to enter a low-power standby state and maintains the tracking state of the monitoring sensor; In response to the target tracking state being stable, the central processing unit controls the monitoring sensor and some auxiliary sensors to work together, and the auxiliary sensors determine the operation based on the latest ambient light. In response to the target tracking status being in the preliminary confirmation state, the central processing unit controls the on-duty sensor and all auxiliary sensors to perform fusion detection, and dynamically optimizes the weight allocation of each sensor according to the latest ambient light. In response to the target tracking status being in a failed state, the central processing unit controls the auxiliary sensor to enter a low-power standby state, controls the guard sensor to switch to a low frame rate wide-area survey mode, and, after re-acquiring the target, the central processing unit dynamically adjusts the guard sensor and the auxiliary sensor based on the latest ambient light provided by the ambient light sensor.

Images