Passive multi-object dynamic band tracking method and system

Abstract

The present application relates to a kind of passive multi-object dynamic waveband tracking method and system, the method is applied to main processor, main processor and sensor array and ambient illuminance sensor interaction, sensor array includes visible light, near infrared and passive long-wave infrared sensor, the sensor in sensor array does not include active emitter and is integrated in uniform shell by coaxial light path, visible light and near infrared sensor share same light splitting prism, the method includes: according to ambient illuminance dynamic configuration value guard sensor and low-power standby auxiliary sensor, when detecting out target to be tracked, wake up auxiliary sensor, control guard and auxiliary sensor to target to be tracked carry out synchronous image acquisition, obtain multiple images;Process multiple images to obtain the bounding box of target to be tracked and the tracking confidence of bounding box, update tracking state according to tracking confidence;According to tracking state and latest ambient illuminance, adjust the working mode of sensor array.The present application can realize intelligent resource scheduling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a passive multi-view dynamic band tracking method and system. BACKGROUND

[0002] In the fields of photography, security monitoring, and autonomous driving, multi-band image acquisition devices (such as multi-view cameras) play a crucial role. These devices typically integrate sensors of different bands, such as visible light, near-infrared (NIR), and long-wave infrared (LWIR), to obtain clear and reliable image information under various complex lighting environments.

[0003] Existing multi-band image fusion schemes in the prior art rely heavily on continuous full-power operation of sensors to achieve high-quality fused images. However, such schemes have a fundamental limitation: they usually assume that the system can use active emission sources (such as near-infrared fill light, laser radar, etc.) without restriction or ignore the power consumption pressure brought by continuous operation of multiple sensors. For passive systems, i.e., systems where all sensors do not contain any active emitters and rely only on natural light or thermal radiation in the environment for passive imaging, the above traditional schemes face severe challenges. First, in terms of information acquisition, the passive constraint means that in extremely low-illumination environments (such as night, heavy fog, dense forest), the system cannot enhance the signal through active fill light, and the signal-to-noise ratio of visible light and near-infrared sensors drops sharply, making traditional detection and tracking algorithms easily fail. Second, in terms of energy consumption, passive systems are often deployed in edge devices that are extremely sensitive to power consumption (such as drones, portable monitoring devices, and long-term surveillance sensor nodes), and if the high-power sensors such as long-wave infrared are required to operate continuously at full power, it will severely restrict the endurance of the device, which is unacceptable in practical applications.

[0004] Therefore, under the strict technical constraint of no active emission source, the existing schemes that rely on continuous full-power operation of multiple sensors are not applicable because their power consumption cannot meet the endurance requirements of edge devices. At the same time, under the premise of no active fill light, reliable detection and tracking in low light become a unique technical challenge. The present application is proposed to overcome this unique challenge under this specific technical constraint. SUMMARY

[0005] The present application aims to provide a passive multi-view dynamic band tracking method and system to solve the deficiencies in the prior art. The technical problems to be solved by the present application are solved by the following technical solutions.

[0006] In a first aspect, the present application provides a passive multi-sensor dynamic waveband tracking method, applied to a main processor, the main processor interacting with a multi-waveband sensor array and an ambient illuminance sensor, the multi-waveband sensor array comprising a visible light sensor, a near-infrared sensor and a passive long-wave infrared sensor belonging to image sensors, all the image sensors not containing active emitters and integrated in a unified housing through coaxial optical paths, the visible light sensor sharing a first light splitting prism with the near-infrared sensor, the method comprising:

[0007] According to the ambient illuminance collected by the ambient illuminance sensor, a guard sensor in the multi-waveband sensor array and an auxiliary sensor in a low-power standby state are dynamically configured, the guard sensor being an image sensor adapted to the ambient illuminance;

[0008] In response to identifying a to-be-tracked target whose detection confidence meets a preset condition in a guard image collected by the guard sensor, a trigger signal is generated, and the auxiliary sensor is controlled to switch to a full-power working state according to the trigger signal;

[0009] The guard sensor and the auxiliary sensor are controlled to synchronously collect images of the to-be-tracked target at the same time, and multiple images associated with the to-be-tracked target are acquired;

[0010] The multiple images are processed to acquire a target bounding box of a current frame corresponding to the to-be-tracked target and a tracking confidence corresponding to the target bounding box, and a target tracking state of the to-be-tracked target is updated according to the tracking confidence corresponding to the target bounding box;

[0011] According to the target tracking state and the latest ambient illuminance collected by the ambient illuminance sensor, a working mode of the multi-waveband sensor array is dynamically adjusted.

[0012] In a second aspect, the present application provides a passive multi-sensor dynamic waveband tracking system, comprising a main processor, a multi-waveband sensor array and an ambient illuminance sensor interacting with the main processor, the multi-waveband sensor array comprising a visible light sensor, a near-infrared sensor and a passive long-wave infrared sensor belonging to image sensors, all the image sensors not containing active emitters and integrated in a unified housing through coaxial optical paths, the visible light sensor sharing a first light splitting prism with the near-infrared sensor;

[0013] The main processor comprises a central processing unit, an image signal processor and a tracking engine;

[0014] The ambient illuminance sensor collects ambient illuminance;

[0015] The central processor dynamically configures the value-keeping sensor and the auxiliary sensor in a low-power standby state in the multi-band sensor array according to the ambient illumination;

[0016] The image signal processor receives the value-keeping image collected by the value-keeping sensor and provides the value-keeping image to the tracking engine;

[0017] The tracking engine reports to the central processor in response to identifying a to-be-tracked target with a detection confidence satisfying a preset condition in the value-keeping image, the central processor generates a trigger signal, and controls the auxiliary sensor to switch to a full-power working state based on the trigger signal;

[0018] The central processor controls the value-keeping sensor and the auxiliary sensor to synchronously collect images of the to-be-tracked target at the same time;

[0019] The image signal processor receives the multi-channel image associated with the to-be-tracked target and provides the multi-channel image to the tracking engine;

[0020] The tracking engine processes the multi-channel image, obtains a target bounding box of a current frame corresponding to the to-be-tracked target and a tracking confidence corresponding to the target bounding box, and provides the tracking confidence to the central processor;

[0021] The central processor updates a target tracking state of the to-be-tracked target according to the tracking confidence, and dynamically adjusts a working mode of the multi-band sensor array according to the target tracking state and the latest ambient illumination collected by the ambient illumination sensor.

[0022] The technical scheme provided by the application dynamically configures the value-keeping sensor and the low-power standby auxiliary sensor according to the ambient illumination collected by the ambient illumination sensor, maintains the value-keeping state of a single sensor during a period without a to-be-tracked target, avoids energy consumption caused by full-power operation of multiple sensors, and thus greatly reduces static power consumption; based on a detection confidence trigger mechanism, the auxiliary sensor is quickly controlled to switch to a full-power working state when a to-be-tracked target is identified, seamless switching from a low-power value-keeping mode to a high-performance tracking mode is realized, and continuous and stable tracking of a dynamic target is ensured; during a multiple-sensor cooperative working phase, the sensors are controlled to synchronously collect images, the multi-channel images are fused, a target bounding box of the to-be-tracked target is generated and a tracking confidence thereof is calculated, the target tracking state is updated according to the tracking confidence, and the working mode of the multi-band sensor array is dynamically adjusted according to the latest target tracking state and the latest ambient illumination, forming a real-time resource scheduling closed loop based on the tracking state and environmental perception, resources can be adaptively allocated according to task requirements and environmental changes, and an optimal balance among power consumption, computing power and performance is realized on the premise of ensuring tracking reliability.

[0023] Further, different sensor combinations are selected based on ambient illuminance, and the auxiliary sensor is controlled to switch to a full-power working state when the tracking target is monitored, which can effectively fuse the complementary advantages of different waveband sensors, overcome the sensing limitations of a single sensor under different environmental conditions, ensure that robust and accurate tracking results can be output under any light and environmental conditions, and improve all-weather working capability and environmental adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A passive multi-eye dynamic waveband tracking method according to an embodiment of the present application is shown in the figure.

[0025] Figure 2 An implementation process for determining a target bounding box according to an embodiment of the present application is shown in the figure. Figure 1 ;

[0026] Figure 3 An implementation process for determining a target bounding box according to an embodiment of the present application is shown in the figure. Figure 2 ;

[0027] Figure 4 A passive multi-eye dynamic waveband tracking system according to an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0029] An embodiment of the present application provides a passive multi-eye dynamic waveband tracking method, applied to a main processor, which interacts with a multi-waveband sensor array and an ambient illuminance sensor. The multi-waveband sensor array includes a visible light sensor, a near-infrared sensor and a passive long-wave infrared sensor, which belong to image sensors. All the image sensors do not contain active emitters and only passively receive existing energy in the external environment for imaging. This passive design without active emitters is the physical basis and prerequisite for the implementation of the present scheme. The passive design can realize completely passive and concealed detection, significantly reduce power consumption to prolong the endurance time, and ensure that the performance of all image sensors is strictly subject to the natural environment, so that intelligent scheduling based on ambient illuminance becomes possible. On this basis, the main processor dynamically and flexibly configures the state (such as a low-power standby state and a full-power working state) of each image sensor according to the ambient illuminance, to realize on-demand allocation of computing and sensing resources. This dynamic configuration mechanism is an intelligent decision made on the basis of the above passive design, and ultimately achieves the core performance of maximum efficiency.

[0030] The visible light sensor, the near-infrared sensor and the passive long-wave infrared sensor are respectively in the visible light, the near-infrared and the long-wave infrared band, in order to eliminate parallax and realize pixel-level accurate registration among multi-band images, to support multi-modal fusion and stable tracking, all image sensors are precisely integrated in a unified housing through coaxial light paths, ensuring optical axis consistency and excellent mechanical structure rigidity, and improving long-term stability and reliability in complex environments such as vibration and temperature difference. Since the visible light and the near-infrared spectrum are adjacent and have highly similar optical characteristics, the visible light sensor and the near-infrared sensor share the first light splitting prism, and a single optical component can be used to efficiently separate the two signals, which is the most compact, efficient and low-cost reasonable design scheme for realizing multi-band coaxial integration. The ambient illuminance sensor as a visual perception unit is used to perceive the ambient light intensity in situ, and provides key decision input for dynamic scheduling and power consumption optimization of the multi-band sensor array.

[0031] It should be noted that the image sensor is placed inside the unified housing and connected to the optical window on the housing through coaxial light paths to accurately image and be protected; the ambient illuminance sensor is placed on the surface of the housing, for example, to perceive the real ambient light intensity in situ without obstruction.

[0032] As shown in Figure 1 The passive multi-view dynamic band tracking method of the embodiment of the present application includes the following steps:

[0033] Step 101, according to the ambient illuminance collected by the ambient illuminance sensor, dynamically configure the value guard sensor in the multi-band sensor array and the auxiliary sensor in the low-power standby state, and the value guard sensor is an image sensor adapted to the ambient illuminance.

[0034] The ambient illuminance sensor is used to monitor the ambient illuminance (i.e. ambient brightness) and report the monitoring data to the main processor. The ambient illuminance sensor usually works in a low-power periodic sampling mode, which reduces power consumption while maintaining the provision of ambient illuminance to the main processor. Among them, the ambient illuminance sensor can timely report data after monitoring the ambient illuminance, to sample and report based on fixed low-power period, to provide continuous ambient illuminance trend data for the main processor; or it can be reported to the main processor when it is monitored that the ambient illuminance changes more than a threshold value, so that the main processor can quickly respond to sudden environmental changes.

[0035] The main processor dynamically configures the working mode of the image sensors in the multi-band sensor array based on the interval range of the ambient illuminance after receiving the ambient illuminance reported by the ambient illuminance sensor: the image sensor adapted to the ambient illuminance is set as a guard sensor, and at least one image sensor in the remaining image sensors is set as a low-power standby auxiliary sensor. The guard sensor assumes the main sensing responsibility and provides high-quality image data stream in the current environment. The auxiliary sensor in the low-power standby state is a backup sensor and is used to quickly switch to a full-power working state based on the control of the main processor to cooperate with the guard sensor for image acquisition. By reasonably configuring the guard sensor and the auxiliary sensor based on the ambient illuminance, the multi-band sensor array can be converted from a full-power static working mode to an on-demand intelligent scheduling sensing system, thereby achieving a balance between performance, power consumption and working reliability.

[0036] Step 102, in response to identifying a to-be-tracked target with detection confidence satisfying a preset condition in the guard image collected by the guard sensor, a trigger signal is generated, and the auxiliary sensor is controlled to switch to a full-power working state according to the trigger signal.

[0037] The guard sensor reports the collected guard image to the main processor, and the main processor identifies the guard image after obtaining the guard image, judges whether the guard image contains a to-be-tracked target with detection confidence satisfying a preset condition. If no qualified target is identified, the current configuration state of the multi-band sensor array is maintained; if a qualified target is identified, the main processor generates a trigger signal and controls the auxiliary sensor to switch to a full-power working state based on the trigger signal, so that the auxiliary sensor and the guard sensor cooperate to perform image acquisition. By controlling the auxiliary sensor to switch to a full-power working state, a multi-band cooperative sensing capability can be constructed, and target tracking accuracy and environmental adaptability can be improved.

[0038] Step 103, controlling the guard sensor and the auxiliary sensor to synchronously collect images of the to-be-tracked target at the same time, and obtaining multi-channel images associated with the to-be-tracked target.

[0039] After the auxiliary sensor is controlled to switch to a full-power working state, the main processor sends a synchronization instruction to the guard sensor and the auxiliary sensor to control the guard sensor and the auxiliary sensor to synchronously collect images of the to-be-tracked target at the same time, so as to obtain multi-channel images associated with the to-be-tracked target which are strictly aligned in time. Synchronization is a control means, and the same time is a result requirement, which ensures that the multi-channel images are strictly aligned in time.

[0040] Since the multiple image sensors in the embodiment of the application are integrated in a unified shell through the coaxial light path, the multiple images provided by the guard sensor and the auxiliary sensor in cooperation are naturally registered in space, and the synchronous acquisition ensures strict alignment of the multiple images in time, thereby providing completely consistent and high-reliability data in time and space dimensions for subsequent multi-band image fusion.

[0041] In step 104, the multiple images are processed to obtain a target bounding box corresponding to a current frame of the target to be tracked and a tracking confidence corresponding to the target bounding box, and the target tracking state of the target to be tracked is updated according to the tracking confidence corresponding to the target bounding box.

[0042] The multiple images associated with the target to be tracked provided by the guard sensor and the auxiliary sensor in cooperation correspond to the same time, eliminating motion blur and image misplacement caused by different synchronous acquisition times; the number of images corresponding to the multiple images is equal to the number of image sensors working in cooperation. For example, if the number of auxiliary sensors switched to the full-power working state is 1, combined with the original 1 guard sensor, a total of 2 images are obtained; for another example, if the number of auxiliary sensors switched to the full-power working state is 2, combined with the original 1 guard sensor, a total of 3 images are obtained. Therefore, the obtained multiple images are: a complete set of images strictly registered in time and space, respectively collected and provided by multiple image sensors of different bands at the same synchronous acquisition time.

[0043] By processing the multiple images associated with the target to be tracked, a target bounding box corresponding to a current frame of the target to be tracked is obtained, and a tracking confidence corresponding to the target bounding box of the current frame is obtained. The current frame refers to the time when the multiple images based on the latest synchronous acquisition are processed, which represents the latest perception state of the target to be tracked. The current frame is a data frame concept containing image data and corresponding time stamp, which contains image information and implicitly contains time information. The target bounding box is a rectangular region defining the spatial range of the target to be tracked in the image, indicating the spatial range of the target to be tracked in the two-dimensional image; the tracking confidence is a quantitative evaluation value of the accuracy of the target bounding box. By obtaining the target bounding box, the target to be tracked can be accurately positioned, and by obtaining the tracking confidence of the target bounding box, the reliability of tracking can be evaluated.

[0044] After obtaining the tracking confidence corresponding to the target bounding box, the target tracking state of the target to be tracked is updated according to the tracking confidence corresponding to the target bounding box. The target tracking state is a comprehensive judgment of the existence, stability and tracking quality of the target to be tracked, which is essentially a time series-based, memory-based state that integrates historical information and current observation conditions. It is not an isolated state at a specific time. By updating the target tracking state based on the tracking confidence of the target bounding box, the subsequent tracking strategy can be dynamically adjusted to ensure effective tracking of the target based on the dynamically adjusted tracking strategy.

[0045] In step 105, the working mode of the multi-band sensor array is dynamically adjusted according to the target tracking state and the latest ambient illumination collected by the ambient illumination sensor.

[0046] The target tracking state indicates the detectability of the target to be tracked (such as stability, occlusion, loss), and the ambient illumination indicates the visibility condition of the environment. Combining information in these two dimensions, dynamic optimization scheduling of the working mode of the multi-band sensor array can be achieved. For example, intelligently switching the working mode and parameter configuration of the image sensor: in the case of stable target to be tracked and high ambient illumination, high-precision optical mode is preferred; when there is occlusion or low ambient illumination, automatically switch to an anti-interference mode such as infrared; in the extreme scenario of target to be tracked lost, start the wide-area patrol mode to recapture the target.

[0047] Through the above intelligent sensing scheduling strategy, the target maintenance capability in complex environments is significantly improved, realizing stable tracking and continuous perception in all-weather and all-scenarios, effectively ensuring the reliability of the tracking task.

[0048] According to the ambient illumination collected by the ambient illumination sensor, the above-mentioned embodiments of the present application dynamically configure the value-keeping sensor and the low-power standby auxiliary sensor, maintain the value-keeping state of a single sensor during the period without tracking target, avoid the energy consumption caused by multi-sensor full-power operation, thereby greatly reducing the static power consumption; based on the detection confidence trigger mechanism, when the target to be tracked is identified, the auxiliary sensor is quickly controlled to switch to the full-power working state, realizing seamless switching from the low-power standby mode to the high-performance tracking mode, ensuring the continuous and stable tracking of the dynamic target; during the multi-sensor cooperative working stage, the sensors are controlled to synchronously collect images, the multi-channel images are fused, the target bounding box of the target to be tracked is generated and its tracking confidence is calculated, the target tracking state is updated according to the tracking confidence, and the working mode of the multi-band sensor array is dynamically adjusted according to the latest target tracking state and the latest ambient illumination, forming a real-time resource scheduling closed loop based on the tracking state and environmental perception, which can adaptively allocate resources according to task requirements and environmental changes, achieving an optimal balance among power consumption, computing power and performance on the premise of ensuring tracking reliability.

[0049] Further, different sensor combinations are selected based on ambient illuminance, and the auxiliary sensor is controlled to switch to a full-power working state when a tracking target is monitored, which can effectively fuse the complementary advantages of different waveband sensors, overcome the sensing limitations of a single sensor under different environmental conditions, ensure that robust and accurate tracking results are output under any light and environmental conditions, and improve all-weather working capability and environmental adaptability.

[0050] The process of dynamically configuring the value-keeping sensor and the low-power standby auxiliary sensor based on the ambient illuminance collected by the ambient illuminance sensor is described below.

[0051] In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the daytime threshold, the visible light sensor is configured as the value-keeping sensor, and the near-infrared sensor and the passive long-wave infrared sensor are configured as the auxiliary sensors;

[0052] In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the nighttime threshold and less than or equal to the daytime threshold, the near-infrared sensor is configured as the value-keeping sensor, and the passive long-wave infrared sensor is configured as the auxiliary sensor;

[0053] In response to the ambient illuminance collected by the ambient illuminance sensor being less than or equal to the nighttime threshold, the passive long-wave infrared sensor is configured as the value-keeping sensor, and the near-infrared sensor is configured as the auxiliary sensor.

[0054] In this embodiment, based on the real-time monitoring of the illumination conditions by the ambient illuminance sensor, in combination with the preset daytime threshold and nighttime threshold (the two thresholds are scientific parameters based on the performance critical points of each image sensor, calibrated through actual measurement data and optimized), the primary and secondary roles of different image sensors are assigned to dynamically configure the value-keeping sensor adapted to the ambient illuminance and the auxiliary sensor in a low-power standby state.

[0055] Among them, the characteristics of each image sensor in the multi-waveband sensor array are as follows: the visible light sensor has the advantage of high resolution, can provide rich detail texture and color information, and is the core sensor for realizing fine target recognition and behavior analysis; the near-infrared sensor has certain night vision and penetration ability, is less affected by bad weather, and provides transitional observation support for low-light environments; the passive long-wave infrared sensor does not rely on ambient light at all, directly senses the thermal radiation characteristics of the target, and can realize stable detection under extreme conditions such as complete darkness and strong occlusion. The three types of sensors form a sensing closed loop through mode switching and fusion mechanism, and jointly guarantee the continuity of tracking tasks in complex environments.

[0056] When the ambient illumination is higher than the daytime threshold, the current environment belongs to the strong light mode, the visible light sensor is used as the main sensor (the value guard sensor), and its high resolution advantage can provide clear details and output high-quality images; at the same time, the near-infrared sensor and the passive long-wave infrared sensor are used as auxiliary sensors (backup sensors) to cope with sudden shielding or backlight compensation.

[0057] When the ambient illumination is greater than the nighttime threshold and less than or equal to the daytime threshold, the current environment belongs to the weak light mode (such as the dusk and dawn period), the near-infrared sensor is configured as the value guard sensor, and its night vision enhancement capability is used to maintain observation, and the passive long-wave infrared sensor is used as the auxiliary sensor to cope with deeper light attenuation based on the passive long-wave infrared sensor.

[0058] When the ambient illumination is less than or equal to the nighttime threshold, the current environment is in the extremely low light mode (such as night), the passive long-wave infrared sensor is used as the value guard sensor to realize light source-free perception through thermal radiation detection, and the near-infrared sensor is used as the auxiliary sensor to meet the local light compensation requirement.

[0059] In the above implementation, the value guard sensor and the auxiliary sensor are intelligently scheduled based on the ambient illumination, an efficient architecture of a single main sensor value guard + auxiliary sensor low-power standby is constructed, the image sensor that is adapted to the current environment is dynamically selected as the value guard sensor, and the value guard sensor is combined with the auxiliary sensor in a low-power standby state, which not only ensures low static power consumption during a target tracking period, but also quickly activates the complementary advantages of multiple sensors when a target is found or a tracking challenge is faced through an on-demand wake-up mechanism, thereby overcoming the sensing limitations of a single sensor under different environmental conditions, ensuring that robust and accurate tracking results can be output under any illumination conditions, and realizing reliable tracking at all times and all-weather environmental adaptation.

[0060] In an optional embodiment, a scheme for identifying whether a target to be tracked exists in a value guard image and generating a trigger signal based on the identified target to be tracked includes the following steps:

[0061] An image region including a potential object in the value guard image is obtained;

[0062] The image region is input to a lightweight target detection neural network model, wherein the lightweight target detection neural network model is a network model that is suitable for the waveband of the value guard sensor and is pruned or quantized.

[0063] A target score and a class probability output by the lightweight target detection neural network model are obtained, the target score representing a probability that the image region includes a potential object, and the class probability representing a probability that the potential object belongs to a specific class.

[0064] The target score is multiplied by the maximum value in the category probability to determine a detection confidence;

[0065] In response to the detection confidence being greater than a confidence threshold, the potential object is determined to be a target to be tracked, and a trigger signal is generated.

[0066] For the value guard image reported by the value guard sensor, a series of preprocessing operations are performed on the original value guard image to improve the image quality, so as to provide a clearer, more stable and less noisy input source for subsequent candidate region identification. After image preprocessing, the preprocessed image is processed based on a candidate region identification algorithm to accurately locate the region that may contain a potential object. And through the identification of the candidate image region, the traditional full image scanning can be replaced, and the subsequent calculation amount can be greatly reduced. After identifying and obtaining the candidate image region (such as through cropping), the candidate image region is input into a lightweight target detection neural network model. The model is specially trained and optimized for the current value guard sensor waveband (visible light / near infrared / long wave infrared), has high matching degree of sensing features, and is processed by deep optimization techniques such as pruning and quantization, aiming to complete the task with the smallest calculation overhead and power consumption.

[0067] After the candidate image region is processed based on the lightweight target detection neural network model, the model outputs two key indicators: target score and category probability; the target score is, for example, a value between 0 and 1, indicating the probability that the candidate image region contains any potential object; the category probability is, for example, a vector, indicating the probability distribution of the potential object belonging to each specific category (such as person, vehicle, animal). Then the target score is multiplied by the maximum value in the category probability to obtain a detection confidence. This calculation logic integrates information in two dimensions: a region must both likely contain a target (requiring a high target score) and clearly belong to a certain known category (with a high maximum category probability) for the detection confidence to be high.

[0068] After obtaining the detection confidence based on the above logic, the relationship between the detection confidence and the confidence threshold is determined. If the detection confidence is greater than the confidence threshold, the potential object is determined to be a target to be tracked, and a trigger signal is generated. If the detection confidence is less than or equal to the confidence threshold, the potential object is determined to be an irrelevant target, and the image sensor remains in a low-power value guard state. The trigger signal is a switch that controls the auxiliary sensor to switch to a full-power working state, and based on the signal, the multi-sensor cooperative acquisition and image fusion phase can be entered.

[0069] In the above scheme, through region extraction + lightweight network, only specific regions are intelligently analyzed, the calculation amount can be reduced, and the static power consumption can be reduced; an event-driven mechanism based on detection confidence judgment is adopted, which seamlessly switches from low-power value guard to high-performance tracking when a target to be tracked is identified, which can significantly improve the energy efficiency ratio.

[0070] The following describes the process of identifying a target to be tracked based on detection confidence and triggering sensor cooperation based on three typical lighting scenarios.

[0071] Example 1: Daytime mode (ambient illumination greater than daytime threshold)

[0072] The standby sensor is a visible light sensor, and the auxiliary sensors include a near-infrared sensor and a passive long-wave infrared sensor. In this lighting scenario, the light is sufficient, and the target has rich color and texture features, but there may be shadows, strong light reflections, and other interference.

[0073] The detection confidence calculation process is as follows:

[0074] 1. Obtain image data

[0075] The visible light sensor captures a high-definition color image, the motion detection module senses that there is a motion change in the image, and determines a fast-moving area, such as coordinates (200, 300) with a size of 100x50 pixels, and crops out the area.

[0076] 2. Input to lightweight neural network model

[0077] After adjusting the size of the cropped image block, it is input to a lightweight YOLO model optimized for visible light images. This model is trained using a large number of daytime visible light images (people, cars), and the input is a 3-channel (RGB) image. The model has been pruned and quantized, and focuses on the features of daytime targets such as color, texture, and shape.

[0078] 3. Obtain the output content of the lightweight neural network model

[0079] The lightweight neural network model processes the image block and outputs the most reliable prediction box and its parameters: for example, the target score is 0.95 (the model determines that there is an object in the area), the class probability [person: 0.75, car: 0.20, other: 0.05] (the model judges that the object is most likely a person).

[0080] 4. Calculate detection confidence

[0081] The maximum class probability is 0.75 (taking the probability of "person"), and the target score is 0.95, so the detection confidence is 0.95*0.75=0.7125.

[0082] Assuming the confidence threshold is set to 0.6, since 0.7125>0.6, it is determined that a reliable target is found, a trigger signal is generated, and the low-power standby near-infrared sensor and passive long-wave infrared sensor are awakened, entering a three-sensor fusion tracking mode.

[0083] Example Two: Dusk / Dawn Mode (ambient illumination greater than nighttime threshold, less than or equal to daytime threshold)

[0084] The guard sensor is a near-infrared sensor, and the auxiliary sensor is a passive long-wave infrared sensor. In this light scene, the ambient light is weak, and the signal-to-noise ratio of the visible light sensor decreases, but the near-infrared sensor can still effectively image, and its anti-shadow interference and camouflage recognition ability are more excellent.

[0085] The detection confidence calculation process is as follows:

[0086] 1. Obtain image data

[0087] The near-infrared sensor collects a gray-scale image, and the motion detection module discovers a suspicious area, and crops out the image block.

[0088] 2. Input to a lightweight neural network model

[0089] The cropped gray-scale image block is input to a lightweight YOLO model optimized for the near-infrared band. This model is trained on a large number of near-infrared image data, and uses single-channel gray-scale images as input, which can effectively capture and utilize the unique feature patterns of near-infrared imaging, such as the high reflectivity of vegetation, the infrared response characteristics of certain materials, and the ability to reduce shadow interference.

[0090] 3. Obtain the output content of the lightweight neural network model

[0091] The lightweight neural network model processes the cropped gray-scale image and outputs a target score of 0.88 and a class probability [pedestrian: 0.82, vegetation: 0.15, other: 0.03] (the model excludes most vegetation interference and confirms that it is a pedestrian).

[0092] 4. Calculate the detection confidence

[0093] The maximum class probability is 0.82 (taking the probability of "pedestrian"), and the target score is 0.88, so the detection confidence is 0.88*0.82=0.7216.

[0094] Assuming the confidence threshold is set to 0.6, since 0.7216>0.6, it is determined that a reliable target is found, a trigger signal is generated, the passive long-wave infrared sensor in the low-power standby state is awakened, and the dual-sensor fusion tracking mode is entered.

[0095] Example Three: Night Mode (ambient illumination less than or equal to nighttime threshold)

[0096] The guard sensor adopts a passive long-wave infrared sensor, and the auxiliary sensor is a near-infrared sensor. In this scenario, the environment has no visible light and is in total darkness, and imaging and detection need to rely on the thermal radiation difference between the target and the background.

[0097] The detection confidence calculation process is as follows:

[0098] 1. Obtain image data

[0099] The passive long-wave infrared sensor outputs a frame of thermal radiation image (usually represented as a gray image, where a high pixel value represents a high radiation intensity / high temperature area, and a low pixel value represents a low radiation intensity / low temperature area); based on the radiation intensity difference (such as threshold segmentation or background subtraction), a significant high radiation intensity area (heat source area) is found in the image, which is cropped.

[0100] 2. Input to a lightweight neural network model

[0101] The cropped image block is input to a lightweight YOLO model optimized for thermal imaging. This model takes single-channel thermal radiation intensity data (or grayscale data) as input, and the model is trained based on a thermal imaging dataset, which can effectively identify the shape features of thermal targets (such as the thermal contour of a human body and the thermal signal of a vehicle engine), and is not sensitive to visible light features such as color and texture.

[0102] 3. Obtain the output content of the lightweight neural network model

[0103] The lightweight neural network model processes the input image block and outputs a target score of 0.98 (thermal targets are extremely prominent in cold backgrounds, so the target score is usually high), and class probability [person: 0.90, vehicle: 0.08, animal: 0.02] (the model judges that it is a person according to the thermal contour shape, and effectively distinguishes heat source animals).

[0104] 4. Calculate the detection confidence

[0105] The maximum class probability is 0.90 (taking the probability of "person"), and the target score is 0.98, so the detection confidence is 0.98*0.90=0.882.

[0106] Assuming the confidence threshold is set to 0.6, since 0.882>0.6, it is determined that a reliable target is found, a trigger signal is generated, the near-infrared sensor in the low-power standby state is awakened, and the dual-sensor fusion tracking mode is entered.

[0107] The three embodiments above adopt an adaptive sensor scheduling strategy linked with ambient illumination to dynamically configure the guard sensor and the auxiliary sensor, and combine intelligent detection confidence calculation based on a lightweight waveband-specific model to realize maximized reduction of static power consumption in a target-free period and rapid activation of multi-modal perception capability after a target is found, thereby constructing a highly intelligent tracking system that can seamlessly adapt to full-illumination conditions.

[0108] The process of processing the multi-path image and obtaining the target bounding box of the current frame corresponding to the target to be tracked will be introduced below. The process includes the following steps:

[0109] The obtained multi-path image is subjected to space-time registration.

[0110] Features of each path image are extracted, and the extracted features of each path image are fused to generate comprehensive features.

[0111] The target to be tracked is identified and located based on the comprehensive features, and the target bounding box of the current frame corresponding to the target to be tracked is output.

[0112] After obtaining the multi-path image, each obtained image is first preprocessed, the purpose of which is to unify the data standard and improve the accuracy and robustness of subsequent registration and fusion. The preprocessing operation includes but is not limited to: noise suppression, radiation correction (intensity normalization), geometric distortion correction, image enhancement, etc. After the preprocessing is completed, the multi-path image is subjected to space-time registration.

[0113] The multi-waveband sensor array in the embodiments of the present application adopts a passive coaxial design and supports synchronous data acquisition of the multi-path image sensor. The coaxial optical path design enables the visible light sensor, the near-infrared sensor and the passive long-wave infrared sensor to achieve natural registration at the pixel level or sub-pixel level in space, greatly simplifying the image registration algorithm, and only fine geometric distortion correction is required subsequently. The synchronous acquisition mechanism ensures that all waveband images correspond to the same time, fundamentally eliminating the time misalignment problem caused by target motion. Based on this, when the multi-path image is subjected to space-time registration, the following two processes are included: 1. Spatial registration (pixel alignment); based on the initial alignment basis provided by the coaxial optical path, the geometric transformation parameters (such as a small translation or rotation matrix) obtained through pre-calibration are used to resample the non-reference image (such as a near-infrared image) to achieve sub-pixel level accurate alignment with the reference image (such as a visible light image). This process simultaneously completes the distortion correction of the lenses of each sensor to eliminate optical errors. 2. Time registration (frame alignment); relying on the unified time stamp assigned to the multi-path image by the synchronous acquisition mechanism, the sensor images have completed time alignment when they are collected. Therefore, the multi-path image collected synchronously is processed as a unified data package without the need for software algorithms for time interpolation or compensation.

[0114] After registration is completed, complementary features are extracted from each image and fused to generate comprehensive features, and then the target to be tracked is identified and located based on the comprehensive features to determine the target bounding box of the current frame corresponding to the target to be tracked. In feature extraction, complementary features are extracted from each registered image. For example, rich texture and color features are extracted from the visible light image; stable structural features are extracted from the near-infrared image in a dim light environment; and light-insensitive thermal radiation features are extracted from the long-wave infrared image. In feature fusion, the heterogeneous features from different wavebands are fused at the feature level to generate a comprehensive feature with more comprehensive information and more robustness to environmental interference (such as shadows, camouflage, and strong light). After obtaining the comprehensive feature, the comprehensive feature is input into the identification module, and the target to be tracked is accurately identified and located based on the fused multi-waveband information, and finally the most accurate target bounding box in the current frame is output. The above implementation scheme significantly improves the accuracy and stability of target identification and positioning by fusing the complementary features provided by multi-waveband sensors to construct a comprehensive feature, and finally outputs a target bounding box with higher confidence and higher accuracy. The process of determining the target bounding box is described below through specific examples.

[0115] Example One: Daytime Mode (Three-Way Fusion of Visible Light Sensor, Near-Infrared Sensor, and Passive Long-Wave Infrared Sensor)

[0116] In a daytime sunny scene, the target may be in a shadow, strong light reflection, or a complex background such as a forest or grass, so it is difficult to reliably locate with a single sensor and needs to rely on multiple sensors. The characteristics of the above three types of sensors are as follows: visible light sensor; can provide high-resolution images and contain rich color and texture details, but its performance is easily affected by lighting conditions, and information loss or overexposure may occur in shadow and strong light areas. Near-infrared sensor; has high response capability to vegetation and some artificial materials, has certain ability to penetrate haze, can reduce the effect of shadows, and provides complementary texture and structural information to the visible light sensor. Passive long-wave infrared sensor; directly reflects the thermal radiation characteristics of the object, its imaging is completely independent of environmental lighting conditions, and living targets (such as personnel and vehicles) usually have significant thermal contrast and are very prominent in thermal images, but its spatial resolution is usually low and the texture details are less.

[0117] Figure 2 An implementation process for determining a target bounding box is shown Figure 1 .

[0118] After the multi-channel images provided by the three sensors are spatio-temporally registered, feature extraction is performed on each channel of the images. Specifically, the visible light image (provided by the visible light sensor) is input into a branch CNN (Convolutional Neural Network) 1 to extract a feature map F1, which contains rich edge, corner, color, and texture details; the near-infrared image (provided by the near-infrared sensor) is input into a branch CNN 2 with the same structure but independent weights to extract a feature map F2, which focuses on representing structural information less affected by shadows, vegetation distribution, and material composition identification features of specific artificial materials (such as military camouflage), effectively complementing the visible light features; and the infrared thermal image (provided by the passive long-wave infrared sensor) is input into a branch CNN 3 to extract a feature map F3, which reflects the saliency and approximate shape features of the thermal target region.

[0119] In the feature fusion stage, a strategy based on weighted splicing is used to integrate multi-source features. The feature map F1, the feature map F2, and the feature map F3 are stacked along the channel dimension (if each feature map has 256 channels, then after stacking, there are 768 channels), and a 1x1 convolutional layer is used to fuse the spliced features. This operation is equivalent to learning an adaptive weighted combination mechanism, allowing the network to dynamically adjust the contribution of each sensor feature according to the characteristics of the image region. For example, in the salient region of the thermal target, the long-wave infrared feature is given a higher weight, and in the region where texture structure needs to be distinguished, the visible light feature is emphasized.

[0120] Finally, a comprehensive feature map F fused is generated by fusion, which integrates the complementary advantages of multi-band sensors and carries more information and robustness than single-sensor image features. The comprehensive feature map will be sent to the subsequent task network for final output of a more accurate and robust target bounding box.

[0121] In the feature fusion stage, based on the channel splicing and adaptive weighting fusion strategy, the implementation is as follows:

[0122] First, the feature map F1 (visible light details), the feature map F2 (near-infrared structure), and the feature map F3 (thermal infrared saliency) from the three branches are spliced in the channel dimension to obtain the preliminary fusion feature. If each feature map has 256 channels, then the number of channels of the spliced feature is 768. Then, a 1x1 convolutional layer is used to perform fusion operation on the spliced feature, and the calculation can be represented as:

[0123] F fused =Conv1×1(Concat(F1,F2,F3))

[0124] The 1x1 convolutional layer described above is essentially learning a cross-modal adaptive weighting mechanism, which automatically learns a set of optimal fusion weights through training, so as to dynamically adjust the contribution degree of each modal feature in different image regions. For example: in the presence of obvious hot target area, a higher weight can be assigned to the feature map F3; in the area where texture or color needs to be finely distinguished, the feature map F1 is inclined to be given higher importance; for the target that is partially occluded or in a complex background (such as vegetation), the penetration of the feature map F2 and the saliency of the feature map F3 can be combined to assist identification.

[0125] Finally, the comprehensive feature map F fused will be input into the subsequent shared detection head network (usually composed of RPN (Region Proposal Network) and classification regression sub-network). Based on the provided rich multi-modal features, the network outputs a series of detection results (including the existence confidence, class probability and accurate bounding box coordinates of each candidate target). These detection results are more accurate and robust than single sensor source, providing reliable basis for the final determination and output of high-precision target bounding box of the target to be tracked.

[0126] The core advantage of the above fusion method is to integrate the complementary characteristics of different sensors: visible light provides high-resolution texture and color information; near-infrared provides anti-shading capability and material composition identification features; long-wave infrared provides heat radiation-based, illumination-independent target saliency.

[0127] As an example: when facing a target hidden in the shade of the forest wearing camouflage clothes, the visible light sensor may lose the target due to camouflage and shadow; the near-infrared sensor can prompt the abnormal area according to the material reflection difference between vegetation and camouflage clothes; the passive long-wave infrared sensor can clearly present the heat contour of the human body. Through the proposed fusion method, all the above clues can be used comprehensively to realize accurate detection and positioning of the target.

[0128] Example two: night mode (two-way fusion of near-infrared sensor and passive long-wave infrared sensor)

[0129] In the scene of night without environmental light or with extremely weak environmental light, the present scheme mainly relies on the cooperation of near-infrared sensor and passive long-wave infrared sensor. The near-infrared sensor can image with weak starlight or moonlight in the absence of active light (the present scheme is a passive design), but its signal-to-noise ratio is significantly lower than that in the daytime, and the feature extraction effect is limited; the passive long-wave infrared sensor completely relies on the heat radiation of the object itself for imaging and is not affected by the environmental light. In the cold background at night, the human body, vehicle engine and other heat source targets are extremely prominent.

[0130] Figure 3An implementation process of determining a target bounding box is shown Figure 2 .

[0131] After the spatio-temporal registration of the two images is completed, feature extraction is performed respectively: the near-infrared image (provided by the near-infrared sensor) is input into branch CNN2 to extract feature map F2, which is weak in night signal-to-noise ratio and feature, but still may contain some contour or texture information; the infrared thermal image (provided by the passive long-wave infrared sensor) is input into branch CNN3 to extract feature map F3, which has clear thermal target shape and is used as the dominant feature source in this scenario.

[0132] In the feature fusion stage, an attention-based fusion strategy is adopted, which is suitable for the architecture of the dominant modality (thermal infrared) + auxiliary modality (near-infrared) in this scenario.

[0133] 1. Generating an attention map: the feature map F3 (dominant feature source) is input into a small network (such as a convolutional layer with a Sigmoid activation function) to generate an attention map A with a value between 0 and 1, which is used to identify the importance of each position in the image, such as high-value areas (close to 1) usually correspond to significant thermal targets (such as human bodies and vehicles), and low-value areas (close to 0) represent the background.

[0134] 2. Feature weighting: the near-infrared feature (feature Figure 4 ) of the auxiliary modality is spatially weighted using the attention map A to obtain the modulated feature: F2_attended=A*F2. The physical meaning of this operation is that only in the area where the thermal target may exist, the near-infrared feature is retained or even enhanced (it is believed that it may provide texture and contour details), while in the background area, the near-infrared feature containing a large amount of noise is suppressed.

[0135] 3. Feature fusion: the modulated near-infrared feature and the original long-wave infrared feature are added and fused to obtain the final comprehensive feature map: F fused =F3+F2_attended.

[0136] The core idea of this strategy can be summarized as follows: guided by thermal imaging, enhance low-light imaging. The reliable long-wave infrared feature provides accurate target position prior knowledge, guiding the network to pay attention to and mine the effective information in the near-infrared feature which has more noise but may contain important details.

[0137] 4. The reinforced comprehensive feature map is sent to the shared CNN detection head network for final confidence scoring, classification and bounding box regression, outputting accurate target detection results (such as complete information including confidence, category and target bounding box).

[0138] Among them, the finally generated comprehensive feature map has two advantages: 1. Taking the clear and robust hot target shape as the main body, the high detectability of the target in the complex night environment is ensured; 2. The enhanced low-light details (such as the posture of the person and the local texture of the vehicle) are integrated inside the hot target contour, which not only makes the tracking box more stable, but also improves the ability to distinguish between dense or adjacent targets.

[0139] The scheme for calculating the tracking confidence corresponding to the target bounding box is introduced below. When calculating the tracking confidence corresponding to the target bounding box, the comprehensive feature, the target bounding box of the current frame, and the independent bounding box associated with each target sensor are needed. The target sensor is an image sensor that outputs the target to be tracked. The independent bounding box associated with each target sensor is a bounding box of the target to be tracked determined based on the independent data provided by each target sensor. After multi-image registration and before feature extraction and fusion, the independent data provided by each sensor is analyzed to obtain the independent bounding box associated with each target sensor. The scheme for calculating the tracking confidence includes the following steps:

[0140] According to the comprehensive feature, a response map is generated, and based on the contrast between the global maximum response value in the response map and the response value of the surrounding background area, an appearance similarity score is generated;

[0141] According to the overlap degree of the target bounding box of the current frame and the expected bounding box predicted based on the historical trajectory of the target to be tracked, a motion prediction error is determined;

[0142] The overlap degree between each target sensor associated independent bounding box is calculated, and the average value is calculated to generate a multi-source consistency score;

[0143] According to the appearance similarity score, the motion prediction error and the multi-source consistency score, the tracking confidence corresponding to the target bounding box is determined.

[0144] When calculating the tracking confidence, the appearance matching degree, the motion continuity and the multi-sensor consistency are needed to be considered comprehensively to generate a comprehensive, robust and comprehensive tracking confidence.

[0145] The core idea of the appearance similarity score is to quantitatively evaluate the matching degree and significance of the current frame candidate region and the target historical appearance feature. The specific implementation process is as follows: first, based on the target historical appearance model, a target template for matching in the current frame is generated, and a search area is determined, wherein the target historical appearance model can be understood as a dedicated identity photo library established for the target, and the target template is a specific tool derived from the target historical appearance model for real-time matching in the current frame. Subsequently, in the search area, the similarity between the comprehensive feature map (a form of comprehensive feature) and the target template is calculated by using a correlation filter or a deep learning matching algorithm to generate a response map. The numerical distribution on the response map reflects the similarity score of each position with the target template. By analyzing the peak value characteristics of the response map, a matching significance index can be obtained by calculating the average peak value correlation energy (APCE) or directly regressed by a lightweight scoring network to quantitatively evaluate the contrast between the global maximum response value and its surrounding background area. Combined with the global maximum response value (reflecting the matching degree) and the contrast index (reflecting the matching significance), the appearance similarity score is generated. The higher the score, the more reliable the matching result.

[0146] The core idea of calculating the motion prediction error is to compare the deviation between the predicted position and the actual position based on the motion continuity assumption to evaluate the rationality of the motion trajectory. The specific implementation process is, for example, to use a motion predictor to predict the position (expected bounding box) of the target in the current frame based on the motion state of the target in the previous frames, calculate the overlap between the expected bounding box and the target bounding box of the current frame actually output, and usually use the intersection over union (IoU) to measure. The lower the IoU value, the greater the motion prediction error. The greater the error, the greater the deviation between the actual motion of the target and the predicted situation, which can be used to filter abnormal tracking results.

[0147] The core idea of calculating the multi-source consistency score is to cross-verify using the redundant information between sensors to evaluate the consistency of the tracking results of different sensors. The specific implementation process is to obtain the independent bounding boxes associated with each target sensor (such as visible light sensors, near-infrared sensors, and passive long-wave infrared sensors that output tracking results), and calculate the overlap (IoU) between these independent bounding boxes. Take the average of all IoU values to obtain the multi-source consistency score. The higher the score, the more consistent the judgments of all sensors on the target position, and the stronger the reliability of the tracking results.

[0148] After obtaining the appearance similarity score, motion prediction error and multi-source consistency score, the three parameters of different dimensions are fused into a unified and interpretable comprehensive tracking confidence. The specific implementation is as follows: 1. The three-dimensional parameters are appropriately normalized to eliminate the differences in their dimensions and numerical ranges. It should be noted that the motion prediction error (negative correlation parameter, the smaller the value, the better) needs to be converted into the motion consistency score (positive correlation parameter, the larger the value, the better) to facilitate subsequent unified weighted fusion. 2. According to the specific application scenario, different weights are assigned to each dimension and weighted fusion (such as weighted average) is performed; for example, in a high-speed motion scenario, higher weight can be given to motion consistency, and in a complex background, higher weight can be given to appearance similarity. 3. The comprehensive tracking confidence is generated, which is a scalar value representing the overall evaluation of the reliability of the current frame target bounding box. The higher the value, the more reliable the tracking result. The following two examples are used to illustrate the process of determining the tracking confidence.

[0149] Example one: night mode (near-infrared sensor + passive long-wave infrared sensor)

[0150] 1. Appearance similarity score calculation

[0151] On the comprehensive feature map obtained after feature fusion, the target template is matched with the sliding window in the prediction search area through the tracking algorithm, and a response map is generated. The maximum response value is, for example, Smax = 0.92; the average peak value correlation energy (APCE) of the response map is calculated as a contrast indicator, for example, APCE = 8.5. The appearance similarity score is calculated using the following formula: appearance similarity score = Smax x ln (APCE + 1) = 0.92 x ln (9.5) ≈ 0.92 x 2.25 ≈ 2.07. A high Smax indicates that the algorithm has found a candidate region with high similarity to the target template in the search area, and a high APCE indicates that the matching response of the candidate region is very prominent, and the response value is significantly different from the surrounding background area, and the matching result has high uniqueness, rather than a generally high overall response level. The combination of the two constitutes the appearance similarity score, the former reflects the matching degree, and the latter reflects the significance of the matching result, which can effectively avoid false detection and loss problems. 2.07 is a very high appearance confidence, which indicates that the appearance matching degree is high and the matching result is significant.

[0152] 2. Motion consistency score calculation

[0153] As predicted by the predictor such as Kalman filter, the expected bounding box Bp of the current frame is predicted based on the historical motion state of the target, and the intersection over union (IoU) is calculated with the target bounding box Bt of the current frame (actual tracking result). If IoU = 0.85, the motion prediction error E = 1 - IoU = 0.15. In this example, the IoU value is high (0.85) and the error is small (0.15), indicating that the actual motion trajectory of the target is consistent with the prediction, and the motion consistency is high. If the target suddenly turns sharply or accelerates, the IoU will decrease and the motion prediction error will increase. Convert the motion prediction error E to the positively correlated motion consistency score, and the motion consistency score = 1 - E = IoU = 0.85. The score is high, indicating that the actual motion trajectory of the target is consistent with the prediction of the prediction model, and the motion consistency is high.

[0154] 3. Multi-source consistency score calculation

[0155] When multiple sensors are working, each sensor that outputs a target to be tracked is associated with an independent bounding box. The consistency of these independent bounding boxes is compared. For example, the near-infrared sensor independently tracks to obtain an independent bounding box B1, and the passive long-wave infrared sensor independently tracks to obtain an independent bounding box B2. The intersection over union (IoU) of the two is calculated. Since there are only two sensors, the average value is equal to this IoU value. For example, the intersection over union of the two is 0.95, and the average value is also 0.95. A score of 0.95 is extremely high, indicating that both sensors have captured the same target and their judgments of the target's position are highly consistent, which is strong evidence of tracking reliability. If one of the sensors is temporarily disturbed, its tracking result will drift, causing the intersection over union with the other sensor to decrease, thereby lowering the average value.

[0156] 4. Weighted fusion to obtain tracking confidence

[0157] The above three types of scores are normalized. The appearance similarity score is normalized: 2.07 / 3 ≈ 0.69; the motion consistency score is normalized: 0.85 (already in the range [0, 1], no need to process); the multi-source consistency score is normalized: 0.95 (already in the range [0, 1], no need to process). If the same weight is given to the three factors (α = β = γ = 1 / 3), the tracking confidence = (0.69 + 0.85 + 0.95) / 3 ≈ 0.83, which is a high value (full score 1.0), indicating that the current frame tracking result has high reliability.

[0158] Example Two: Daytime Mode (Visible Light Sensor + Near-Infrared Sensor + Passive Long-Wave Infrared Sensor)

[0159] This example demonstrates the process of calculating the tracking confidence by fusing the information of the visible light sensor, the near-infrared sensor, and the passive long-wave infrared sensor in a daytime environment.

[0160] 1. Appearance similarity score calculation

[0161] Based on the integrated feature map fused by the features provided by the three types of sensors and the target template, a response map is generated (see the foregoing description, which will not be repeated here), and then the maximum response value and the average peak value related energy (APCE) are obtained. For example, the maximum response value Smax = 0.95, the APCE = 9.2 as the contrast index, and the appearance similarity score = Smax x ln (APCE + 1) = 0.95 x ln (10.2) ≈ 0.95 x 2.32 ≈ 2.21 is calculated according to the formula. The value is relatively high, indicating that the appearance matching degree is relatively high, and the matching result is significantly strong.

[0162] 2. Motion consistency score calculation

[0163] The expected bounding box Bp of the current frame is predicted through the Kalman filter, and the intersection over union (IoU) is calculated with the target bounding box Bt (the actual tracking result) of the current frame. The measured IoU = 0.88. Then the motion prediction error E = 1-IoU = 0.12. The motion prediction error is converted into the motion consistency score, and the motion consistency score = IoU = 0.88. The value indicates that the actual motion trajectory of the target is highly consistent with the prediction of the prediction model.

[0164] 3. Multi-source consistency score calculation

[0165] Respectively obtain the independent bounding boxes associated by the three sensors respectively, calculate the intersection over union (IoU) between all the two independent bounding boxes and take the average value. For example, the three intersection over unions are 0.90, 0.82 and 0.85 respectively, and the average value of the intersection over union = (0.90 + 0.82 + 0.85) / 3 ≈ 0.86. The value indicates that the observation results of the three sensors have high consistency.

[0166] 4. Tracking confidence calculation

[0167] The above scores are normalized. The appearance similarity score normalization: 2.21 / 3 ≈ 0.74, the motion consistency score normalization: 0.88 (already in the range of [0, 1]), and the multi-source consistency score normalization: 0.86 (already in the range of [0, 1]). If equal weight weighted fusion (weight is 1 / 3) is adopted, then the tracking confidence value = (0.74 + 0.88 + 0.86) / 3 ≈ 0.83. It indicates that the tracking result has high reliability in the current daytime mode, and the stable tracking state can be maintained.

[0168] The above two embodiments effectively improve the tracking reliability in complex environments by multi-modal feature fusion, comprehensive appearance, motion and multi-source consistency scores, and weighted fusion to generate tracking confidence.

[0169] In the technical solution, "synchronous acquisition" and "coaxial light path" are adopted. The "spatiotemporal natural registration" brought by the combination of the two is the basis for realizing high-performance multi-modal fusion.

[0170] To realize precise synchronous acquisition under the "on-demand wake-up" mechanism, the application has a deep collaborative design at the hardware architecture level. The multi-band sensor array is not a simple stack of sensors, but a hardware-level synchronization is realized through a dedicated synchronous trigger circuit and exposure control logic. When the main processor issues a synchronous acquisition instruction, the circuit ensures that the exposure start and end times of all awakened sensors (guard sensors and auxiliary sensors) are strictly aligned, thereby eliminating the time misalignment between multiple images caused by the difference in sensor start-up or exposure time from the hardware root. This hardware-level synchronization guarantee is the physical basis for subsequent implementation of high-precision multi-modal spatiotemporal registration and fusion, and cannot be achieved by software scheduling alone.

[0171] Further, the aforementioned "coaxial light path" design of the application is a necessary prerequisite and hardware guarantee for realizing the innovative algorithm feature of "multi-source consistency score". Since the visible light, near-infrared, and long-wave infrared sensors are integrated on the same optical axis through a beam splitter prism, they physically ensure that the observation angles for the same scene at any time are completely consistent, realizing pixel-level spatial innate registration between images of different wavebands. It is this inherent spatial consistency guaranteed by hardware that makes the "independent bounding boxes" generated based on the independent data of each sensor directly comparable. On this hardware basis, the intersection over union (IoU) between each pair of independent bounding boxes is calculated to generate the "multi-source consistency score", which has clear physical meaning and high reliability.

[0172] Therefore, the "coaxial light path" hardware design of the application and the subsequent "multi-source consistency score" algorithm feature constitute an inseparable collaborative innovation whole, and the two are closely combined to jointly ensure the accuracy and robustness of the target tracking confidence evaluation in complex environments.

[0173] The hardware foundation of "coaxial light path" and the algorithm feedback of "multi-source consistency verification" form the first layer of enhanced loop. The coaxial light path ensures the pixel-level spatial innate registration between multi-band images, providing a physical possibility for the accurate comparison (IoU calculation) of independent bounding boxes of different sensors at the algorithm level; while the calculation result of the "multi-source consistency score" verifies the accuracy of hardware synchronization and registration in the reverse direction, and provides a reliable confidence basis for fusion tracking. This closed loop, in which hardware provides a cornerstone for algorithms and algorithms provide verification and optimization guidance for hardware, makes the overall reliability of the system far exceed the simple superposition of single hardware or algorithm improvement. The following describes a scheme for updating the target tracking state of the target to be tracked based on the target bounding box corresponding to the tracking confidence, which specifically includes the following steps:

[0174] In response to the tracking confidence corresponding to the target bounding box being greater than a first threshold value, the historical tracking state of the target to be tracked being in a stable state for more than a first time length or the historical tracking state being in a preliminary confirmation state for more than a second time length, it is determined that the target to be tracked enters a high-confidence stable tracking state;

[0175] In response to the tracking confidence corresponding to the target bounding box being greater than a third threshold value and less than or equal to the first threshold value, and the historical tracking state of the target to be tracked being in the preliminary confirmation state for more than a third time length, it is determined that the target to be tracked enters a stable state.

[0176] In response to the tracking confidence corresponding to the target bounding box being less than or equal to a second threshold value, and the historical tracking state of the target to be tracked being in a high-confidence stable tracking state or a stable state, it is determined that the target to be tracked is degraded to a preliminary confirmation state.

[0177] In response to the tracking confidence corresponding to the target bounding box being less than or equal to a second threshold value, and the historical tracking state of the target to be tracked being in a preliminary confirmation state, it is determined that the target to be tracked is in a failure state.

[0178] In response to the tracking confidence corresponding to the target bounding box being greater than the second threshold value and less than or equal to a third threshold value, the historical tracking state of the target to be tracked is maintained.

[0179] The first threshold value is greater than the third threshold value, and the third threshold value is greater than the second threshold value.

[0180] In updating the target tracking state, the instantaneous tracking quality (indicated by the tracking confidence) of the current frame and the persistence of the historical tracking state need to be comprehensively evaluated to determine the upgrade, downgrade or maintenance of the target tracking state. That is, the change of the tracking state needs to be supported by continuous and high-quality evidence, and the judgment of tracking failure is relatively rapid and sensitive, so as to improve the reliability of tracking.

[0181] In the scheme of updating the target tracking state, multiple state levels are defined, and the conversion conditions therebetween are explicitly defined (the first threshold value > the third threshold value > the second threshold value).

[0182] For the high confidence stable tracking state, its entering condition is the most strict. Its instantaneous quality requirement is high: the tracking confidence is greater than the first threshold (the highest threshold); the historical persistence requirement is that the historical state must be stable for a period of time (the first time length) or be confirmed for a period of time (the second time length), and the second time length is usually greater than the first time length. This means that enough reliable evidence needs to be accumulated to allow entering the highest trust level. The preliminary confirmation state can be understood as follows: a possible target has been found, but the reliability of the tracking result thereof is still in doubt, and it needs to be observed for several frames (for a period of time) to decide whether to convert it into a stable state or to determine that it is invalid due to poor performance. If the target is a high confidence stable tracking state, it means that the tracking result of the target has the highest degree of trust.

[0183] For the stable state, its entering condition is a standard condition. Its instantaneous quality requirement is medium: the tracking confidence is greater than the third threshold (the intermediate threshold) and less than or equal to the first threshold; the historical persistence requirement is that the historical state must be confirmed for a period of time (the third time length). That is, the tracking quality of the target is good and reliable, and it is a normal tracking state.

[0184] For the preliminary confirmation state, the entering condition is: 1. Downgrade, if the tracking confidence is less than or equal to the second threshold (the lowest threshold), the target falls from a higher state (the stable state or the high confidence stable tracking state), and then is downgraded to this state; 2. Initial state, when a new target is just detected, it may first enter this state. The preliminary confirmation state is a buffer or warning state, if the tracking result of the target is doubtful, but it is not completely given up, this state gives a waiting period, if the quality of the subsequent frames rises, the target can be restored; if it is continuously poor, it is determined to be invalid.

[0185] For the invalid state, its entering condition is the most sensitive. The instantaneous quality requirement is that the tracking confidence is less than or equal to the second threshold (the lowest threshold); the historical state requirement is that the target is already in the preliminary confirmation state. That is, in the preliminary confirmation state, if it is found that the tracking confidence of the current frame is low, it is determined that the target has been lost, and the re-detection logic needs to be triggered to search for the target in the subsequent frames.

[0186] For state maintenance, the condition is that the tracking confidence is between the second threshold and the third threshold, that is, greater than the second threshold and less than or equal to the third threshold, to maintain the historical tracking state of the target to be tracked; this can be regarded as a gray area, the tracking quality is neither good enough to upgrade nor poor enough to downgrade, and the current state is selected not to be changed, and the change of the subsequent frames is continued to be observed.

[0187] In the above embodiments, the first threshold (the highest) is set to ensure that only when the appearance, motion, and multi-source consistency are all excellent, the auxiliary sensor can be turned off; the second threshold (the lowest) is set to be very sensitive to ensure that the trend of tracking failure can be quickly captured; the third threshold (the middle) is a standard of “good performance”. The introduction of multiple time length parameters (the first, second, and third time lengths) requires that the state transition must be persistent, which effectively filters transient noise and is a very unconventional design that is not easily thought of by those skilled in the art. This state machine control is combined with the “passive” constraint. It is emphasized that because the passive sensor has low signal-to-noise ratio and high uncertainty in weak light, a complex and cautious state machine needs to be designed to manage risks and power consumption. This is a customized solution for a specific problem.

[0188] The target tracking state updating scheme based on tracking confidence and time duration described above balances between reliable tracking and sensitive loss judgment by setting high-confidence stable tracking state, stable state, preliminary confirmation state, and failure state, and defining strict conversion conditions of the three confidence thresholds and the duration, thereby improving the tracking robustness and stability in complex scenes.

[0189] The target tracking state in the embodiments of the present application includes high-confidence stable tracking state, stable state, preliminary confirmation state, and failure state. When the working mode of the multi-waveband sensor array is dynamically adjusted according to the target tracking state and the latest ambient illuminance collected by the ambient illuminance sensor, the following is included:

[0190] In response to the target tracking state being the high-confidence stable tracking state, the auxiliary sensor is controlled to enter a low-power standby state, and the tracking state of the value guard sensor is maintained;

[0191] In response to the target tracking state being the stable state, the value guard sensor and part of the auxiliary sensor are controlled to work cooperatively, and the part of the auxiliary sensor is determined based on the latest ambient illuminance;

[0192] In response to the target tracking state being the preliminary confirmation state, the value guard sensor and all auxiliary sensors are controlled to perform fusion detection, and the weight distribution of each sensor is dynamically optimized according to the latest ambient illuminance;

[0193] In response to the target tracking state being the failure state, the auxiliary sensor is controlled to enter a low-power standby state, the value guard sensor is controlled to switch to a low-frame-rate wide-area patrol mode, and in response to recapturing the target, the value guard sensor and the auxiliary sensor are dynamically adjusted based on the latest ambient illuminance.

[0194] If the target tracking state is a high-confidence stable tracking state, that is, the target to be tracked has been firmly locked, the energy-saving precision mode can be entered, in which the auxiliary sensors enter a low-power standby state, and the main sensor (the guard sensor, the image sensor most suitable for the current environment illumination, such as the visible light sensor in the daytime) continues to maintain the working state, realizing the maximum energy saving and prolonging the device endurance on the premise of ensuring the tracking effect.

[0195] If the target tracking state is a stable state, there may be some uncertainties (such as occasional partial occlusion, poor image quality) in the process of tracking the target, at which time the guard sensor and part of the auxiliary sensors need to work cooperatively, that is, a part of the auxiliary sensors are called to supplement information to effectively track the target. The called part of the auxiliary sensors is determined based on the environment illumination. For example, in the daytime with good light, the passive long-wave infrared sensor is called to provide complementary thermal feature information to the visible light to enhance the recognition ability of the occluded or disguised target; in the evening or poor light, the passive long-wave infrared sensor is called to use its thermal imaging characteristics independent of light to provide key thermal radiation data support for the near-infrared sensor as the guard sensor, and to overcome the uncertainty in the low light. Through moderate resource investment, the uncertainty is resolved to ensure tracking stability, while avoiding the waste of resources caused by waking up all sensors.

[0196] If the target tracking state is a preliminary confirmation state, such as the initial state in the above embodiment or the state determined by degradation, the full verification mode is entered, and the guard sensor and all auxiliary sensors perform fusion detection to collect as much information as possible for cross verification. In this process, the weight distribution of each sensor is dynamically optimized according to the latest environment illumination, for example, the image information provided by the visible light sensor in the daytime has the highest weight and is used as the main judgment basis, and the information provided by the thermal imaging sensor at night has the highest weight. By scheduling various sensors, the complementary advantages of multi-band information (such as the details of visible light, the micro-light structure of near-infrared, and the thermal features of infrared) can be comprehensively utilized, the target identity and state are verified to the greatest extent, and the most sufficient data basis is provided for state updating.

[0197] If target tracking fails, the auxiliary sensor enters a low-power standby state to reduce power consumption, while the primary sensor switches to a low-frame-rate wide-area survey mode, meaning the primary sensor transitions from a close-monitoring mode to a surround-viewing mode. Lowering the refresh rate further saves energy, and wide-area surveying expands the field of view, making it easier to rediscover the target. Survey parameters are configured based on the latest ambient light levels to intelligently adjust them. For example, in high-light daytime conditions, the primary visible light sensor uses lower gain and higher resolution for wide-area scanning; at dusk, the primary near-infrared sensor automatically increases sensitivity (gain) and may employ specific image enhancement algorithms to optimize the probability of detection in low-light environments. If the target is recaptured based on wide-area surveying (whether it's a previous target or a new one), the primary sensor in the multi-band sensor array and the auxiliary sensor in low-power standby mode are dynamically reconfigured based on the latest ambient light dynamics. The above implementation scheme can achieve an optimal balance between tracking performance and energy saving by dynamically adjusting the working mode of the multi-band sensor array, significantly extending the equipment's battery life, while ensuring tracking reliability and environmental adaptability in different scenarios.

[0198] This invention provides a passive multi-view dynamic band tracking system, such as... ​ As shown, it includes a main processor 10, a multi-band sensor array 20 that interacts with the main processor 10, and an ambient light sensor 30. The multi-band sensor array 20 includes a visible light sensor 21, a near-infrared sensor 22, and a passive long-wave infrared sensor 23, which are image sensors. All image sensors do not contain active emitters and are integrated in a unified housing through a coaxial optical path. The visible light sensor 21 and the near-infrared sensor 22 share a first beam splitter.

[0199] The main processor 10 includes: a central processing unit 11, an image signal processor 12, and a tracking engine 13;

[0200] Ambient illuminance sensor 30 collects ambient illuminance data;

[0201] The central processing unit 11 dynamically configures the on-duty sensors and the auxiliary sensors in low-power standby mode in the multi-band sensor array 20 according to the ambient light level.

[0202] Image signal processor 12 receives the monitored images acquired by the monitored sensor and provides the monitored images to tracking engine 13;

[0203] The tracking engine 13 responds to the identification of a target to be tracked in the monitored image whose detection confidence meets the preset conditions, and reports it to the central processing unit 11. The central processing unit 11 generates a trigger signal and controls the auxiliary sensor to switch to full power operation based on the trigger signal 11.

[0204] The central processor 11 controls the guard sensors and the auxiliary sensors to synchronously capture images of the target to be tracked at the same time;

[0205] The image signal processor 12 receives multiple images associated with the target to be tracked and provides the images to the tracking engine 13;

[0206] The tracking engine 13 processes the multiple images to obtain a target bounding box of a current frame corresponding to the target to be tracked and a tracking confidence corresponding to the target bounding box, and provides the tracking confidence to the central processor 11;

[0207] The central processor 11 updates a target tracking state of the target to be tracked according to the tracking confidence corresponding to the target bounding box, and dynamically adjusts a working mode of the multi-band sensor array according to the target tracking state and a latest ambient illumination collected by the ambient illumination sensor 30.

[0208] Optionally, in response to the ambient illumination collected by the ambient illumination sensor 30 being greater than a daytime threshold, the central processor 11 configures the visible light sensor 21 as the guard sensor, and configures the near-infrared sensor 22 and the passive long-wave infrared sensor 23 as the auxiliary sensors;

[0209] In response to the ambient illumination collected by the ambient illumination sensor 30 being greater than a nighttime threshold and less than or equal to the daytime threshold, the central processor 11 configures the near-infrared sensor 22 as the guard sensor, and configures the passive long-wave infrared sensor 23 as the auxiliary sensor;

[0210] In response to the ambient illumination collected by the ambient illumination sensor 30 being less than or equal to the nighttime threshold, the passive long-wave infrared sensor 23 is configured as the guard sensor, and the near-infrared sensor 22 is configured as the auxiliary sensor.

[0211] Optionally, the image signal processor 12 pre-processes the guard image and provides the pre-processed guard image to the tracking engine 13; the tracking engine 13 obtains the pre-processed guard image and obtains a candidate image region including a potential object in the pre-processed guard image;

[0212] The tracking engine 13 inputs the candidate image region to a lightweight target detection neural network model, wherein the lightweight target detection neural network model is a network model that is pruned or quantized and is suitable for the wave band of the guard sensor;

[0213] The tracking engine 13 obtains a targetness score and a class probability output by the lightweight target detection neural network model, wherein the targetness score represents a probability that the candidate image region includes the potential object, and the class probability represents a probability that the potential object belongs to a specific class;

[0214] The tracking engine 13 multiplies the target score with the maximum value in the category probability to determine a detection confidence, determines that the potential object is a target to be tracked in response to the detection confidence being greater than a confidence threshold, and reports the presence of the target to be tracked in the watch image to the central processor 11, which generates a trigger signal.

[0215] Optionally, the image signal processor 12 pre-processes the multi-channel images and provides the pre-processed multi-channel images to the tracking engine 13, which performs spatio-temporal registration on the pre-processed multi-channel images.

[0216] The tracking engine 13 extracts features of each channel of images, fuses the extracted features of each channel of images to generate comprehensive features, and identifies and locates the target to be tracked based on the comprehensive features to output a target bounding box of the current frame corresponding to the target to be tracked.

[0217] The tracking engine 13 calculates a tracking confidence corresponding to the target bounding box according to the comprehensive features, the target bounding box of the current frame, and independent bounding boxes respectively associated with the target sensors.

[0218] Optionally, the tracking engine 13 generates a response map according to the comprehensive features, and generates an appearance similarity score based on a contrast between a global maximum response value in the response map and a response value of a surrounding background region.

[0219] The tracking engine 13 determines a motion prediction error according to an overlap degree between the target bounding box of the current frame and an expected bounding box predicted based on a historical trajectory of the target to be tracked.

[0220] The tracking engine 13 calculates an overlap degree between each two of the independent bounding boxes respectively associated with the target sensors, and generates a multi-source consistency score by calculating an average value.

[0221] The tracking engine 13 determines the tracking confidence corresponding to the target bounding box according to the appearance similarity score, the motion prediction error, and the multi-source consistency score.

[0222] Optionally, in response to the tracking confidence corresponding to the target bounding box being greater than a first threshold, a historical tracking state of the target to be tracked being in a stable state for more than a first time length or being in a preliminary confirmation state for more than a second time length, the central processor 11 determines that the target to be tracked enters a high-confidence stable tracking state.

[0223] In response to the tracking confidence corresponding to the target bounding box being greater than a third threshold and less than or equal to the first threshold, and the historical tracking state of the target to be tracked being in the preliminary confirmation state for more than a third time length, the central processor 11 determines that the target to be tracked enters a stable state.

[0224] In response to the tracking confidence corresponding to the target bounding box being less than or equal to the second threshold value and the historical tracking state of the target to be tracked being the high-confidence stable tracking state or the stable state, the central processor 11 determines that the target to be tracked is degraded to the preliminary confirmation state;

[0225] In response to the tracking confidence corresponding to the target bounding box being less than or equal to the second threshold value and the historical tracking state of the target to be tracked being the preliminary confirmation state, the central processor 11 determines that the target to be tracked is in the invalid state.

[0226] In response to the tracking confidence corresponding to the target bounding box being greater than the second threshold value and less than or equal to the third threshold value, the central processor 11 determines that the target to be tracked maintains the historical tracking state.

[0227] The first threshold value is greater than the third threshold value, and the third threshold value is greater than the second threshold value.

[0228] Optionally, the target tracking state includes a high-confidence stable tracking state, a stable state, a preliminary confirmation state, and an invalid state.

[0229] In response to the target tracking state being the high-confidence stable tracking state, the central processor 11 controls the auxiliary sensor to enter a low-power standby state and maintains the tracking state of the value-guarding sensor.

[0230] In response to the target tracking state being the stable state, the central processor 11 controls the value-guarding sensor and part of the auxiliary sensor to work cooperatively, and the part of the auxiliary sensor is determined based on the latest ambient illuminance.

[0231] In response to the target tracking state being the preliminary confirmation state, the central processor 11 controls the value-guarding sensor and all auxiliary sensors to perform fusion detection, and dynamically optimizes the weight distribution of each sensor according to the latest ambient illuminance.

[0232] In response to the target tracking state being the invalid state, the central processor 11 controls the auxiliary sensor to enter a low-power standby state, controls the value-guarding sensor to switch to a low-frame-rate wide-area patrol mode, and in response to recapturing the target, the central processor 11 dynamically adjusts the value-guarding sensor and the auxiliary sensor based on the latest ambient illuminance.

[0233] In the embodiment, the preliminary confirmation state is set to prevent frequent state transitions when the target is temporarily blocked or the image quality fluctuates instantaneously, thereby avoiding system oscillation and improving stability. The high-confidence stable tracking state is set to ensure that the most aggressive energy-saving strategy is executed only when it is absolutely necessary, i.e., all auxiliary sensors are turned off.

[0234] The power consumption control of the "on-demand wake-up" and the resource scheduling of the "state machine" drive form an enhanced loop. The "on-demand wake-up" mechanism under the "passive" constraint greatly reduces the static power consumption of the system, and the four-state machine (high confidence stability, stability, preliminary confirmation, and failure) based on multi-dimensional confidence and time persistence intelligently manages the resource allocation from "low-power value guard" to "full fusion tracking" to "recapture". The state machine makes decisions based on tracking confidence and historical state to determine which sensors to wake up and what fusion weight to use, thereby achieving precise on-demand allocation of computing resources, sensing resources, and energy resources. This not only brings significant reduction in power consumption, but also improves the tracking accuracy and robustness of the system under the same power budget by investing sufficient resources in key links.

[0235] The present application produces a "synergistic gain" effect through the deep fusion of the above hardware-algorithm-power consumption control. The overall performance of the system is: while significantly reducing the static power consumption, the tracking stability and accuracy in challenging scenes such as target occlusion, sudden light changes, and complex backgrounds are greatly improved through multi-modal complementary fusion and intelligent resource scheduling. This simultaneous optimization of power consumption and performance, which are traditionally contradictory indicators, is a concentrated embodiment of the "1+1 greater than 2" technical effect of the present application, and its overall performance far exceeds that of systems using traditional continuous fusion solutions or simple sensor switching strategies.

[0236] It should be noted that the above detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs.

[0237] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A passive multi-view dynamic band tracking method, applied to a main processor, wherein the main processor interacts with a multi-band sensor array and an ambient light sensor, the multi-band sensor array including a visible light sensor, a near-infrared sensor, and a passive long-wave infrared sensor belonging to image sensors, all image sensors do not contain active emitters and are integrated in a unified housing via a coaxial optical path, the visible light sensor and the near-infrared sensor share a first beam splitter, characterized in that, The method includes: Based on the ambient illuminance collected by the ambient illuminance sensor, the standby sensor and the auxiliary sensor in the multi-band sensor array are dynamically configured. The standby sensor is an image sensor adapted to the ambient illuminance. In response to identifying a target to be tracked in the monitored image acquired by the monitored sensor whose detection confidence meets a preset condition, a trigger signal is generated, and the auxiliary sensor is controlled to switch to full power operation state according to the trigger signal; The system controls the monitoring sensor and the auxiliary sensor to simultaneously acquire images of the target to be tracked at the same time, thereby obtaining multiple images associated with the target to be tracked. The multi-channel images are processed to obtain the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box, and the target tracking status of the target to be tracked is updated according to the tracking confidence corresponding to the target bounding box. The operating mode of the multi-band sensor array is dynamically adjusted based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor.

2. The method according to claim 1, characterized in that, The step of dynamically configuring the standby sensors and auxiliary sensors in a low-power standby state in the multi-band sensor array based on the ambient illuminance collected by the ambient illuminance sensor includes: In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the daytime threshold, the visible light sensor is configured as the standby sensor, and the near-infrared sensor and the passive long-wave infrared sensor are configured as the auxiliary sensors. In response to the ambient illuminance collected by the ambient illuminance sensor being greater than the nighttime threshold and less than or equal to the daytime threshold, the near-infrared sensor is configured as the monitoring sensor, and the passive long-wave infrared sensor is configured as the auxiliary sensor. In response to the ambient illuminance collected by the ambient illuminance sensor being less than or equal to the nighttime threshold, the passive long-wave infrared sensor is configured as the monitoring sensor, and the near-infrared sensor is configured as the auxiliary sensor.

3. The method according to claim 1, characterized in that, The response to identifying a target to be tracked in the monitored image acquired by the monitored sensor whose detection confidence level meets a preset condition, and generating a trigger signal, includes: Obtain candidate image regions containing potential objects from the monitored image; The candidate image region is input into a lightweight target detection neural network model, wherein the lightweight target detection neural network model is a pruned or quantized network model suitable for the monitored sensor band; Obtain the target score and category probability output by the lightweight object detection neural network model, wherein the target score represents the probability that the candidate image region includes the potential object, and the category probability represents the probability that the potential object belongs to a specific category; The detection confidence is determined by multiplying the target score by the maximum value among the category probabilities. In response to the detection confidence level being greater than a confidence threshold, the potential object is determined to be a target to be tracked, and the trigger signal is generated.

4. The method according to claim 1, characterized in that, The step of processing the multi-channel images to obtain the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box includes: Spatiotemporal registration is performed on the acquired multi-channel images; Extract features from each image stream, and then fuse the extracted features to generate a comprehensive feature. Based on the comprehensive features, the target to be tracked is identified and located, and the target bounding box of the current frame corresponding to the target to be tracked is output; Based on the comprehensive features, the target bounding box of the current frame, and the independent bounding boxes associated with each target sensor, the tracking confidence corresponding to the target bounding box is calculated, and the target sensor is the image sensor that outputs the target to be tracked.

5. The method according to claim 4, characterized in that, The step of calculating the tracking confidence corresponding to the target bounding box based on the integrated features, the target bounding box of the current frame, and the independent bounding boxes associated with each target sensor includes: A response map is generated based on the comprehensive features, and an appearance similarity score is generated based on the contrast between the global maximum response value in the response map and the response value of the surrounding background area. The motion prediction error is determined based on the degree of overlap between the target bounding box in the current frame and the expected bounding box predicted based on the historical trajectory of the target to be tracked. Calculate the overlap between each pair of independent bounding boxes associated with each target sensor, and calculate the average value to generate a multi-source consistency score; The tracking confidence level corresponding to the target bounding box is determined based on the appearance similarity score, the motion prediction error, and the multi-source consistency score.

6. The method according to claim 1, characterized in that, The step of updating the target tracking state of the target to be tracked based on the tracking confidence corresponding to the target bounding box includes: In response to the tracking confidence level corresponding to the target bounding box being greater than a first threshold, if the historical tracking state of the target to be tracked has been in a stable state for more than a first duration or the historical tracking state has been in a preliminary confirmation state for more than a second duration, it is determined that the target to be tracked has entered a high-confidence stable tracking state. In response to the tracking confidence level corresponding to the target bounding box being greater than a third threshold, less than or equal to a first threshold, and the historical tracking status of the target to be tracked being in a preliminary confirmation state for more than a third duration, it is determined that the target to be tracked has entered a stable state. In response to the tracking confidence level corresponding to the target bounding box being less than or equal to the second threshold, and the historical tracking state of the target to be tracked being a high-confidence stable tracking state or a stable state, the target to be tracked is determined to be downgraded to a preliminary confirmation state. In response to the tracking confidence level corresponding to the target bounding box being less than or equal to the second threshold, and the historical tracking status of the target to be tracked being a preliminary confirmation status, the target to be tracked is determined to be in a failed state. In response to the tracking confidence level corresponding to the target bounding box being greater than a second threshold and less than or equal to a third threshold, the historical tracking state of the target to be tracked is maintained. Wherein, the first threshold is greater than the third threshold, and the third threshold is greater than the second threshold.

7. The method according to any one of claims 1 to 6, characterized in that, The target tracking status includes high-confidence stable tracking status, stable status, preliminary confirmation status, and failure status; The step of dynamically adjusting the operating mode of the multi-band sensor array based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor includes: In response to the target tracking state being a high-confidence stable tracking state, the auxiliary sensor is controlled to enter a low-power standby state, and the tracking state of the guard sensor is maintained; In response to the target tracking state being stable, the system controls the on-duty sensor and some auxiliary sensors to work together, with the auxiliary sensors determining the latest ambient illuminance based on the latest ambient light level. In response to the target tracking status being in the preliminary confirmation state, the system controls the monitoring sensor and all auxiliary sensors to perform fusion detection, and dynamically optimizes the weight allocation of each sensor based on the latest ambient light level. In response to the target tracking state being in a failed state, the auxiliary sensor is controlled to enter a low-power standby state, the guard sensor is switched to a low frame rate wide-area survey mode, and in response to re-acquiring the target, the guard sensor and the auxiliary sensor are dynamically adjusted based on the latest ambient light.

8. A passive multi-view dynamic band tracking system, comprising a main processor, a multi-band sensor array interacting with the main processor, and an ambient light sensor, wherein the multi-band sensor array includes a visible light sensor, a near-infrared sensor, and a passive long-wave infrared sensor belonging to image sensors; all image sensors do not contain active emitters and are integrated in a unified housing via a coaxial optical path; the visible light sensor and the near-infrared sensor share a first beam splitter, characterized in that: The main processor includes: a central processing unit, an image signal processor, and a tracking engine; The ambient illuminance sensor collects ambient illuminance; The central processing unit dynamically configures the on-duty sensors and the auxiliary sensors in low-power standby mode in the multi-band sensor array according to the ambient illuminance. The image signal processor receives the monitored image acquired by the monitored sensor and provides the monitored image to the tracking engine; The tracking engine responds to the identification of a target to be tracked in the monitored image whose detection confidence meets a preset condition, and reports it to the central processing unit. The central processing unit generates a trigger signal and controls the auxiliary sensor to switch to full power operation based on the trigger signal. The central processing unit controls the monitoring sensor and the auxiliary sensor to synchronously acquire images of the target to be tracked at the same time. The image signal processor receives multiple images associated with the target to be tracked and provides them to the tracking engine; The tracking engine processes the multiple images, obtains the target bounding box of the current frame corresponding to the target to be tracked and the tracking confidence corresponding to the target bounding box, and provides the tracking confidence to the central processing unit; The central processing unit updates the target tracking status of the target to be tracked based on the tracking confidence level, and dynamically adjusts the operating mode of the multi-band sensor array based on the target tracking status and the latest ambient illuminance collected by the ambient illuminance sensor.

9. The system according to claim 8, characterized in that, The tracking engine performs spatiotemporal registration on the multi-channel images; The tracking engine extracts features from each image stream, fuses the extracted features to generate a comprehensive feature, and identifies and locates the target to be tracked based on the comprehensive feature, outputting the target bounding box of the current frame corresponding to the target to be tracked. The tracking engine generates a response map based on the comprehensive features, and generates an appearance similarity score based on the contrast between the global maximum response value in the response map and the response value of the surrounding background area. The tracking engine determines the motion prediction error based on the degree of overlap between the target bounding box in the current frame and the expected bounding box predicted based on the historical trajectory of the target to be tracked. The tracking engine calculates the overlap between each pair of independent bounding boxes associated with each target sensor and calculates the average value to generate a multi-source consistency score. The target sensor is the image sensor that outputs the target to be tracked. The tracking engine determines the tracking confidence level corresponding to the target bounding box based on the appearance similarity score, the motion prediction error, and the multi-source consistency score.

10. The system according to claim 8 or 9, characterized in that, The target tracking status includes high-confidence stable tracking status, stable status, preliminary confirmation status, and failure status; In response to the target tracking state being a high-confidence stable tracking state, the central processing unit controls the auxiliary sensor to enter a low-power standby state and maintains the tracking state of the monitoring sensor; In response to the target tracking state being stable, the central processing unit controls the monitoring sensor and some auxiliary sensors to work together, and the auxiliary sensors determine the operation based on the latest ambient light. In response to the target tracking status being in the preliminary confirmation state, the central processing unit controls the on-duty sensor and all auxiliary sensors to perform fusion detection, and dynamically optimizes the weight allocation of each sensor according to the latest ambient light. In response to the target tracking status being in a failed state, the central processing unit controls the auxiliary sensor to enter a low-power standby state, controls the guard sensor to switch to a low frame rate wide-area survey mode, and, after re-acquiring the target, the central processing unit dynamically adjusts the guard sensor and the auxiliary sensor based on the latest ambient light provided by the ambient light sensor.

Images