A dual-mode target tracking system for infrared images

Abstract

This invention discloses a dual-mode target tracking system based on infrared images, specifically relating to the field of computer vision technology. It includes a multimodal sensing end and a dual-mode target tracking model. The sensing end acquires and preprocesses visible light and infrared images. The tracking model analyzes the reliability of the two modes in real time through a confidence assessment module. Based on the assessment results, a mode switching control module dynamically drives a joint tracking filtering module to switch between single visible light, single infrared, and multimodal fusion tracking modes. When a target is determined to be lost, the system schedules a cross-modal reacquisition module, prioritizing the use of complementary modal images and heterogeneous features for target search. This invention solves the problem of single-band tracking failure in complex environments by improving the continuity and robustness of target tracking through smooth switching and cross-modal compensation mechanisms.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a dual-mode target tracking system for infrared images. Background Technology

[0002] Image-guided weapons play an increasingly important role in modern warfare. Among them, the KDD missile, which adopts a terminal image guidance system, acquires image information of the target area through the seeker to achieve precise tracking and strike of the target. The terminal acquisition and control phase of the KDD missile often adopts the "human-in-the-loop" control mode, in which the pilot performs target search, lock and track correction through the seeker image. This process places extremely high demands on the reliability and robustness of the seeker tracking system.

[0003] Currently, the KDD missile's seeker tracking system mainly adopts a single infrared band imaging system. Infrared imaging guidance has advantages such as all-weather operation capability and good anti-jamming performance, and can meet basic tracking requirements in conventional combat environments. However, a single infrared seeker has inherent technical limitations in actual battlefield environments.

[0004] In existing technologies, traditional image trackers, during operation, will directly enter a "tracking lost" state when no image region with a correlation higher than a specified threshold is found in the seeker image. This design is essentially a "lock or lose" binary processing mechanism, lacking adaptability to environmental changes and fault tolerance and recovery capabilities for tracking failures. Specifically, when short-term interference occurs, such as smoke interference, infrared decoy release, or sudden changes in illumination, even if the interference lasts only a few image frames, existing systems will still determine that tracking is lost, requiring the pilot to re-search for and lock onto the target. At the critical moment of the terminal attack, this process not only delays the flight of the aircraft but may even lead to the failure of the attack mission. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a dual-mode target tracking system based on infrared images to address the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a dual-mode target tracking system for infrared images, comprising a multimodal sensing end and a dual-mode target tracking model; the multimodal sensing end includes a visible light imaging module, an infrared imaging module, and an image preprocessing module; the dual-mode target tracking model includes a multimodal confidence assessment module, a mode switching control module, a joint tracking filtering module, and a cross-modal reacquisition module; the multimodal sensing end is connected to the dual-mode target tracking model via a data bus.

[0007] The visible light imaging module and infrared imaging module of the multimodal sensing end respectively acquire visible light image data and infrared image data, and transmit the acquired image data to the image preprocessing module for processing;

[0008] After preprocessing the received image data, the image preprocessing module sends the preprocessed visible light image data and infrared image data to each module of the dual-mode target tracking model through the data bus.

[0009] The multimodal confidence evaluation module in the dual-mode target tracking model evaluates the confidence of the current visible light tracking mode and the confidence of the current infrared tracking mode in real time based on the received preprocessed image data, and transmits the evaluation results to the mode switching control module.

[0010] The mode switching control module dynamically selects and controls the joint tracking filter module to work in single visible light tracking mode, single infrared tracking mode, or multimodal fusion tracking mode based on the evaluation results of the multimodal confidence assessment module, and transmits the instructions to the joint tracking filter module.

[0011] The joint tracking filtering module performs tracking filtering on the target in the selected tracking mode according to the instructions of the mode switching control module, and outputs target status information. When the joint tracking filtering module determines that the target tracking is lost in the current tracking mode, the mode switching control module schedules the cross-modal re-acquisition module to re-acquire the target using the pre-processed image data provided by the multimodal sensing end. The re-acquisition process preferentially uses image data from another modality that is different from the lost mode.

[0012] Preferably, as a preferred embodiment of the dual-mode target tracking system for infrared images described in this invention, it includes a visible light imaging module and an infrared imaging module at the multimodal sensing end, which respectively acquire visible light image data and infrared image data, and transmit the acquired image data to the image preprocessing module for processing, specifically including the following:

[0013] The visible light imaging module and the infrared imaging module are controlled by the same hardware trigger pulse to achieve frame-synchronous acquisition of dual-channel images and capture visible light image data and infrared image data of the target area.

[0014] The image preprocessing module receives image data from the visible light imaging module and the infrared imaging module, and preprocesses each frame of the visible light image. With infrared images By assigning the same time series label, spatial registration is performed on them to calculate the coordinate mapping relationship between the visible light image coordinate system and the infrared image coordinate system, and spatial registration parameters are generated.

[0015] According to the spatial registration parameters, the visible light image and the infrared image are transformed to the same reference coordinate system to form a spatially aligned dual-modal image pair. The spatially aligned dual-modal images are then filtered, denoised, and enhanced. The processed visible light image data, infrared image data, and the spatial registration parameters are then sent to each module of the dual-mode target tracking model via the data bus.

[0016] Preferably, as a preferred embodiment of the dual-mode target tracking system for infrared images described in this invention, it includes the multimodal confidence assessment module evaluating the confidence level of the current visible light tracking mode and the confidence level of the current infrared tracking mode in real time based on the received preprocessed image data, and transmitting the evaluation results to the mode switching control module, specifically including the following:

[0017] The system receives preprocessed visible light and infrared image data, and performs multi-dimensional evaluation of the visible light and infrared tracking states of the current frame. It extracts visible light and infrared image evaluation metrics, normalizes these metrics, and then uses a weighted fusion method to generate the confidence level of the visible light tracking mode. and the confidence level of the current infrared tracking mode and will and Transmitted to the mode switching control module;

[0018] Furthermore, the visible light image evaluation metrics include: image contrast metrics, edge intensity metrics, target region grayscale distribution entropy metrics, and illumination intensity metrics. The image contrast metrics use the standard deviation of the image to measure local contrast for visible light image patches within the tracking area. Its contrast ratio is defined as The edge intensity index is calculated using an edge detection method, which measures the average value of the image gradient magnitude for visible light image patches within the tracking area. The edge strength index is The target area grayscale distribution entropy index is used to measure the uniformity and information content of the grayscale distribution within the target area. The illumination intensity index is represented by the average brightness value of the visible light image patch within the tracking area, and the specific formula is as follows: ,in, It is the size of the image patch. It is the average gray value of the image patch. and These represent the gradients of the image in the horizontal and vertical directions, respectively.

[0019] Furthermore, the infrared image evaluation metrics include: target-background temperature difference metric, thermal gradient distribution metric, and saturation pixel ratio metric. The target-background temperature difference metric measures the average grayscale difference between the target area and the surrounding background area, and the specific formula is as follows: The thermal gradient distribution index measures the drastic change in heat in a target region within an infrared image, reflecting the thermal structural details of the target. The specific formula is as follows: The saturation pixel ratio index is the proportion of pixels in the target area whose grayscale value reaches the image sensor's saturation value to the total number of pixels in the target area. The specific formula is as follows: ;in, It is the average gray value of the target area in the infrared image. It is the average gray value of the background area surrounding the target in the infrared image. The pixel grayscale values ​​of the infrared image. Infrared image At pixel The gradient along the horizontal direction, Infrared image At pixel The gradient along the vertical direction, To achieve a preset saturation threshold for grayscale values ​​within the target area. The number of pixels, This represents the total number of pixels within the target area.

[0020] Preferably, as a preferred embodiment of the dual-mode target tracking system for infrared images described in this invention, the system includes a mode switching control module that dynamically selects and controls the joint tracking filter module to operate in single visible light tracking mode, single infrared tracking mode, or multimodal fusion tracking mode based on the evaluation results of the multimodal confidence assessment module, and transmits instructions to the joint tracking filter module. Specifically, this includes the following:

[0021] Receive evaluation results from the multimodal confidence evaluation module, including the confidence level of the visible light tracking mode in the current frame. Confidence level of infrared tracking mode And internally preset the first confidence threshold. Second confidence threshold Satisfying the relationship ;

[0022] When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When the current visible light image quality is high and the infrared image quality is low, the mode switching control module generates a switching instruction with mode code "01" and transmits the instruction to the joint tracking filter module to control it to switch to single visible light tracking mode.

[0023] When the confidence level of the infrared tracking mode And the confidence level of the visible light tracking mode When the current infrared image quality is high and the visible light image quality is low, the mode switching control module generates a switching instruction with mode code "10" and transmits the instruction to the joint tracking filter module to control it to switch to single infrared tracking mode.

[0024] When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When it is determined that the quality of both the visible light image and the infrared image is at a high level, the mode switching control module generates a switching instruction with a mode code of "00" and transmits the instruction to the joint tracking filter module to control it to switch to the multimodal fusion tracking mode.

[0025] When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When the current visible light and infrared image quality are both determined to be at a low level, the mode switching control module generates a switching command with mode code "11", enters the low confidence warning mode, and performs the following operations:

[0026] Send a hold command to the joint tracking filter module to maintain the target state of the previous frame and not perform observation updates;

[0027] Send a recapture request to the cross-modal recapture module to resume target tracking in any modality;

[0028] When the low confidence warning mode continues to exceed the preset frame threshold of 30 frames, a manual intervention request signal is sent to the operator.

[0029] A time-domain smoothing mechanism is introduced to avoid frequent mode jumps near the confidence threshold, and a switching rise threshold is set. and switch drop threshold Confidence of the visible light tracking mode in the current frame Confidence level of infrared tracking mode All are within the lag interval corresponding to their respective modes. If the condition does not meet the hard switching conditions of the above four modes, the mode switching control module will keep the current mode code unchanged. This is the preset hysteresis width.

[0030] Preferably, as a preferred embodiment of the dual-mode target tracking system for infrared images according to the present invention, it includes the joint tracking filtering module performing tracking filtering processing on the target in the selected tracking mode according to the instructions of the mode switching control module, and outputting target status information; when the joint tracking filtering module determines that the target tracking is lost in the current tracking mode, the mode switching control module schedules the cross-modal re-acquisition module to re-acquire the target using the preprocessed image data provided by the multimodal sensing end. The re-acquisition process preferentially uses image data from another modality different from the lost mode, specifically including the following:

[0031] The module receives a mode switching command from the mode switching control module and switches to the corresponding tracking mode according to the working mode code in the command. The joint tracking filtering module includes a visible light tracking unit, an infrared tracking unit, a multi-modal fusion tracking unit, a tracking status monitoring unit, and a tracking loss determination unit.

[0032] The visible light tracking unit is activated upon receiving a switching command with mode code "01". In single visible light tracking mode, it tracks the target based on visible light image data, employing a template matching tracking algorithm. Using the visible light image features of the target region as a template, it searches for the best matching position in the visible light image of the current frame and extracts the orientation gradient histogram features and color histogram features. The output visible light tracking result includes: the target center position coordinates. Target Scale and tracking confidence And send it to the tracking status monitoring unit;

[0033] The infrared tracking unit is activated upon receiving a switching command with mode code "10". In single infrared tracking mode, it tracks the target based on infrared image data, employing a template matching tracking algorithm. Using the infrared image features of the target region as a template, it searches for the optimal matching position in the current frame's infrared image and extracts thermal gradient distribution features and local binary pattern features. The output infrared tracking result includes the target center coordinates. Target Scale and tracking confidence And send it to the tracking status monitoring unit;

[0034] The multimodal fusion tracking unit is activated upon receiving a switching command with mode code "00". In multimodal fusion tracking mode, it fuses visible light tracking results and infrared tracking results, generates fused target state information using a Bayesian fusion method, and outputs the fused target state information, including the fused target position. , Integration of target scale and fusion confidence And send it to the tracking status monitoring unit, wherein, The visible light tracking confidence score for the current frame. The infrared tracking confidence level for the current frame;

[0035] The tracking status monitoring unit receives the tracking results output by the visible light tracking unit, the infrared tracking unit, and the multimodal fusion tracking unit, directly obtains the current tracking confidence level, and calculates the tracking error. ,in, For actual tracking location, The position predicted by the particle filter;

[0036] The tracking loss determination unit is used to determine the tracking loss based on tracking confidence, tracking error, and a preset tracking loss threshold. The system determines whether the target has lost tracking in the current tracking mode and generates a tracking loss signal. The determination logic includes:

[0037] (1) When the tracking confidence level in the current mode is lower than the tracking loss threshold, tracking is determined to be lost. The tracking loss threshold is... The value ranges from 0.3 to 0.5. In this embodiment... ;

[0038] (2) The tracking error exceeds the preset maximum permissible error: The tracking was determined to be lost, among which The value should be 0.3 to 0.5 times the target scale;

[0039] When tracking loss is detected, the tracking loss detection unit generates a tracking loss signal and sends the signal to the mode switching control module. The mode switching control module receives the tracking loss signal and queries the current multimodal confidence assessment results, including the latest visible light tracking confidence and infrared tracking confidence values. Based on the retrieved confidence assessment results, it determines the currently available confidence modes.

[0040] when , The cross-modal recapture module is scheduled to use a high-confidence mode to guide the lost mode for recapture.

[0041] when and It directly sends a manual intervention request to the operator without performing automatic recapture.

[0042] The determination result and scheduling instruction are sent to the cross-modal re-acquisition module. The cross-modal re-acquisition module uses the preprocessed image data provided by the multimodal sensing end to re-acquire the target. The re-acquisition process preferentially uses image data from another modality different from the lost mode, specifically including:

[0043] Initial search region construction: Obtain the target state of the previous lost frame and establish candidate search regions at the corresponding spatial locations in another modality;

[0044] If the image is lost in single visible light mode, retrieve the infrared preprocessed image at the same time stamp and perform full-image matching using the local binary mode of the infrared image; if the image is lost in single infrared mode, retrieve the visible light preprocessed image at the same time stamp and perform matching using color histogram and orientation gradient histogram features to find the target.

[0045] After the matching is completed, the cross-modal recapture module calculates the recapture confidence score. When the confidence score reaches a preset threshold, the module will recapture the target value. The output of the reacquisition target center position is fed back to the mode switching control module, which then instructs the tracking filter module to reinitialize the filter parameters and start the tracking process.

[0046] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements the functional modules of a dual-mode infrared image target tracking system as described above.

[0047] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements a dual-mode target tracking system for infrared images as described above.

[0048] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0049] By combining multimodal confidence assessment and mode switching control, adaptive dynamic switching between visible light and infrared tracking modes is achieved, effectively solving the bottleneck problem of single sensors being prone to failure under drastic changes in lighting, smoke obstruction, and complex backgrounds. The introduction of a time-domain smoothing mechanism and hysteresis width ensures the smoothness of tracking commands and the stability of system output. This invention has a unique cross-modal reacquisition function, which can quickly utilize the heterogeneous features of complementary modes for guided search when the target is lost, significantly improving the robustness and reliability of the system in extreme environments. Through early warning and manual intervention mechanisms, a complete all-weather, all-scenario target tracking closed loop is constructed. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0051] Figure 1 This is a flowchart of a dual-mode target tracking system for infrared images according to the present invention.

[0052] Figure 2 This is a diagram showing the working state transition of the mode switching control module provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0055] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0056] Example 1

[0057] This embodiment provides, for example Figure 1 The diagram illustrates a dual-mode target tracking system based on infrared images, specifically comprising a multimodal sensing end and a dual-mode target tracking model. The multimodal sensing end includes a visible light imaging module, an infrared imaging module, and an image preprocessing module. The dual-mode target tracking model includes a multimodal confidence assessment module, a mode switching control module, a joint tracking filtering module, and a cross-modal reacquisition module. The multimodal sensing end is connected to the dual-mode target tracking model via a data bus.

[0058] The visible light imaging module and infrared imaging module of the multimodal sensing end respectively acquire visible light image data and infrared image data, and transmit the acquired image data to the image preprocessing module for processing;

[0059] After preprocessing the received image data, the image preprocessing module sends the preprocessed visible light image data and infrared image data to each module of the dual-mode target tracking model through the data bus.

[0060] The multimodal confidence evaluation module in the dual-mode target tracking model evaluates the confidence of the current visible light tracking mode and the confidence of the current infrared tracking mode in real time based on the received preprocessed image data, and transmits the evaluation results to the mode switching control module.

[0061] The mode switching control module dynamically selects and controls the joint tracking filter module to work in single visible light tracking mode, single infrared tracking mode, or multimodal fusion tracking mode based on the evaluation results of the multimodal confidence assessment module, and transmits the instructions to the joint tracking filter module.

[0062] The joint tracking filtering module performs tracking filtering on the target in the selected tracking mode according to the instructions of the mode switching control module, and outputs target status information. When the joint tracking filtering module determines that the target tracking is lost in the current tracking mode, the mode switching control module schedules the cross-modal re-acquisition module to re-acquire the target using the pre-processed image data provided by the multimodal sensing end. The re-acquisition process preferentially uses image data from another modality that is different from the lost mode.

[0063] In this embodiment, the multimodal sensing end specifically refers to the visible light imaging module and the infrared imaging module of the multimodal sensing end, which respectively acquire visible light image data and infrared image data, and transmit the acquired image data to the image preprocessing module for processing, specifically including the following:

[0064] The visible light imaging module and the infrared imaging module are controlled by the same hardware trigger pulse to achieve frame-synchronous acquisition of dual-channel images and capture visible light image data and infrared image data of the target area.

[0065] The image preprocessing module receives image data from the visible light imaging module and the infrared imaging module, and preprocesses each frame of the visible light image. With infrared images By assigning the same time series label, spatial registration is performed on them to calculate the coordinate mapping relationship between the visible light image coordinate system and the infrared image coordinate system, and spatial registration parameters are generated.

[0066] According to the spatial registration parameters, the visible light image and the infrared image are transformed to the same reference coordinate system to form a spatially aligned dual-modal image pair. The spatially aligned dual-modal images are then filtered, denoised, and enhanced. The processed visible light image data, infrared image data, and the spatial registration parameters are then sent to each module of the dual-mode target tracking model via the data bus.

[0067] In this embodiment, the multimodal confidence assessment module is specifically described. Based on the received preprocessed image data, this module evaluates the confidence levels of the current visible light tracking mode and the current infrared tracking mode in real time, and transmits the evaluation results to the mode switching control module. Specifically, this includes the following:

[0068] The system receives preprocessed visible light and infrared image data, and performs multi-dimensional evaluation of the visible light and infrared tracking states of the current frame. It extracts visible light and infrared image evaluation metrics, normalizes these metrics, and then uses a weighted fusion method to generate the confidence level of the visible light tracking mode. and the confidence level of the current infrared tracking mode and will and Transmitted to the mode switching control module;

[0069] Furthermore, the visible light image evaluation metrics include: image contrast metrics, edge intensity metrics, target region grayscale distribution entropy metrics, and illumination intensity metrics. The image contrast metrics use the standard deviation of the image to measure local contrast for visible light image patches within the tracking area. Its contrast ratio is defined as The edge intensity index is calculated using an edge detection method, which measures the average value of the image gradient magnitude for visible light image patches within the tracking area. The edge strength index is The target area grayscale distribution entropy index is used to measure the uniformity and information content of the grayscale distribution within the target area. The illumination intensity index is represented by the average brightness value of the visible light image patch within the tracking area, and the specific formula is as follows: ,in, It is the size of the image patch. It is the average gray value of the image patch. and These represent the gradients of the image in the horizontal and vertical directions, respectively.

[0070] Furthermore, the infrared image evaluation metrics include: target-background temperature difference metric, thermal gradient distribution metric, and saturation pixel ratio metric. The target-background temperature difference metric measures the average grayscale difference between the target area and the surrounding background area, and the specific formula is as follows: The thermal gradient distribution index measures the drastic change in heat in a target region within an infrared image, reflecting the thermal structural details of the target. The specific formula is as follows: The saturation pixel ratio index is the proportion of pixels in the target area whose grayscale value reaches the image sensor's saturation value to the total number of pixels in the target area. The specific formula is as follows: ;in, It is the average gray value of the target area in the infrared image. It is the average gray value of the background area surrounding the target in the infrared image. The pixel grayscale values ​​of the infrared image. Infrared image At pixel The gradient along the horizontal direction, Infrared image At pixel The gradient along the vertical direction, To achieve a preset saturation threshold for grayscale values ​​within the target area. The number of pixels, The total number of pixels within the target area; saturation threshold. The value is taken as 90% to 100% of the maximum grayscale value of the image sensor.

[0071] In this embodiment, the mode switching control module is specifically described. Based on the evaluation results of the multimodal confidence assessment module, the mode switching control module dynamically selects and controls the joint tracking filter module to operate in single visible light tracking mode, single infrared tracking mode, or multimodal fusion tracking mode, and transmits instructions to the joint tracking filter module. Specifically, this includes the following:

[0072] Receive evaluation results from the multimodal confidence evaluation module, including the confidence level of the visible light tracking mode in the current frame. Confidence level of infrared tracking mode And internally preset the first confidence threshold. Second confidence threshold Satisfying the relationship ;

[0073] When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When the current visible light image quality is high and the infrared image quality is low, the mode switching control module generates a switching instruction with mode code "01" and transmits the instruction to the joint tracking filter module to control it to switch to single visible light tracking mode.

[0074] When the confidence level of the infrared tracking mode And the confidence level of the visible light tracking mode When the current infrared image quality is high and the visible light image quality is low, the mode switching control module generates a switching instruction with mode code "10" and transmits the instruction to the joint tracking filter module to control it to switch to single infrared tracking mode.

[0075] When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When it is determined that the quality of both the visible light image and the infrared image is at a high level, the mode switching control module generates a switching instruction with a mode code of "00" and transmits the instruction to the joint tracking filter module to control it to switch to the multimodal fusion tracking mode.

[0076] When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When the current visible light and infrared image quality are both determined to be at a low level, the mode switching control module generates a switching command with mode code "11", enters the low confidence warning mode, and performs the following operations:

[0077] Send a hold command to the joint tracking filter module to maintain the target state of the previous frame and not perform observation updates;

[0078] Send a recapture request to the cross-modal recapture module to resume target tracking in any modality;

[0079] When the low confidence warning mode continues to exceed the preset frame threshold of 30 frames, a manual intervention request signal is sent to the operator.

[0080] A time-domain smoothing mechanism is introduced to avoid frequent mode jumps near the confidence threshold, and a switching rise threshold is set. and switch drop threshold Confidence of the visible light tracking mode in the current frame Confidence level of infrared tracking mode All are within the lag interval corresponding to their respective modes. If the condition does not meet the hard switching conditions of the above four modes, the mode switching control module will keep the current mode code unchanged. The preset hysteresis width has a value range of 0.05 to 0.1. In this embodiment... ,like Figure 2 As shown.

[0081] In this embodiment, the joint tracking filtering module is specifically described. According to the instructions of the mode switching control module, the joint tracking filtering module performs tracking filtering on the target in the selected tracking mode and outputs target status information. When the joint tracking filtering module determines that target tracking is lost in the current tracking mode, the mode switching control module schedules the cross-modal re-acquisition module to re-acquire the target using the pre-processed image data provided by the multimodal sensing end. The re-acquisition process preferentially uses image data from another modality different from the lost mode, specifically including the following:

[0082] The module receives a mode switching command from the mode switching control module and switches to the corresponding tracking mode according to the working mode code in the command. The joint tracking filtering module includes a visible light tracking unit, an infrared tracking unit, a multi-modal fusion tracking unit, a tracking status monitoring unit, and a tracking loss determination unit.

[0083] The visible light tracking unit is activated upon receiving a switching command with mode code "01". In single visible light tracking mode, it tracks the target based on visible light image data, employing a template matching tracking algorithm. Using the visible light image features of the target region as a template, it searches for the best matching position in the visible light image of the current frame and extracts the orientation gradient histogram features and color histogram features. The output visible light tracking result includes: the target center position coordinates. Target Scale and tracking confidence And send it to the tracking status monitoring unit;

[0084] The infrared tracking unit is activated upon receiving a switching command with mode code "10". In single infrared tracking mode, it tracks the target based on infrared image data, employing a template matching tracking algorithm. Using the infrared image features of the target region as a template, it searches for the optimal matching position in the current frame's infrared image and extracts thermal gradient distribution features and local binary pattern features. The output infrared tracking result includes the target center coordinates. Target Scale and tracking confidence And send it to the tracking status monitoring unit;

[0085] The multimodal fusion tracking unit is activated upon receiving a switching command with mode code "00". In multimodal fusion tracking mode, it fuses visible light tracking results and infrared tracking results, generates fused target state information using a Bayesian fusion method, and outputs the fused target state information, including the fused target position. , Integration of target scale and fusion confidence And send it to the tracking status monitoring unit, wherein, The visible light tracking confidence score for the current frame. The infrared tracking confidence level for the current frame;

[0086] The tracking status monitoring unit receives the tracking results output by the visible light tracking unit, the infrared tracking unit, and the multimodal fusion tracking unit, directly obtains the current tracking confidence level, and calculates the tracking error. ,in, For actual tracking location, The position predicted by the particle filter;

[0087] The tracking loss determination unit is used to determine the tracking loss based on tracking confidence, tracking error, and a preset tracking loss threshold. The system determines whether the target has lost tracking in the current tracking mode and generates a tracking loss signal. The determination logic includes:

[0088] (1) When the tracking confidence level in the current mode is lower than the tracking loss threshold, tracking is determined to be lost, wherein the tracking loss threshold is: The value ranges from 0.3 to 0.5. In this embodiment... ;

[0089] (2) Tracking error Exceeding the preset maximum allowable error : The tracking was determined to be lost, among which The value should be 0.3 to 0.5 times the target scale;

[0090] When tracking loss is detected, the tracking loss detection unit generates a tracking loss signal and sends the signal to the mode switching control module. The mode switching control module receives the tracking loss signal and queries the current multimodal confidence assessment results, including the latest visible light tracking confidence and infrared tracking confidence values. Based on the retrieved confidence assessment results, it determines the currently available confidence modes.

[0091] when , The cross-modal recapture module is scheduled to use a high-confidence mode to guide the lost mode for recapture.

[0092] when and It directly sends a manual intervention request to the operator without performing automatic recapture.

[0093] The determination result and scheduling instruction are sent to the cross-modal re-acquisition module. The cross-modal re-acquisition module uses the preprocessed image data provided by the multimodal sensing end to re-acquire the target. The re-acquisition process preferentially uses image data from another modality different from the lost mode, specifically including:

[0094] Initial search region construction: Obtain the target state of the previous lost frame and establish candidate search regions at the corresponding spatial locations in another modality;

[0095] If the image is lost in single visible light mode, retrieve the infrared preprocessed image at the same time stamp and perform full-image matching using the local binary mode of the infrared image; if the image is lost in single infrared mode, retrieve the visible light preprocessed image at the same time stamp and perform matching using color histogram and orientation gradient histogram features to find the target.

[0096] After the matching is completed, the cross-modal recapture module calculates the recapture confidence score. When the confidence score reaches a preset threshold, the module will recapture the target value. The output of the reacquisition target center position is fed back to the mode switching control module, which then instructs the tracking filter module to reinitialize the filter parameters and start the tracking process.

[0097] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0098] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the functional modules of a dual-mode target tracking system for infrared images as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0099] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A dual-mode target tracking system based on infrared images, characterized in that: The system includes a multimodal sensing terminal and a dual-mode target tracking model. The multimodal sensing terminal includes a visible light imaging module, an infrared imaging module, and an image preprocessing module. The dual-mode target tracking model includes a multimodal confidence assessment module, a mode switching control module, a joint tracking filtering module, and a cross-modal reacquisition module. The multimodal sensing terminal is connected to the dual-mode target tracking model via a data bus. The visible light imaging module and infrared imaging module of the multimodal sensing end respectively acquire visible light image data and infrared image data, and transmit the acquired image data to the image preprocessing module for processing; After preprocessing the received image data, the image preprocessing module sends the preprocessed visible light image data and infrared image data to each module of the dual-mode target tracking model through the data bus. The multimodal confidence evaluation module in the dual-mode target tracking model evaluates the confidence of the current visible light tracking mode and the confidence of the current infrared tracking mode in real time based on the received preprocessed image data, and transmits the evaluation results to the mode switching control module. The mode switching control module dynamically selects and controls the joint tracking filter module to work in single visible light tracking mode, single infrared tracking mode, or multimodal fusion tracking mode based on the evaluation results of the multimodal confidence assessment module, and transmits the instructions to the joint tracking filter module. The joint tracking filtering module performs tracking filtering on the target in the selected tracking mode according to the instructions of the mode switching control module, and outputs target status information. When the joint tracking filtering module determines that the target tracking is lost in the current tracking mode, the mode switching control module schedules the cross-modal re-acquisition module to re-acquire the target using the preprocessed image data provided by the multimodal sensing end. The re-acquisition process preferentially uses image data from another modality that is different from the lost mode.

2. The dual-mode target tracking system for infrared images according to claim 1, characterized in that: The visible light imaging module and the infrared imaging module are controlled by the same hardware trigger pulse to achieve frame-synchronous acquisition of dual-channel images and capture visible light image data and infrared image data of the target area. The image preprocessing module receives image data from the visible light imaging module and the infrared imaging module, and preprocesses each frame of the visible light image. With infrared images By assigning the same time series label, spatial registration is performed on them to calculate the coordinate mapping relationship between the visible light image coordinate system and the infrared image coordinate system, and spatial registration parameters are generated. According to the spatial registration parameters, the visible light image and the infrared image are transformed to the same reference coordinate system to form a spatially aligned dual-modal image pair. The spatially aligned dual-modal images are then filtered, denoised, and enhanced. The processed visible light image data, infrared image data, and the spatial registration parameters are then sent to each module of the dual-mode target tracking model via the data bus.

3. The dual-mode target tracking system for infrared images according to claim 1, characterized in that: The multimodal confidence assessment module evaluates the confidence levels of the current visible light tracking mode and the current infrared tracking mode in real time based on the received preprocessed image data, and transmits the evaluation results to the mode switching control module. Specifically, this includes the following: The system receives preprocessed visible light and infrared image data, and performs multi-dimensional evaluation of the visible light and infrared tracking states of the current frame. It extracts visible light and infrared image evaluation metrics, normalizes these metrics, and then uses a weighted fusion method to generate the confidence level of the visible light tracking mode. and the confidence level of the current infrared tracking mode and will and Transmitted to the mode switching control module; The visible light image evaluation metrics include: image contrast index, edge intensity index, target region gray-level distribution entropy index, and illumination intensity index. The image contrast index uses the standard deviation of the image to measure local contrast for visible light image patches within the tracking area. Its contrast ratio is defined as The edge intensity index is calculated using an edge detection method, which measures the average value of the image gradient magnitude for visible light image patches within the tracking area. The edge strength index is The target area grayscale distribution entropy index is used to measure the uniformity and information content of the grayscale distribution within the target area. The illumination intensity index is represented by the average brightness value of the visible light image patch within the tracking area, and the specific formula is as follows: ,in, It is the size of the image patch. It is the average gray value of the image patch. and These represent the gradients of the image in the horizontal and vertical directions, respectively.

4. The dual-mode target tracking system for infrared images according to claim 3, characterized in that: The infrared image evaluation metrics include: target-background temperature difference metric, thermal gradient distribution metric, and saturation pixel ratio metric. The target-background temperature difference metric measures the average grayscale difference between the target area and the surrounding background area, and the specific formula is as follows: The thermal gradient distribution index measures the degree of thermal change in a target area in an infrared image, and the specific formula is as follows: The saturation pixel ratio index is the proportion of pixels in the target area whose grayscale value reaches the image sensor's saturation value to the total number of pixels in the target area. The specific formula is as follows: ;in, It is the average gray value of the target area in the infrared image. It is the average gray value of the background area surrounding the target in the infrared image. These are the pixel grayscale values ​​of the infrared image. Infrared image At pixel The gradient along the horizontal direction, Infrared image At pixel The gradient along the vertical direction, To achieve a preset saturation threshold for grayscale values ​​within the target area. The number of pixels, This represents the total number of pixels within the target area.

5. The dual-mode target tracking system for infrared images according to claim 1, characterized in that: The mode switching control module dynamically selects and controls the joint tracking filter module to operate in single visible light tracking mode, single infrared tracking mode, or multimodal fusion tracking mode based on the evaluation results of the multimodal confidence assessment module, and transmits instructions to the joint tracking filter module, specifically including the following: Receive evaluation results from the multimodal confidence evaluation module, including the confidence level of the visible light tracking mode in the current frame. Confidence level of infrared tracking mode And internally preset the first confidence threshold. Second confidence threshold Satisfying the relationship ; When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When the current visible light image quality is high and the infrared image quality is low, the mode switching control module generates a switching instruction with mode code "01" and transmits the instruction to the joint tracking filter module to control it to switch to single visible light tracking mode. When the confidence level of the infrared tracking mode And the confidence level of the visible light tracking mode When the current infrared image quality is high and the visible light image quality is low, the mode switching control module generates a switching instruction with mode code "10" and transmits the instruction to the joint tracking filter module to control it to switch to single infrared tracking mode. When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When it is determined that the quality of both the visible light image and the infrared image is at a high level, the mode switching control module generates a switching instruction with a mode code of "00" and transmits the instruction to the joint tracking filter module to control it to switch to the multimodal fusion tracking mode. When the confidence level of the visible light tracking mode And the confidence level of the infrared tracking mode When the visible light and infrared image quality are both determined to be at a low level, the mode switching control module generates a switching instruction with mode code "11" to enter the low confidence warning mode.

6. The dual-mode target tracking system for infrared images according to claim 5, characterized in that: The system enters the low-confidence warning mode and performs the following operations: Send a hold command to the joint tracking filter module to maintain the target state of the previous frame and not perform observation updates; Send a recapture request to the cross-modal recapture module to resume target tracking in any modality; When the low confidence warning mode continues to exceed the preset frame threshold, a manual intervention request signal is sent to the operator. A time-domain smoothing mechanism is introduced to avoid frequent mode jumps near the confidence threshold, and a switching rise threshold is set. and switch drop threshold When the confidence level of the visible light tracking mode of the current frame Confidence level of infrared tracking mode All are within the lag interval corresponding to their respective modes. If the condition does not meet the hard switching conditions of the above four modes, the mode switching control module will keep the current mode code unchanged. This is the preset hysteresis width.

7. The dual-mode target tracking system for infrared images according to claim 1, characterized in that: The joint tracking filter module specifically includes the following: The module receives a mode switching command from the mode switching control module and switches to the corresponding tracking mode according to the working mode code in the command. The joint tracking filtering module includes a visible light tracking unit, an infrared tracking unit, a multi-modal fusion tracking unit, a tracking status monitoring unit, and a tracking loss determination unit. The visible light tracking unit is activated when it receives a switching instruction with mode code "01". In single visible light tracking mode, it tracks the target based on visible light image data, uses a template matching tracking algorithm, uses the visible light image features of the target area as a template, searches for the best matching position in the visible light image of the current frame, and extracts the orientation gradient histogram features and color histogram features. Output visible light tracking results, including: target center position coordinates Target Scale and tracking confidence And send it to the tracking status monitoring unit; The infrared tracking unit is activated upon receiving a switching command with mode code "10". In single infrared tracking mode, it tracks the target based on infrared image data, employing a template matching tracking algorithm. Using the infrared image features of the target region as a template, it searches for the optimal matching position in the current frame's infrared image and extracts thermal gradient distribution features and local binary pattern features. The output infrared tracking result includes the target center coordinates. Target Scale and tracking confidence And send it to the tracking status monitoring unit; The multimodal fusion tracking unit is activated upon receiving a switching command with mode code "00". In multimodal fusion tracking mode, it fuses visible light tracking results and infrared tracking results, generates fused target state information using a Bayesian fusion method, and outputs the fused target state information, including the fused target position. , Integration of target scale and fusion confidence And send it to the tracking status monitoring unit, wherein, The visible light tracking confidence score for the current frame. The infrared tracking confidence level for the current frame; The tracking status monitoring unit receives the tracking results output by the visible light tracking unit, the infrared tracking unit, and the multimodal fusion tracking unit, directly obtains the current tracking confidence level, and calculates the tracking error. ,in, For actual tracking location, The position predicted by the particle filter.

8. The dual-mode target tracking system for infrared images according to claim 7, characterized in that: The tracking loss determination unit is used to determine the tracking loss based on tracking confidence, tracking error, and a preset tracking loss threshold. The system determines whether the target has lost tracking in the current tracking mode and generates a tracking loss signal. The determination logic includes: (1) When the tracking confidence level in the current mode is lower than the tracking loss threshold, tracking is determined to be lost, wherein the tracking loss threshold is: ; (2) The tracking error exceeds the preset maximum permissible error: The tracking was determined to be lost, among which The value should be 0.3 to 0.5 times the target scale. When tracking loss is detected, the tracking loss detection unit generates a tracking loss signal and sends the signal to the mode switching control module. The mode switching control module receives the tracking loss signal and queries the current multimodal confidence assessment results, including the latest visible light tracking confidence and infrared tracking confidence values. Based on the retrieved confidence assessment results, it determines the currently available confidence modes. when , The cross-modal recapture module is scheduled to use a high-confidence mode to guide the lost mode for recapture. when and It directly sends a manual intervention request to the operator without performing automatic recapture. The determination result and scheduling instructions are sent to the cross-modal recapture module.

9. A dual-mode target tracking system for infrared images according to claim 7, characterized in that: The cross-modal recapture module utilizes the preprocessed image data provided by the multimodal sensing end to recapture the target. The recapture process prioritizes the use of image data from another modality different from the lost mode, specifically including: Initial search region construction: Obtain the target state of the previous frame that was lost, and establish candidate search regions at the corresponding spatial locations in another modality; If the image is lost in single visible light mode, retrieve the infrared preprocessed image at the same time stamp and perform full-image matching using the local binary mode of the infrared image; if the image is lost in single infrared mode, retrieve the visible light preprocessed image at the same time stamp and perform matching using color histogram and orientation gradient histogram features to find the target. After the matching is completed, the cross-modal recapture module calculates the recapture confidence score. When the confidence score reaches a preset threshold, the module will recapture the target value. The output of the reacquisition target center position is fed back to the mode switching control module, which then instructs the tracking filter module to reinitialize the filter parameters and start the tracking process.

Images