A multi-modal target automatic identification and tracking method and system based on an optoelectronic pod

Abstract

The present application relates to the technical field of optoelectronic pod target tracking, and discloses a multi-modal target automatic identification and tracking method and system based on an optoelectronic pod. The multi-modal data acquisition module of the system is based on an optoelectronic pod integrated with visible light, infrared thermal imaging and laser ranging sensors, acquires multi-modal sensing data of a target and completes space-time synchronization calibration; the dynamic feature extraction module processes multi-modal data using a multi-scale convolutional neural network, generates multi-level feature maps of the target and performs salient region labeling; the adaptive tracking decision module deploys a recurrent neural network to predict the motion trajectory of the target according to the labeling result, generates tracking control instructions through an optimization algorithm; the execution control module drives the optoelectronic pod pan-tilt mechanism, adjusts the pointing direction of the sensors and realizes target tracking; the feedback optimization module monitors the tracking state in real time, calculates tracking deviation indicators and generates strategy adjustment parameters to dynamically optimize the tracking strategy.

Description

Technical Field

[0001] This invention relates to the field of optoelectronic pod target tracking technology, specifically to a multimodal target automatic identification and tracking method and system based on optoelectronic pods. Background Technology

[0002] In numerous fields such as aerospace, security monitoring, forest fire prevention, and maritime search and rescue, electro-optical pods serve as crucial payload equipment, and their target recognition and tracking capabilities have always been a core focus of technological development. With the continuous expansion of application scenarios, increasingly higher demands are being placed on the accuracy, stability, and environmental adaptability of target recognition and tracking in electro-optical pods. Currently, most target recognition and tracking systems on the market rely on a single type of sensor for data acquisition, such as visible light sensors or infrared thermal imaging sensors. Systems using a single sensor often face significant limitations in complex environments: visible light sensors are significantly affected by lighting conditions, making it difficult to clearly capture target features in adverse weather conditions such as nighttime, heavy fog, and heavy rain, resulting in a substantial decrease in target recognition accuracy; while infrared thermal imaging sensors can identify targets in low-light environments, their ability to present detailed target features is weak, and target confusion easily occurs when multiple targets have similar thermal radiation characteristics; and using a laser rangefinder alone can only acquire distance information about the target, failing to fully present key information such as the target's shape and movement trends, making it difficult to support accurate target recognition and tracking decisions.

[0003] Even though some systems attempt to integrate multiple sensors to form a multimodal data acquisition mode, the problem of spatiotemporal asynchrony of multimodal data is common. Due to the differences in response speed and data acquisition frequency of different sensors, the acquired visible light data, infrared thermal imaging data, and laser ranging data often have delays in the time dimension and deviations in the spatial dimension. This makes it impossible to effectively fuse multimodal data, which not only makes it difficult to take advantage of multi-sensor collaboration, but may also affect the accuracy of target recognition due to data conflicts.

[0004] In the target feature extraction stage, traditional systems often employ simple feature extraction algorithms, such as edge detection and grayscale histograms. These algorithms can only extract shallow features of the target and cannot capture detailed information and deep features of the target at different scales. When the target is in a complex background or undergoes shape changes due to motion, shallow features are easily confused with background features, resulting in insufficient feature extraction accuracy and affecting the subsequent target recognition and tracking performance. Furthermore, existing systems lack effective filtering mechanisms for feature maps, failing to highlight salient areas of the target. This forces the subsequent tracking decision module to process a large amount of redundant information, reducing the system's operational efficiency.

[0005] In terms of tracking decision-making and execution, traditional systems often rely on pre-defined motion models, such as uniform linear motion or uniformly accelerated motion. These models struggle to adapt to the complex motion states of targets. When a target undergoes sudden movements such as changes in direction or speed, the pre-defined model cannot accurately predict its trajectory, leading to delayed tracking commands and subsequent tracking deviations. Furthermore, insufficient coordination between the execution control module and the tracking decision-making module results in the gimbal mechanism's adjustment speed and precision failing to match the requirements of tracking commands. This is especially problematic when the target is moving rapidly, as the sensor's pointing cannot be adjusted in a timely manner, further reducing tracking stability.

[0006] Existing systems generally lack effective feedback optimization mechanisms, and can only perform target identification and tracking according to fixed strategies, unable to dynamically adjust system parameters based on real-time tracking status. When environmental conditions change, or the target's motion state changes, leading to increased tracking deviation, the system cannot detect the problem and optimize it in time, and can only continue to operate according to the original strategy, ultimately resulting in tracking failure. This makes it difficult to meet the high reliability requirements for target identification and tracking in complex scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for automatic identification and tracking of multimodal targets based on an optoelectronic pod, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides a multimodal target automatic identification and tracking system based on an optoelectronic pod, the system comprising:

[0009] The multimodal data acquisition module integrates a visible light sensor, an infrared thermal imaging sensor, and a laser rangefinder based on an optoelectronic pod to acquire multimodal perception data of the target and perform spatiotemporal synchronization calibration on the multimodal perception data;

[0010] The dynamic feature extraction module applies a multi-scale convolutional neural network to multimodal sensing data to generate multi-level feature maps of the target and performs salient region annotation on the multi-level feature maps.

[0011] The adaptive tracking decision module deploys a recurrent neural network based on the salient region annotation results to predict the target's trajectory and generates tracking control commands through an optimization algorithm.

[0012] The execution control module drives the gimbal mechanism of the optoelectronic pod based on tracking control commands, and adjusts the sensor pointing to achieve target tracking;

[0013] The feedback optimization module monitors the target tracking status in real time, calculates the tracking deviation index, generates tracking strategy adjustment parameters, and dynamically optimizes the multimodal target automatic recognition and tracking strategy.

[0014] Preferably, the multimodal data acquisition module includes:

[0015] Three sensing devices—visible light sensor, infrared thermal imaging sensor, and laser ranging sensor—are deployed on the optoelectronic pod platform to collect visible light image data, infrared thermal radiation data, and laser ranging point cloud data of the target, respectively.

[0016] A simulated multi-sensor collaborative acquisition mechanism is used to dynamically adjust the sampling frequency and resolution. Based on prior information about the target, the core area of ​​the target is designated as the high sampling rate area, and the outer area of ​​the target is designated as the low sampling rate area. Sensor parameters are dynamically configured through control commands to obtain the final multimodal sensing data.

[0017] Preferably, the dynamic feature extraction module applies a multi-scale convolutional neural network to the multimodal sensing data to generate multi-level feature maps of the target, including:

[0018] The multimodal sensing data is normalized by mean, and the normalized multimodal sensing data is integrated into a multidimensional tensor according to the modal dimension.

[0019] A multi-scale convolutional neural network structure is constructed, including an encoding path and a decoding path. In the encoding path, a deep convolutional network is used to extract hierarchical features from the input multimodal sensing data to generate feature maps at different scales. In the decoding path, the spatial details of the high-level feature maps are restored through upsampling operations.

[0020] One-dimensional convolution is applied to each output layer of the multi-scale convolutional neural network structure to adjust the feature channels, and the feature maps of different scales output by the multi-scale convolutional neural network structure are fused to generate a multi-level feature map of the target.

[0021] Preferably, the dynamic feature extraction module performs salient region annotation on the multi-level feature map, including:

[0022] The feature response value of each region in the multi-level feature map is calculated by weighted fusion of visible light image data, infrared thermal radiation data and laser ranging point cloud data.

[0023] A first threshold and a second threshold for the feature response value are preset. The feature response value is compared with the first threshold and the second threshold, respectively. If the feature response value is less than the first threshold, the region is marked as a low significance region. If the feature response value is greater than the first threshold and less than the second threshold, the region is marked as a medium significance region. If the feature response value is greater than the second threshold, the region is marked as a high significance region.

[0024] Different labels are used to annotate the salient regions of the multi-level feature map, defining low salient regions, medium salient regions, and high salient regions.

[0025] Preferably, the adaptive tracking decision module generates tracking control commands through an optimization algorithm, including:

[0026] Based on the salient region annotation results, tracking points of different densities are deployed in the target spatial domain; differentiated prediction strategies are adopted for different salient regions, and the target motion is sequence-modeled by a recurrent neural network to predict the initial trajectory. The prediction time step is dynamically adjusted according to the salient region annotation results, wherein the prediction time step of the highly salient region is smaller than that of the medium salient region, and the prediction time step of the medium salient region is smaller than that of the low salient region.

[0027] The application of state estimation algorithm dynamically adjusts the prediction time step, introduces a multi-model interactive motion model, and continuously corrects the parameters in the motion model through iterative filtering algorithm, stopping when the maximum number of iterations is reached; and generates tracking control commands based on path planning algorithm.

[0028] Preferably, the execution control module includes:

[0029] A global coordinate system is defined with the installation point of the optoelectronic pod as the origin; the intrinsic and extrinsic parameters of the gimbal mechanism are obtained through the calibration process.

[0030] The tracking control commands are converted to the global coordinate system to obtain the tracking control commands in the global coordinate system; in the global coordinate system, the tracking control commands are aligned with the current pose of the gimbal mechanism to generate gimbal adjustment commands.

[0031] The tilt and azimuth angles of the gimbal mechanism are controlled by gimbal adjustment commands to adjust the sensor's pointing direction.

[0032] Preferably, the feedback optimization module monitors the target tracking status in real time, including:

[0033] Collect the execution parameters of the gimbal mechanism, synchronously acquire the actual position data of the target, calculate the deviation between the actual position data and the predicted position data, and generate a tracking deviation index;

[0034] The system presets a first threshold and a second threshold for tracking deviation. If the tracking deviation index is less than the first threshold, the tracking status is considered stable. If the tracking deviation index is greater than the first threshold but less than the second threshold, the tracking status needs fine-tuning. If the tracking deviation index is greater than the second threshold, the tracking status is considered abnormal.

[0035] Preferably, the feedback optimization module dynamically optimizes the multimodal target automatic identification and tracking strategy, including:

[0036] Based on the tracking deviation index, establish optimization objectives, including tracking accuracy and response speed objectives; and set constraints.

[0037] Using a heuristic optimization algorithm, multiple sets of tracking strategy parameter schemes are randomly generated. The completion status of each scheme is evaluated to obtain a score for the completion status of the optimization goal. The scores for the completion status of the optimization goal are then weighted and summed to obtain a comprehensive score. The scheme with the highest comprehensive score is selected from the multiple schemes as the final optimization scheme.

[0038] The selected final optimization scheme is applied to the multimodal target automatic identification and tracking strategy.

[0039] Preferably, the system further includes a data management module for:

[0040] When the preset cycle is met, historical tracking data of the target is obtained, including historical location data and historical sensor parameters;

[0041] Obtain real-time tracking data of the target, including real-time location data and real-time sensor parameters;

[0042] Construct a data deviation vector based on historical and real-time tracking data;

[0043] Retrieve the set of deviation modulus values ​​that satisfy the data deviation vector. If the variance of the deviation modulus set is greater than or equal to the variance threshold, calculate the data storage ratio.

[0044] Historical tracking data is cleaned up according to the data storage ratio.

[0045] Preferably, the present invention also includes a method for automatic identification and tracking of multimodal targets based on an optoelectronic pod, wherein the automatic identification and tracking of multimodal targets is performed based on the above-mentioned automatic identification and tracking system for multimodal targets based on an optoelectronic pod.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] This multimodal target automatic identification and tracking system based on an optoelectronic pod effectively solves many problems existing in traditional optoelectronic pod target identification and tracking systems through the collaborative design of multiple modules, demonstrating significant advantages. At the data acquisition level, the system innovatively integrates a visible light sensor, an infrared thermal imaging sensor, and a laser rangefinder into the optoelectronic pod. These three sensors each have their own advantages and complement each other: the visible light sensor can clearly capture detailed features such as the target's color and texture; the infrared thermal imaging sensor can accurately identify targets in low-visibility environments such as nighttime, heavy fog, and dense smoke by using the difference in thermal radiation between the target and the environment; and the laser rangefinder can acquire the target's distance information in real time, providing spatial location information for subsequent tracking decisions. Simultaneously, the system addresses the spatiotemporal asynchrony problem of multimodal data by setting up a dedicated spatiotemporal synchronization calibration mechanism. By aligning the timestamps of data collected by different sensors and calibrating the spatial coordinates, it ensures that the target data collected by the three sensors is highly consistent in both time and space dimensions. This achieves efficient fusion of multimodal data, not only enriching the dimensions of target perception data but also enabling the system to acquire comprehensive and accurate target information in various complex environments, freeing it from the dependence on environmental conditions of traditional single-sensor systems.

[0048] In the target feature extraction stage, the system employs a multi-scale convolutional neural network to process multimodal fusion data. This network can automatically adapt to feature changes of the target at different scales, extracting shallow and deep features of the target layer by layer from the multimodal data to form a complete multi-level feature map. Compared to traditional shallow feature extraction algorithms, the features extracted by the multi-scale convolutional neural network are more representative and discriminative, accurately capturing the unique features of the target in complex backgrounds. Even when the target is in motion or its shape changes, its core features can be effectively extracted. Simultaneously, the system annotates the generated multi-level feature map with salient regions, automatically selecting the feature regions that best represent the essence of the target and eliminating redundant information in the background. This allows the subsequent tracking and decision-making module to focus on key features, reducing the amount of invalid data processed, significantly improving the system's operating efficiency, and laying a solid foundation for accurate target recognition.

[0049] In terms of tracking decision-making and execution, the adaptive tracking decision-making module deploys a recurrent neural network to predict the target's motion trajectory based on the salient region annotation results of the feature map. The recurrent neural network possesses powerful time-series data processing capabilities, enabling it to dynamically learn the target's motion patterns by combining historical tracking data with real-time acquired target feature information. Even when the target exhibits complex motion states such as changes in direction or speed, it can accurately predict its subsequent trajectory, overcoming the limitations of traditional preset motion models on the target's motion state. Based on this, the module analyzes the predicted trajectory through optimization algorithms, generating precise tracking control commands to ensure that the commands closely match the target's actual motion trend. The execution control module and the tracking decision-making module work in close coordination, enabling rapid response to tracking control commands and driving the gimbal mechanism of the optoelectronic pod to precisely adjust the sensor pointing. Regardless of whether the target is moving slowly or rapidly, the sensor can be aligned with the target in a timely manner, effectively avoiding tracking deviations caused by delayed tracking commands and untimely gimbal adjustments in traditional systems, significantly improving the stability and accuracy of target tracking.

[0050] The feedback optimization module further endows the system with continuous self-optimization capabilities. This module monitors the target tracking status in real time, calculates tracking deviation indicators through a scientific evaluation system, and accurately identifies problems that occur during tracking, such as sensor pointing deviation and inaccurate trajectory prediction. Based on these deviation indicators, the module automatically generates corresponding tracking strategy adjustment parameters. These parameters can specifically optimize key aspects such as the synchronization accuracy of multimodal data acquisition, feature extraction algorithm parameters, trajectory prediction model coefficients, and the adjustment speed of the gimbal mechanism. By dynamically adjusting the operating parameters of each module, the system can continuously optimize the target recognition and tracking strategy according to real-time tracking status and environmental changes, avoiding the drawbacks of traditional systems that cannot adapt to complex scenarios due to fixed strategies. Even in the event of sudden environmental changes, such as a sudden downpour or a target suddenly changing direction at high speed, the system can quickly adjust its operating status through feedback optimization, maintaining efficient target recognition and stable tracking, significantly improving the system's reliability and adaptability in complex application scenarios. Attached Figure Description

[0051] Figure 1 This is a timing diagram of the multimodal target automatic identification and tracking system based on an optoelectronic pod described in this invention;

[0052] Figure 2 This is a schematic diagram illustrating the working principle of the multimodal data acquisition module.

[0053] Figure 3 This is a schematic diagram illustrating the working principle of the dynamic feature extraction module. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Please see Figure 1 This invention provides a method and system for automatic identification and tracking of multimodal targets based on an optoelectronic pod. The system includes: a multimodal data acquisition module, a dynamic feature extraction module, an adaptive tracking decision module, an execution control module, and a feedback optimization module.

[0056] The multimodal data acquisition module, based on an optoelectronic pod platform, deploys visible light sensors, infrared thermal imaging sensors, and laser rangefinders to acquire visible light image data, infrared thermal radiation data, and laser rangefinder point cloud data of the target. It also performs spatiotemporal synchronization calibration on this multimodal sensing data to ensure data consistency in time and space. The dynamic feature extraction module receives the multimodal sensing data, processes it using a multi-scale convolutional neural network, generates multi-level feature maps of the target, and labels these feature maps with salient regions to distinguish regions of different salience. The adaptive tracking decision module, based on the salient region labeling results, deploys a recurrent neural network to predict the target's trajectory and generates tracking control commands through optimization algorithms to guide tracking behavior. The execution control module, based on the tracking control commands, drives the gimbal mechanism of the optoelectronic pod to adjust the pitch and azimuth angles of the sensors, achieving precise control of sensor pointing. The feedback optimization module monitors the target tracking status in real time, calculates tracking deviation indicators, generates tracking strategy adjustment parameters, and dynamically optimizes the entire identification and tracking strategy to ensure continuous and efficient system operation.

[0057] Example 1: See Figure 2 Precise sensor integration and configuration are performed on an optoelectronic pod platform. This platform, as the foundational structure supporting the core sensing equipment, must meet the rigid requirements of multi-sensor collaborative operation in both its mechanical design and electrical interfaces. The visible light sensor, employing a high-resolution CMOS imaging device, is responsible for capturing the texture and color information of the target within the visible spectrum, outputting a sequential image data stream. The infrared thermal imaging sensor, based on the principle of microbolometers, senses the thermal radiation characteristics of the target object, generating infrared image data characterizing the temperature distribution. The laser rangefinder emits a laser beam and receives the echo, calculating the time of flight of the light wave to output point cloud data describing the target's distance. During physical installation, the parallelism of the optical axes and their relative positions of these three sensors must be fully considered, and mechanical calibration should be used to minimize observation errors caused by parallax.

[0058] The simulated multi-sensor collaborative acquisition mechanism is the core element for achieving efficient data acquisition. This mechanism dynamically manages the sampling frequency and resolution parameters of each sensor through a central control unit. Based on prior target information, such as estimated target size, speed, or category attributes, the control unit divides the target's observation area into high-sampling-rate and low-sampling-rate zones. The high-sampling-rate zone typically corresponds to the core parts of the target or key points in its trajectory. Within this zone, the system instructs the visible light and infrared sensors to acquire data at the highest frame rate and full resolution, while the laser rangefinder increases its scanning density. The low-sampling-rate zone targets the target's periphery or background areas. The system automatically reduces the sampling frequency and may use methods such as region cropping or pixel merging to reduce the amount of data. This dynamic configuration is transmitted to each sensor driver through a real-time generated control command set. This command set includes specific parameters such as exposure time, gain, frame rate, and scanning mode, ultimately enabling the system to acquire optimal multimodal sensing data with limited resources.

[0059] Spatiotemporal synchronization calibration is a prerequisite for ensuring effective fusion and subsequent processing of multimodal data. This process consists of two levels: time synchronization and spatial synchronization. Time synchronization relies on a high-precision hardware clock source to assign a unified timestamp to all sensor data. The timestamp accuracy needs to reach the microsecond level to eliminate motion blur caused by differences in sampling times. When the data stream is transmitted to the processing unit via a high-speed bus, the software algorithm further interpolates and aligns the timestamps to ensure that each frame of visible light image, infrared image, and laser point cloud data corresponds to the observation state at the same moment. Spatial synchronization is achieved through a pre-calibrated inter-sensor transformation matrix, which describes the rotation and translation relationships between the coordinate systems of each sensor. During processing, the laser point cloud data is projected onto the image coordinate system, and the infrared image is also aligned pixel-level with the visible light image through a registration algorithm. This unifies the data from all modalities into the same spatiotemporal reference system, forming standardized multimodal sensing data that can be used for subsequent feature extraction. The software implementation of the entire data acquisition module is based on a multi-threaded architecture. Each sensor has an independent acquisition thread, and the threads communicate and exchange data through shared memory and synchronization primitives. The parsing and issuance of control commands, sensor status monitoring, and temporary data caching and synchronization are all coordinated by a single main scheduling thread. This design ensures the real-time performance and reliability of data acquisition. The final output multimodal sensing data packet not only contains the aligned raw sensor data but also includes metadata such as acquisition parameters, timestamps, and coordinate system transformation information.

[0060] Example 2: See Figure 3The dynamic feature extraction module's processing flow begins with meticulous preprocessing of the multimodal sensing data. The raw visible light images, infrared thermal images, and laser point cloud data from the data acquisition module first undergo mean normalization. This step unifies the different numerical ranges of each modality into the same scale. Pixel values ​​in visible light images are typically linearly transformed to between 0 and 1, infrared thermal radiation data undergoes minimum-maximum normalization based on its temperature sensitivity, and laser ranging point cloud data is standardized according to its spatial coordinate range. The normalized multimodal data is then integrated according to modal dimensions to form a multidimensional tensor data structure with multiple channels. This tensor serves as the input to the subsequent deep neural network, maintaining the correspondence between the various modal data.

[0061] The constructed multi-scale convolutional neural network adopts an encoder-decoder architecture. The encoding path consists of multiple stacked convolutional and pooling layers, each using a convolutional kernel of a specific size for feature extraction. During encoding, shallower layers capture local details of the target, such as edges and texture information. As the network depth increases, deeper layers gradually extract more abstract and global semantic features. A non-linear activation function is added after each convolutional layer to enhance the network's expressive power, while pooling operations gradually reduce the spatial dimension of the feature maps and expand the receptive field. The decoding path gradually restores the spatial resolution of the feature maps through upsampling operations and uses transposed convolution or interpolation methods to fuse high-level semantic features with shallow detail features. A skip connection mechanism is also introduced in this process, directly passing feature maps of the corresponding scale in the encoding path to the decoding path, helping the network to better reconstruct the spatial detail information of the target.

[0062] In different output layers of a multi-scale convolutional neural network, one-dimensional convolution is applied to adjust and optimize feature channels. This operation effectively integrates feature information at different scales and adjusts the number of feature channels to maintain consistency. Subsequently, feature maps from multiple scales are fused using weighted summation or channel concatenation to generate multi-level feature maps containing rich contextual information. These feature maps contain both detailed information about the target and abstract features at the semantic level. The salient region annotation process first calculates the feature response value for each spatial location in the multi-level feature map. This value is obtained by weighted fusion of features from visible light, infrared, and point cloud data. The weight coefficients are dynamically adjusted based on the importance and reliability of different modalities. Visible light data has a higher weight under good lighting conditions, infrared data has an increased weight under low light or camouflage conditions, and laser point cloud data has a higher weight for distance measurement. The calculation of the feature response value comprehensively considers the intensity, contrast, and spatial distribution characteristics of each modality, forming a comprehensive saliency metric.

[0063] The preset first and second thresholds for feature response values ​​divide the feature region into three saliency levels. These thresholds were obtained through statistical analysis of extensive experimental data and are adaptable to different scenarios and target types. When the feature response value of a region is below the first threshold, the region is marked as a low-saliency region, typically corresponding to background or irrelevant areas. Regions with feature response values ​​between the first and second thresholds are marked as medium-saliency regions, potentially containing relevant contextual information about the target. Regions with feature response values ​​exceeding the second threshold are marked as high-saliency regions, often representing the core parts or key feature areas of the target. Different saliency regions are labeled with differentiated tags: low-saliency regions are typically represented by a value of 0 or a specific color code, medium-saliency regions by a value of 1 or another set of color codes, and high-saliency regions by a value of 2 or a striking color. This labeling method not only preserves saliency level information but also provides an intuitive visual reference for subsequent tracking decisions. The multi-level feature map with completed saliency labeling contains rich target information, providing accurate input data for the adaptive tracking decision module.

[0064] Taking the tracking of a moving target at sea as an example, an electro-optical pod deployed at a coastal observation station needs to continuously identify and track a specific vessel. The multimodal data acquisition module simultaneously acquires visible light images, infrared thermal images, and laser ranging data of the target. The visible light sensor captures the vessel's outline and deck structure under sunlight conditions, the infrared thermal imaging sensor detects the thermal radiation characteristics of the vessel's engine, and the laser ranging sensor accurately measures the real-time distance between the vessel and the observation station. After receiving this multimodal data, the dynamic feature extraction module first performs mean normalization. The pixel values ​​of the visible light image are converted to the 0-1 range, the infrared thermal imaging data is normalized according to its temperature range, and the laser ranging data is scaled according to its maximum range. The normalized three types of data are integrated into a three-dimensional tensor along the channel dimension, where the visible light data occupies the RGB channel, the infrared data is a separate channel, and the laser ranging data is converted into a depth map as a fourth channel.

[0065] This multidimensional tensor is fed into a multi-scale convolutional neural network for processing. The network's encoding path contains multiple convolutional layers; shallow layers identify the ship's edge and contour features, while deeper layers extract more abstract features such as ship type and structural information. In the decoding path, higher-level features are gradually restored to spatial details through upsampling operations and then fused with shallower features. The network ultimately outputs a multi-level feature map containing features at different scales, which preserves both the ship's detailed features and semantic-level information. When labeling salient regions on the generated multi-level feature map, the feature response value for each spatial location is calculated. The feature response value for the ship's contour region is high in the visible light channel, significant in the engine heat source region in the infrared channel, and prominent in the ship's physical region in the laser ranging channel. By weighted fusion of these modal feature response values, a comprehensive feature response map is obtained.

[0066] Preset feature response values, first and second thresholds, categorize the vessel area into different saliency levels. The vessel's engine area, possessing both significant thermal radiation and geometric features, has a feature response value exceeding the second threshold and is marked as a high-saliency area. The vessel's deck area, exhibiting clear features in the visible light image but weak thermal radiation characteristics, has a feature response value between the two thresholds and is marked as a medium-saliency area. The sea surface area surrounding the vessel has a low feature response value, below the first threshold, and is marked as a low-saliency area. These saliency areas are labeled using different tags: high-saliency areas are marked in red, medium-saliency areas in yellow, and low-saliency areas in blue. The labeled feature map clearly shows the differences in importance between different areas of the vessel, providing a basis for subsequent tracking decisions.

[0067] Example 3: The adaptive tracking decision module deploys differentiated tracking point distribution patterns within the target spatial domain based on the salient region annotation results provided by the dynamic feature extraction module. Dense tracking point clouds are deployed in high-salient regions to improve tracking accuracy, medium-salient regions use a medium-density distribution, and low-salient regions use sparse distribution to optimize computational resources. Corresponding prediction strategies are adopted for regions with different salient levels. A recurrent neural network constructed using gated recurrent units models the target motion sequence. The network input is the spatiotemporal coordinate sequence of historical trajectory points, and the output is the position prediction for the future time step. The initial trajectory prediction is generated based on the current observation sequence. The prediction time step is dynamically adjusted according to the regional salientity. Shorter prediction step steps are used in high-salient regions to maintain tracking sensitivity, medium-salient regions use medium step steps, and low-salient regions are configured with longer prediction step steps to achieve forward-looking prediction.

[0068] A state estimation algorithm based on maximum a posteriori probability estimation is applied to adjust the prediction time step online. This algorithm adaptively corrects the step step parameter by analyzing the statistical characteristics of the prediction error. A multi-model interactive motion modeling framework is introduced, including various motion modes such as uniform velocity model, uniform acceleration model, and cooperative turning model. The models are weighted and fused through interactive probabilities. The iterative filtering algorithm uses an expectation-maximization framework for parameter optimization. In each iteration, the state estimate and model parameters are recalculated. The optimization process terminates when the number of iterations reaches a preset maximum value or the parameter change is lower than the convergence threshold.

[0069] Based on nonlinear model predictive control theory, a tracking control command is generated, and the following optimization problem is constructed:

[0070]

[0071] in: Indicates the length of the prediction time domain. It is the predicted position vector at time k. It is the reference position vector at time k. It is the state error weight matrix. It controls the length of the time domain. It is the control increment vector at time k. This is the control increment weight matrix. By solving this optimization problem, the optimal control sequence is obtained, and the first control command is extracted and output to the actuator.

[0072] The execution control module establishes a right-handed Cartesian coordinate system with the physical installation point of the optoelectronic pod as the origin. Through a precise calibration process, it obtains the intrinsic parameter matrix and extrinsic parameter transformation relationship of the gimbal mechanism. The intrinsic parameter matrix includes optical parameters such as focal length, principal point coordinates, and distortion coefficients, while the extrinsic parameter transformation describes the rotation and translation relationship between the gimbal coordinate system and the global coordinate system. The tracking control command is transformed from the predictive coordinate system to the global coordinate system using quaternion interpolation for a smooth transition, ensuring continuity and stability during the coordinate system transformation process. In the global coordinate system, the transformed tracking control command is aligned with the current gimbal pose, and the deviation between the current azimuth and pitch angles and the required pose is calculated. When generating gimbal adjustment commands, an incremental control strategy is used, converting the pose deviation into motor drive signals through a proportional-integral-derivative controller. High-precision servo drivers control the pitch and azimuth motors of the gimbal mechanism, employing a closed-loop control method to correct execution errors in real time, achieving precise and rapid adjustment of the sensor pointing. Throughout the entire process, the gimbal load status and motion parameters are continuously monitored. When abnormal vibration or overload is detected, the protection mechanism is automatically triggered to ensure the reliability and stability of the system operation.

[0073] Example 4: Synchronous acquisition and processing of multiple data sources. The execution parameters of the gimbal mechanism include azimuth, pitch, angular velocity, and acceleration data, which are acquired in real time through encoders and inertial measurement units mounted on the gimbal motor. The actual position data of the target is obtained through a multimodal sensor fusion algorithm. This algorithm integrates the target pixel coordinates in the visible light image, the temperature distribution characteristics of the infrared thermal imaging, and the spatial point cloud information of the laser ranging to calculate the target's three-dimensional position coordinates in the global coordinate system. The deviation between the actual position data and the predicted position data is calculated using the Euclidean distance formula. The difference between the two coordinate points is calculated in three-dimensional space to generate a quantified tracking deviation index value, which reflects the gap between the current tracking accuracy and the expected performance.

[0074] The preset tracking deviation threshold system includes two key values: the first threshold is set as the upper limit of normal fluctuations within the allowable range, and the second threshold is defined as the critical warning line for system performance. When the calculated tracking deviation index is less than the first threshold, the system determines that the current tracking state is within a stable range and maintains the existing parameters. If the deviation index is between the first and second thresholds, the system generates a judgment that the state needs fine-tuning and initiates the parameter adjustment process. When the deviation index exceeds the second threshold, the system determines that the tracking state is abnormal and triggers an emergency handling mechanism. These threshold values ​​are dynamically configured according to specific application scenarios and target characteristics; for example, a higher threshold is used for high-speed moving targets, while a lower threshold is set for precision tracking scenarios. The process of dynamically optimizing the multimodal target automatic recognition and tracking strategy is based on quantitative evaluation. The optimization target system includes two core dimensions: tracking accuracy and response speed. The tracking accuracy target aims to minimize position deviation, while the response speed target aims to maximize the system adjustment rate. Constraints include factors such as the upper limit of computing resource consumption, the limit of gimbal mechanical movement, and energy consumption constraints. These conditions together constitute the boundary constraints of the optimization problem. The heuristic optimization algorithm is implemented using an improved genetic algorithm, which uses chromosome encoding to represent different combinations of tracking strategy parameters. Each chromosome contains key parameters such as filter coefficients, prediction step size, and control gain.

[0075] Multiple randomly generated tracking strategy parameter schemes form an initial population, with each scheme representing a complete system parameter configuration. When evaluating each scheme, the system runs with that parameter configuration in a simulation environment, collecting measured data on tracking accuracy and response speed. Accuracy is evaluated by calculating the root mean square value of the position deviation, while speed is evaluated by measuring the time delay from the occurrence of the deviation to the completion of the adjustment. The achievement of both optimization objectives is scored on a percentage basis, with the accuracy score inversely proportional to the deviation value and the speed score inversely proportional to the delay time. Finally, different weighting coefficients are assigned to the two scores according to application requirements, and a weighted sum is obtained to obtain a comprehensive score. The parameter configuration with the highest comprehensive score is selected as the final optimization scheme from all evaluated schemes. This selection process considers not only the total score but also analyzes the performance stability of each scheme in a specific scenario. The selected final optimization scheme is applied to the actual operating system through the parameter configuration interface, updating key settings such as the filtering parameters of the control algorithm, the confidence coefficient of the motion model, and the time-domain parameters of trajectory prediction. The entire optimization process adopts a closed-loop operation mode, periodically re-evaluating system performance and triggering a new round of optimization to ensure that the system can adapt to environmental changes and changes in target characteristics. Table 1 shows the scheme evaluation data within a certain optimization cycle, reflecting the performance under different parameter configurations.

[0076] Table 1: Evaluation Results of Tracking Strategy Parameter Schemes

[0077]

[0078] The weighting is 0.6 for accuracy and 0.4 for speed. The overall score is calculated as: Accuracy score × 0.6 + Speed ​​score × 0.4. Based on the evaluation results, Scheme 002 received the highest overall score, and its parameter configuration will be adopted as the new operating parameters. After the parameter update, the system continues to monitor actual performance, recording new deviation data to provide input for the next round of optimization, forming a closed-loop optimization mechanism for continuous improvement.

[0079] Example 5: A preset time-cycle triggering mechanism. This cycle can be dynamically adjusted according to the system's operating status, typically set to a fixed time interval or an event trigger based on accumulated data. When the cycle condition is met, the module first retrieves historical tracking data of the target from persistent storage. This data includes historical position coordinate sequences, historical sensor parameter configuration records, and corresponding timestamp information. Historical position data is stored in three-dimensional coordinates, including azimuth, pitch, and distance values. Historical sensor parameters cover device configuration information such as focal length, exposure time, and sampling rate. Simultaneously, the module obtains real-time tracking data through a data interface, including the target's real-time position coordinates and real-time sensor operating parameters. This data comes from the ongoing tracking task and has the latest timestamp.

[0080] Based on historical and real-time tracking data, the module constructs a data deviation vector to describe data change characteristics. Each dimension of the deviation vector corresponds to a data indicator; for example, the location deviation dimension calculates the Euclidean distance between historical and real-time locations, while the parameter deviation dimension calculates the difference in sensor parameter configurations. For location data, deviation calculation considers the difference in spatial coordinates; for sensor parameters, deviation calculation uses relative change rates or absolute differences, depending on the parameter type. All deviation components are normalized and combined into a multi-dimensional deviation vector, which comprehensively reflects the degree of difference between historical and real-time data. When retrieving the set of deviation moduli that satisfy the data deviation vector, the module first calculates the deviation moduli corresponding to each historical data point and the real-time data. The deviation moduli are obtained by calculating the norm of the deviation vector, typically using the L2 norm or a custom norm weighted according to the application scenario. These moduli form a time series set, reflecting the pattern of data deviation changes over time. When the variance calculation result of the deviation moduli set shows a value greater than or equal to a preset variance threshold, it indicates that the data fluctuation is large, with many abnormal or outdated data, requiring cleanup. The variance threshold is set based on storage capacity requirements and data importance, and is usually determined by statistical analysis of historical data characteristics.

[0081] An adaptive algorithm based on variance is used to calculate the data storage ratio, which defines the percentage of historical data to be retained out of the total data. When the variance is large, the storage ratio decreases accordingly, allowing for the cleaning of more data; when the variance is small, the storage ratio increases, retaining more historical data. The ratio calculation considers factors such as current storage space utilization, data importance, and system performance requirements, and determines the final ratio value through a weighted function. Based on the calculated data storage ratio, the module cleans up the historical tracking data. The cleanup process employs a priority strategy, first deleting older, less frequently used data records, while retaining recently used and key node data. For location data, trajectory integrity is considered during cleanup to avoid disrupting continuous trajectory segments; for sensor parameter data, key node records of parameter changes are retained. After cleanup, the module updates the data index and metadata information to ensure the accessibility and consistency of the remaining data. The entire data management process is executed automatically on a regular basis, forming a continuous data maintenance mechanism that ensures data validity and optimizes the efficiency of storage resource utilization.

[0082] The data management process is illustrated with a concrete example: Assume the system is configured to perform a data management cycle every 24 hours. At the start of a certain cycle, the module retrieves historical tracking data from the past 30 days, containing 100,000 location records and corresponding sensor parameters. Real-time data shows the current target location as (120.5, 35.8, 1000), with the sensor configured in telephoto mode and a high sampling rate. The deviation vector between each historical data point and the real-time data is calculated, including location deviation and parameter configuration deviation. The calculated deviation magnitude set shows a variance of 0.85, higher than the set variance threshold of 0.7. Based on a preset algorithm, the data storage ratio is calculated to be 65%. The module then cleans up 35% of the historical data, prioritizing the deletion of earlier records with larger deviations, while retaining the main data and key node records from the most recent 7 days. After cleaning, the amount of data stored in the system is optimized while maintaining the integrity of important historical information.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0084] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An opto-electronic pod-based multi-modal target automatic identification and tracking system, characterized in that, The method comprises the following steps: A multi-modal data acquisition module is used to acquire multi-modal perception data of a target based on an integrated visible light sensor, an infrared thermal imaging sensor and a laser ranging sensor in an optoelectronic pod, and to calibrate the multi-modal perception data in time and space; A dynamic feature extraction module is used to apply a multi-scale convolutional neural network to the multi-modal perception data to generate multi-level feature maps of the target, and to label significant regions in the multi-level feature maps; An adaptive tracking decision module is used to deploy a recurrent neural network according to the significant region labeling results, to predict a target motion trajectory, and to generate tracking control instructions through an optimization algorithm; An execution control module is used to drive a gimbal mechanism of the optoelectronic pod based on the tracking control instructions, to adjust the pointing direction of the sensors to achieve target tracking; A feedback optimization module is used to monitor the target tracking state in real time, to calculate tracking deviation indicators, to generate tracking strategy adjustment parameters, and to dynamically optimize the multi-modal target automatic identification and tracking strategy. The dynamic feature extraction module labels significant regions in the multi-level feature maps, which comprises the following steps: The feature response value of each region in the multi-level feature maps is calculated by weighting and fusing the visible light image data, the infrared thermal radiation data and the laser ranging point cloud data; A first threshold value of the feature response value and a second threshold value of the feature response value are preset, and the feature response value is compared with the first threshold value of the feature response value and the second threshold value of the feature response value, respectively; if the feature response value is less than the first threshold value of the feature response value, the region is marked as a low significant region; if the feature response value is greater than the first threshold value of the feature response value and less than the second threshold value of the feature response value, the region is marked as a medium significant region; if the feature response value is greater than the second threshold value of the feature response value, the region is marked as a high significant region; Different labels are used to label significant regions in the multi-level feature maps, and low significant regions, medium significant regions and high significant regions are defined.

2. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 1, characterized in that, The multi-modal data acquisition module comprises: Three sensing devices, i.e., a visible light sensor, an infrared thermal imaging sensor and a laser ranging sensor, are deployed on the platform of the optoelectronic pod to collect visible light image data, infrared thermal radiation data and laser ranging point cloud data of the target, respectively; A multi-sensor collaborative collection mechanism is simulated to dynamically adjust the sampling frequency and resolution; based on the prior information of the target, the core region of the target is recorded as a high sampling rate region, and the peripheral region of the target is recorded as a low sampling rate region; sensor parameters are dynamically configured through control instructions to obtain the final multi-modal perception data.

3. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 2, characterized in that, The dynamic feature extraction module applies a multi-scale convolutional neural network to the multi-modal perception data to generate multi-level feature maps of the target, which comprises the following steps: The multi-modal perception data is subjected to mean normalization processing, and the normalized multi-modal perception data is integrated into a multi-dimensional tensor according to the modal dimension; A multi-scale convolutional neural network structure is constructed, which includes an encoding path and a decoding path; in the encoding path, a deep convolutional network is used to extract hierarchical features from the input multi-modal perception data to generate feature maps of different scales; in the decoding path, the high-level feature maps are restored to spatial details through an upsampling operation; One-dimensional convolution is applied to each output layer of the multi-scale convolutional neural network structure to adjust the feature channel, and different scale feature maps output by the multi-scale convolutional neural network structure are fused to generate a multi-level feature map of the target.

4. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 1, characterized in that, The adaptive tracking decision module generates a tracking control instruction through an optimization algorithm, and the tracking control instruction comprises: According to the salient region labeling result, different densities of tracking points are deployed inside the target space domain; different prediction strategies are adopted for different salient regions, a sequence model of target motion is established through a recurrent neural network for initial trajectory prediction, and the prediction time step is dynamically adjusted according to the salient region labeling result, wherein the prediction time step of a high salient region is smaller than that of a medium salient region, and the prediction time step of the medium salient region is smaller than that of a low salient region; A state estimation algorithm is applied to dynamically adjust the prediction time step, a motion model of multi-model interaction is introduced, parameters in the motion model are continuously corrected through an iterative filtering algorithm, and the process is stopped when the maximum number of iterations is reached; and a tracking control instruction is generated based on a path planning algorithm.

5. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 4, characterized in that, The execution control module comprises: A global coordinate system is defined with the photoelectric pod mounting point as the origin; the intrinsic and extrinsic parameters of the holder mechanism are obtained through a calibration process; The tracking control instruction is converted into the global coordinate system to obtain the tracking control instruction in the global coordinate system; the tracking control instruction is aligned with the current pose of the holder mechanism in the global coordinate system to generate a holder adjustment instruction; The holder adjustment instruction is used to control the pitch angle and azimuth angle of the holder mechanism to realize sensor pointing adjustment.

6. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 5, characterized in that, The feedback optimization module comprises: Execution parameters of the holder mechanism are collected, actual position data of the target are synchronously obtained, a deviation between the actual position data and predicted position data is calculated, and a tracking deviation index is generated; A first threshold of the tracking deviation index and a second threshold of the tracking deviation index are preset; if the tracking deviation index is less than the first threshold of the tracking deviation index, it is determined that the tracking state is stable; if the tracking deviation index is greater than the first threshold of the tracking deviation index and less than the second threshold of the tracking deviation index, it is determined that the tracking state needs to be fine-tuned; and if the tracking deviation index is greater than the second threshold of the tracking deviation index, it is determined that the tracking state is abnormal.

7. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 6, characterized in that, The feedback optimization module dynamically optimizes the multi-modal target automatic identification tracking strategy, and the method comprises: Based on the tracking deviation index, an optimization target is established, the optimization target comprises a tracking accuracy target and a response speed target; and a restriction condition is set; A heuristic optimization algorithm is used to randomly generate a plurality of tracking strategy parameter schemes, the completion of the optimization target is evaluated for each scheme, a completion score of the optimization target is obtained, a weighted sum of the completion scores of the optimization target is calculated, a comprehensive score is obtained, and a scheme with the highest comprehensive score is selected from the plurality of schemes as a final optimization scheme; The selected final optimization scheme is applied to the multi-modal target automatic identification tracking strategy.

8. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 7, characterized in that, The data management module is further configured to: When a preset period is met, historical tracking data of the target is obtained, including historical position data and historical sensor parameters; Real-time tracking data of the target is obtained, including real-time position data and real-time sensor parameters; According to the historical tracking data and the real-time tracking data, a data bias vector is constructed; A bias modulus value set satisfying the data bias vector is searched, and when a variance of the bias modulus value set is greater than or equal to a variance threshold, a data storage ratio is calculated; According to the data storage ratio, the historical tracking data is cleaned up.

9. A method for optoelectronic pod-based multi-modal target automatic recognition tracking, characterized in that, The multi-modal target automatic identification tracking system based on the optoelectronic pod according to any one of claims 1 to 8 is used for multi-modal target automatic identification tracking.

Images