Multi-modal target automatic identification and tracking method and system based on photoelectric pod

By integrating multiple sensors and multi-scale convolutional neural networks, combined with recurrent neural networks and feedback optimization modules, the accuracy and stability issues of the optoelectronic pod target recognition and tracking system in complex environments were solved, achieving efficient and adaptive target recognition and tracking.

CN120913024AActive Publication Date: 2025-11-07CHENGDU HAOFU TECH CO LTD

Patent Information

Application Number
CN202511438300.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-10-10
Publication Date
2025-11-07
Estimated Expiration
2045-10-10

AI Technical Summary

Technical Problem

Existing optoelectronic pod target recognition and tracking systems suffer from low accuracy and poor stability in complex environments. They also suffer from spatiotemporal asynchrony of multimodal data, insufficient feature extraction, lagging tracking decisions, and a lack of feedback optimization mechanisms, making them unsuitable for complex scenarios.

Method used

It integrates a visible light sensor, an infrared thermal imaging sensor, and a laser rangefinder sensor. It uses a multi-scale convolutional neural network to extract multi-level features. The adaptive tracking decision module combines a recurrent neural network to predict target motion, and the feedback optimization module adjusts the strategy in real time.

Benefits of technology

It achieves high-precision and stable target recognition and tracking in complex environments, efficient fusion of multimodal data, and the system has self-optimization capabilities to adapt to changes in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120913024A_ABST
    Figure CN120913024A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of photoelectric pod target tracking, and discloses a multi-mode target automatic identification tracking method and system based on a photoelectric pod. A multi-modal data acquisition module of the system integrates visible light, infrared thermal imaging and laser ranging sensors based on a photoelectric pod, acquires target multi-modal sensing data and completes time-space synchronous calibration; the dynamic feature extraction module adopts a multi-scale convolutional neural network to process multi-modal data, generates a target multi-level feature map and performs significance region labeling; the adaptive tracking decision module deploys a recurrent neural network to predict a target motion trajectory according to a labeling result, and generates a tracking control instruction through an optimization algorithm; the execution control module drives the photoelectric pod holder mechanism and adjusts the orientation of the sensor to realize target tracking; the feedback optimization module monitors the tracking state in real time, calculates a tracking deviation index, and generates a strategy adjustment parameter to dynamically optimize the tracking strategy.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optoelectronic pod target tracking, in particular to a multi-modal target automatic identification and tracking method and system based on an optoelectronic pod. BACKGROUND

[0002] In many fields such as aerospace, security monitoring, forest fire prevention, maritime search and rescue, as an important load equipment, the target identification and tracking function carried by the optoelectronic pod has always been the core direction of technical research and development. With the continuous expansion of application scenarios, higher and higher requirements are put forward for the precision, stability and environmental adaptability of the optoelectronic pod target identification and tracking. At present, most of the target identification and tracking systems carried by the optoelectronic pod on the market rely on a single type of sensor for data collection, such as only using visible light sensors or infrared thermal imaging sensors. The system using a single sensor often faces obvious limitations in complex environments: the visible light sensor is significantly affected by light conditions, and in night, heavy fog, heavy rain and other bad weather, it is difficult to clearly capture the target features, resulting in a significant reduction in target identification accuracy; the infrared thermal imaging sensor can identify targets in low light environments, but it has weak ability to present detailed features of the target, and when the thermal radiation features of multiple targets are similar, target confusion problems are easy to occur; and using a laser ranging sensor alone can only obtain distance information of the target, and cannot fully present the shape, motion trend and other key contents of the target, making it difficult to support accurate target identification and tracking decisions.

[0003] Even if some systems try to integrate multiple sensors to form a multi-modal data collection mode, there is still a problem of time and space synchronization of multi-modal data. Due to the differences in response speed and data collection frequency of different sensors, the collected visible light data, infrared thermal imaging data and laser ranging data often have delays in the time dimension and deviations in the space dimension, resulting in ineffective fusion of multi-modal data, which not only makes it difficult to take advantage of the synergy of multiple sensors, but also may affect the accuracy of target identification due to data conflicts.

[0004] In the target feature extraction link, traditional systems mostly use simple feature extraction algorithms such as edge detection, gray histogram, etc. Such algorithms can only extract shallow features of the target, and cannot capture detailed information and deep features of the target at different scales. When the target is in a complex background or changes in shape due to motion, the shallow features are easy to be confused with the background features, resulting in insufficient feature extraction accuracy, which further affects the effect of subsequent target identification and tracking. At the same time, the existing system lacks an effective feature map screening mechanism, which cannot highlight the saliency region of the target, so that the subsequent tracking decision module needs to process a large amount of redundant information, reducing the running efficiency of the system.

[0005] In the aspects of tracking decision and execution, the tracking decision algorithm of the traditional system is mostly based on preset motion models, such as uniform linear motion, uniform acceleration motion and the like. Such models are difficult to adapt to the complex motion state of the target. When the target appears sudden motion such as direction change and speed change, the preset model cannot accurately predict the motion trajectory of the target, resulting in tracking instruction lag, and further tracking deviation. In addition, the coordination between the execution control module and the tracking decision module is insufficient, and the adjustment speed and accuracy of the gimbal mechanism cannot match the requirements of the tracking instruction, especially when the target moves quickly, the sensor cannot be adjusted in time, which further reduces the stability of tracking.

[0006] The existing system generally lacks effective feedback optimization mechanism, and can only identify and track the target according to the fixed strategy, and cannot dynamically adjust the system parameters according to the real-time tracking state. When the environmental conditions change or the tracking deviation increases due to the change of the target motion state, the system cannot timely find the problem and optimize it, and can only continue to run according to the original strategy, eventually leading to tracking failure, which is difficult to meet the high reliability demand of target identification and tracking in complex scenes. SUMMARY

[0007] The purpose of the present application is to provide a multi-modal target automatic identification and tracking method and system based on an optoelectronic pod, to solve the problems raised in the above background art.

[0008] To achieve the above purpose, the present application provides a multi-modal target automatic identification and tracking system based on an optoelectronic pod, which comprises:

[0009] A multi-modal data acquisition module based on an optoelectronic pod integrates a visible light sensor, an infrared thermal imaging sensor and a laser ranging sensor to obtain multi-modal perception data of the target, and calibrates the multi-modal perception data in time and space synchronously;

[0010] A dynamic feature extraction module applies a multi-scale convolutional neural network to the multi-modal perception data to generate multi-level feature maps of the target, and labels the salient regions of the multi-level feature maps;

[0011] An adaptive tracking decision module deploys a recurrent neural network according to the salient region labeling results, predicts the motion trajectory of the target, and generates tracking control instructions through an optimization algorithm;

[0012] An execution control module drives the gimbal mechanism of the optoelectronic pod based on the tracking control instructions, adjusts the pointing of the sensor to realize target tracking;

[0013] A feedback optimization module monitors the target tracking state in real time, calculates the tracking deviation index, generates tracking strategy adjustment parameters, and dynamically optimizes the multi-modal target automatic identification and tracking strategy.

[0014] Preferably, the multi-modal data acquisition module comprises:

[0015] The visible light sensor, the infrared thermal imaging sensor and the laser ranging sensor are deployed on the optical pod platform to collect visible light image data, infrared thermal radiation data and laser ranging point cloud data of the target respectively;

[0016] A multi-sensor cooperative collection mechanism is simulated to dynamically adjust the sampling frequency and resolution; based on the target prior information, the target core area is recorded as a high sampling rate area and the target peripheral area is recorded as a low sampling rate area; sensor parameters are dynamically configured through control instructions to obtain the final multi-modal perception data.

[0017] Preferably, the dynamic feature extraction module applies a multi-scale convolutional neural network to the multi-modal perception data to generate a multi-level feature map of the target, including:

[0018] The multi-modal perception data is subjected to mean normalization processing, and the normalized multi-modal perception data is integrated into a multi-dimensional tensor according to the modal dimension;

[0019] A multi-scale convolutional neural network structure is constructed, including an encoding path and a decoding path; in the encoding path, a deep convolutional network is used to perform hierarchical feature extraction on the input multi-modal perception data to generate feature maps of different scales; in the decoding path, high-level feature maps are restored to spatial details through upsampling operations;

[0020] One-dimensional convolution is applied to each output layer of the multi-scale convolutional neural network structure to adjust the feature channels, fuse the feature maps of different scales output by the multi-scale convolutional neural network structure, and generate a multi-level feature map of the target.

[0021] Preferably, the dynamic feature extraction module performs salient region annotation on the multi-level feature map, including:

[0022] The feature response value of each region in the multi-level feature map is calculated, and the feature response value is obtained by weighted fusion of the visible light image data, the infrared thermal radiation data and the laser ranging point cloud data;

[0023] A first threshold value of the feature response value and a second threshold value of the feature response value are preset, and the feature response value is compared with the first threshold value of the feature response value and the second threshold value of the feature response value respectively; if the feature response value is less than the first threshold value of the feature response value, the region is marked as a low saliency area; if the feature response value is greater than the first threshold value of the feature response value and less than the second threshold value of the feature response value, the region is marked as a medium saliency area; if the feature response value is greater than the second threshold value of the feature response value, the region is marked as a high saliency area;

[0024] Different labels are used to perform salient region annotation on the multi-level feature map, and low saliency areas, medium saliency areas and high saliency areas are defined.

[0025] Preferably, the adaptive tracking decision module generates tracking control instructions through an optimization algorithm, which includes:

[0026] According to the salient region annotation result, different densities of tracking points are deployed inside the target space domain; different prediction strategies are adopted for different salient regions, sequence modeling of target motion is performed through a recurrent neural network, initial trajectory prediction is performed, and the prediction time step is dynamically adjusted according to the salient region annotation result, wherein the prediction time step of a high salient region is smaller than that of a medium salient region, and the prediction time step of a medium salient region is smaller than that of a low salient region.

[0027] A state estimation algorithm is applied to dynamically adjust the prediction time step, a multi-model interactive motion model is introduced, and the parameters in the motion model are continuously corrected through an iterative filtering algorithm, and the process is stopped when the maximum number of iterations is reached; and a path planning algorithm is used to generate tracking control instructions.

[0028] Preferably, the execution control module includes:

[0029] A global coordinate system is defined with the photoelectric pod mounting point as the origin, and the intrinsic and extrinsic parameters of the gimbal mechanism are obtained through a calibration process.

[0030] The tracking control instructions are converted into the global coordinate system to obtain tracking control instructions in the global coordinate system; in the global coordinate system, the tracking control instructions are aligned with the current pose of the gimbal mechanism to generate gimbal adjustment instructions.

[0031] The pitch angle and azimuth angle of the gimbal mechanism are controlled through the gimbal adjustment instructions to realize sensor pointing adjustment.

[0032] Preferably, the feedback optimization module monitors the target tracking state in real time, which includes:

[0033] The execution parameters of the gimbal mechanism are collected, the actual position data of the target is synchronously obtained, the deviation between the actual position data and the predicted position data is calculated, and a tracking deviation index is generated.

[0034] A first threshold value and a second threshold value of the tracking deviation index are preset; if the tracking deviation index is less than the first threshold value of the tracking deviation index, it is determined that the tracking state is stable; if the tracking deviation index is greater than the first threshold value and less than the second threshold value, it is determined that the tracking state needs to be fine-tuned; and if the tracking deviation index is greater than the second threshold value, it is determined that the tracking state is abnormal.

[0035] Preferably, the feedback optimization module dynamically optimizes the multi-modal target automatic identification tracking strategy, which includes:

[0036] Based on the tracking deviation index, an optimization target is established, and the optimization target includes a tracking accuracy target and a response speed target; a restriction condition is set;

[0037] A heuristic optimization algorithm is used to randomly generate multiple sets of tracking strategy parameter schemes, evaluate the completion of the optimization target for each scheme, obtain a completion score of the optimization target, and perform a weighted summation on the completion score of the optimization target to obtain a comprehensive score, and select a scheme with the highest comprehensive score from the multiple schemes as a final optimization scheme;

[0038] The selected final optimization scheme is applied to the multi-modal target automatic identification tracking strategy.

[0039] Preferably, the system further comprises a data management module for:

[0040] When a preset period is met, historical tracking data of the target is obtained, including historical position data and historical sensor parameters;

[0041] Real-time tracking data of the target is obtained, including real-time position data and real-time sensor parameters;

[0042] A data deviation vector is constructed according to the historical tracking data and the real-time tracking data;

[0043] A set of deviation modulus values satisfying the data deviation vector is searched, and when the variance of the set of deviation modulus values is greater than or equal to a variance threshold, a data storage ratio is calculated;

[0044] The historical tracking data is cleaned according to the data storage ratio.

[0045] Preferably, the application further comprises a multi-modal target automatic identification tracking method based on an optoelectronic pod, which performs multi-modal target automatic identification tracking based on the above-mentioned multi-modal target automatic identification tracking system based on the optoelectronic pod.

[0046] Compared with the prior art, the application has the following beneficial effects:

[0047] The multi-modal target automatic identification and tracking system based on the photoelectric pod solves many problems existing in the traditional photoelectric pod target identification and tracking system through the collaborative design of multiple modules, and shows significant advantages. In the data acquisition layer, the system innovatively integrates visible light sensors, infrared thermal imaging sensors and laser ranging sensors in the photoelectric pod. Each of the three sensors has its own advantages and complements each other: the visible light sensor can clearly capture the color, texture and other detailed features of the target; the infrared thermal imaging sensor can accurately identify the target by the difference in thermal radiation between the target and the environment in low-visibility environments such as night, heavy fog and thick smoke; the laser ranging sensor can obtain the distance information of the target in real time, providing spatial position basis for subsequent tracking decision. At the same time, the system sets up a special time and space synchronization calibration mechanism to solve the time and space asynchronization problem of multi-modal data. By aligning the time stamps of the data collected by different sensors and calibrating the spatial coordinates, the system ensures that the target data collected by the three sensors is highly consistent in time and space dimensions, realizes the efficient fusion of multi-modal data, and not only enriches the dimensions of target perception data, but also enables the system to obtain comprehensive and accurate target information in various complex environments, and gets rid of the dependence of traditional single sensor system on environmental conditions.

[0048] In the target feature extraction link, the system uses a multi-scale convolutional neural network to process multi-modal fusion data. This network can automatically adapt to the feature changes of the target at different scales, extract the shallow and deep features of the target from multi-modal data layer by layer, and form a complete multi-level feature map. Compared with traditional shallow feature extraction algorithms, the features extracted by the multi-scale convolutional neural network are more representative and distinguishable, and can accurately capture the unique features of the target in a complex background. Even if the target is in motion or its shape changes, it can also effectively extract its core features. At the same time, the system performs salient region labeling on the generated multi-level feature map, automatically filters out the feature regions that best represent the essence of the target through algorithms, and eliminates redundant information in the background, so that the subsequent tracking decision module can focus on key features and reduce the processing amount of invalid data, significantly improving the system's running efficiency and laying a solid foundation for accurate target identification.

[0049] In terms of tracking decision and execution, the adaptive tracking decision module deploys a recurrent neural network for target trajectory prediction based on the salient region annotation results of the feature map. The recurrent neural network has strong time series data processing capability, can combine historical tracking data and real-time collected target feature information, dynamically learn the motion law of the target, and accurately predict the subsequent motion trajectory even if the target appears complex motion states such as direction change and speed change, thus breaking the limitation of traditional preset motion model on target motion state. On this basis, the module analyzes the predicted trajectory through an optimization algorithm to generate accurate tracking control instructions to ensure that the instructions can closely match the actual motion trend of the target. The execution control module highly cooperates with the tracking decision module to quickly respond to the tracking control instructions, drive the gimbal mechanism of the optoelectronic pod to accurately adjust the sensor pointing direction. Whether the target is in slow or fast motion state, the sensor can be quickly aimed at the target, effectively avoiding the tracking deviation problem caused by the lag of tracking instructions and the untimely adjustment of the gimbal in traditional systems, and greatly improving the stability and precision of target tracking.

[0050] The setting of the feedback optimization module also enables the system to have the ability of continuous self-optimization. The module monitors the target tracking state in real time, calculates the tracking deviation indicators through the construction of a scientific evaluation system, and accurately identifies the problems in the tracking process, such as sensor pointing deviation and inaccurate trajectory prediction. Based on these deviation indicators, the module automatically generates corresponding tracking strategy adjustment parameters, which can specifically optimize the synchronization accuracy of multi-modal data acquisition, the algorithm parameters of feature extraction, the model coefficients of trajectory prediction, and the adjustment speed of the gimbal mechanism, etc. By dynamically adjusting the running parameters of each module of the system, the system can continuously optimize the target recognition and tracking strategy according to the real-time tracking state and environmental changes, avoiding the disadvantages of traditional systems that cannot adapt to complex scenes due to fixed strategies. Even in the case of sudden environmental condition changes, such as sudden heavy rain and sudden high-speed direction change of the target, the system can quickly adjust the running state through feedback optimization to maintain efficient recognition and stable tracking of the target at all times, significantly improving the reliability and adaptability of the system in complex application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A timing diagram of the multi-modal target automatic recognition and tracking system based on the optoelectronic pod described in the present application;

[0052] Figure 2 A working principle diagram of the multi-modal data acquisition module;

[0053] Figure 3 A working principle diagram of the dynamic feature extraction module. DETAILED DESCRIPTION

[0054] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0055] With reference to Figure 1 The present application provides a multi-modal target automatic identification and tracking method and system based on an optoelectronic pod. The system comprises a multi-modal data acquisition module, a dynamic feature extraction module, an adaptive tracking decision module, an execution control module, and a feedback optimization module.

[0056] The multi-modal data acquisition module is based on an optoelectronic pod platform, and deploys visible light sensors, infrared thermal imaging sensors, and laser ranging sensors to acquire visible light image data, infrared thermal radiation data, and laser ranging point cloud data of a target, and performs spatio-temporal synchronization calibration on these multi-modal perception data to ensure consistency in time and space. The dynamic feature extraction module receives multi-modal perception data, applies a multi-scale convolutional neural network for processing, generates multi-level feature maps of the target, and labels significant regions of these feature maps to distinguish regions of different significance. The adaptive tracking decision module predicts the target motion trajectory based on the significant region labeling results, and generates tracking control instructions through an optimization algorithm to guide the tracking behavior. The execution control module drives the gimbal mechanism of the optoelectronic pod based on the tracking control instructions to adjust the pitch angle and azimuth angle of the sensors, achieving precise control of the sensor pointing. The feedback optimization module monitors the target tracking state in real time, calculates tracking deviation indicators, generates tracking strategy adjustment parameters, dynamically optimizes the entire identification and tracking strategy, and ensures continuous and efficient operation of the system.

[0057] Embodiment 1: With reference to Figure 2 Precise sensor integration and configuration are performed on the optoelectronic pod platform, which serves as the basic structure for carrying core perception devices. The mechanical design and electrical interface of the platform need to meet the rigid requirements of multi-sensor collaborative work. The visible light sensor uses a high-resolution CMOS imager, which captures texture and color information of the target in the visible light spectrum range, and its output is a sequence of image data streams. The infrared thermal imaging sensor is based on the microbolometer principle and perceives the thermal radiation characteristics of the target object to generate infrared image data representing temperature distribution. The laser ranging sensor emits a laser beam and receives the echo to calculate the time of flight, and then outputs point cloud data describing the distance information of the target. When physically installed, the three sensors need to fully consider the parallelism of the optical axis and the relative position relationship, and reduce the observation error caused by parallax through mechanical calibration.

[0058] The simulation multi-sensor cooperative acquisition mechanism is the core link to realize efficient data acquisition. The mechanism dynamically manages the sampling frequency and resolution parameters of each sensor through a central control unit. According to the input target prior information, such as the estimated size, motion speed or category attribute of the target, the control unit divides the observation area of the target into high sampling rate area and low sampling rate area. The high sampling rate area usually corresponds to the core part of the target or the key points of the motion trajectory. In this area, the system instructs the visible light sensor and the infrared sensor to collect data at the highest frame rate and full resolution, while the laser ranging sensor increases the scanning density. The low sampling rate area is for the peripheral or background area of the target. The system automatically reduces the sampling frequency and can use region cropping or pixel merging to reduce the data volume. This dynamic configuration generates control instruction sets in real time, which are sent to each sensor driver. The instruction set includes specific exposure time, gain, frame rate, scanning mode and other parameters, finally making the system acquire multi-modal perception data with optimal information quantity under limited resources.

[0059] The space-time synchronization calibration is the premise to ensure the effective fusion and subsequent processing of multi-modal data. This process includes time synchronization and space synchronization. Time synchronization relies on a high-precision hardware clock source to stamp all sensor data with a unified timestamp. The timestamp precision needs to reach microseconds to eliminate motion blur caused by sampling time differences. When the data stream is transmitted to the processing unit through the high-speed bus, the software algorithm further interpolates and aligns the timestamp to ensure that each frame of visible light image, infrared image and laser point cloud data corresponds to the observation state at the same time. Space synchronization is achieved through the pre-calibrated inter-sensor transformation matrix, which describes the rotation and translation relationship between the coordinate systems of each sensor. In the processing process, the laser point cloud data is projected into the image coordinate system, and the infrared image is aligned with the visible light image at the pixel level through the registration algorithm, so that all modal data are unified into the same space-time reference system, forming standardized multi-modal perception data that can be used for subsequent feature extraction. The software implementation of the entire data acquisition module is based on a multi-thread architecture, with each sensor having an independent acquisition thread. Threads communicate and exchange data through shared memory and synchronization primitives. The analysis and issuance of control instructions, the state monitoring of sensors, the temporary caching and synchronization of data are all coordinated by a main scheduling thread. This design ensures the real-time and reliability of data acquisition. The final output multi-modal perception data packet not only contains the aligned sensor raw data, but also includes metadata such as acquisition parameters, timestamps and coordinate system conversion information.

[0060] Example 2: see Figure 3The processing flow of the dynamic feature extraction module begins with a detailed preprocessing operation on the multi-modal perception data. The raw visible light image, infrared thermal image, and laser point cloud data from the data acquisition module first need to be subjected to mean normalization. This step unifies the different numerical ranges of the various modal data to the same scale range. The pixel values of the visible light image are usually linearly converted to between 0 and 1, the infrared thermal radiation data is normalized according to its temperature sensitivity characteristics, and the laser ranging point cloud data is standardized according to its spatial coordinate range. The normalized multi-modal data is integrated according to the modal dimension to form a multi-dimensional tensor data structure with multiple channels. This tensor serves as the input of the subsequent deep neural network, maintaining the correspondence between the modal data.

[0061] The multi-scale convolutional neural network constructed adopts an encoder-decoder architecture. The encoding path is stacked with multiple convolutional layers and pooling layers, each of which uses a convolution kernel of a specific size for feature extraction. In the encoding process, the shallow network captures local detailed features of the target, such as edge and texture information, and as the network depth increases, the deeper network gradually extracts more abstract and global semantic features. A nonlinear activation function is added after each convolutional layer to enhance the network's expression ability, and the pooling operation gradually reduces the spatial dimension of the feature map, expanding the receptive field. The decoding path gradually restores the spatial resolution of the feature map through upsampling operations, using transposed convolution or interpolation methods to fuse high-level semantic features with shallow-level detail features. In this process, a skip connection mechanism is introduced to directly pass the feature maps of the corresponding scale in the encoding path to the decoding path, helping the network better reconstruct the spatial detail information of the target.

[0062] At different output levels of the multi-scale convolutional neural network, one-dimensional convolution is applied for feature channel adjustment and optimization. This operation can effectively integrate feature information of different scales and adjust the number of feature channels to remain consistent. Subsequently, the feature maps of multiple scales are fused using weighted summation or channel concatenation to generate multi-level feature maps containing rich contextual information. These feature maps contain both detailed information of the target and abstract features at the semantic level. The salient region labeling process first calculates the feature response value of each spatial position in the multi-level feature map. This value is obtained by weighting and fusing the features of visible light, infrared, and point cloud data. The weight coefficients are dynamically adjusted according to the importance and reliability of different modal data. The weight of visible light data is higher when the lighting conditions are good, the weight of infrared data increases in low light or camouflage situations, and the weight of laser point cloud data is higher in distance measurement. The calculation of the feature response value takes into account the intensity, contrast, and spatial distribution characteristics of the modal features, forming a comprehensive saliency metric.

[0063] The preset feature response value first threshold and second threshold divide the feature region into three saliency levels, which are obtained by a large amount of experimental data statistical analysis and can adapt to different scenes and target types. When the feature response value of a certain region is lower than the first threshold, the region is marked as a low saliency area, which usually corresponds to the background or irrelevant area. The region with feature response value between the first and second thresholds is marked as a medium saliency area, which may contain relevant context information of the target. And the region with feature response value exceeding the second threshold is marked as a high saliency area, which is often the core part or key feature area of the target. Different saliency areas are labeled using differentiated labels, and the low saliency area is usually represented by the value 0 or a specific color code, the medium saliency area is represented by the value 1 or another set of color coding, and the high saliency area is represented by the value 2 or a prominent color. This labeling method not only retains the saliency level information, but also provides an intuitive visual reference for subsequent tracking decision-making. The multi-level feature map with saliency labeling contains rich target information and provides accurate input data for the adaptive tracking decision-making module.

[0064] Taking the tracking of a moving target at sea as an example, the optoelectronic pod is deployed at a coastal observation station, and needs to continuously identify and track a specific ship. The multi-modal data acquisition module synchronously acquires visible light images, infrared thermal imaging and laser ranging data of the target. The visible light sensor captures the contour and deck structure of the ship under daylight conditions, the infrared thermal imaging sensor detects the thermal radiation characteristics of the ship engine, and the laser ranging sensor accurately measures the real-time distance between the ship and the observation station. After receiving these multi-modal data, the dynamic feature extraction module first performs mean normalization. The pixel values of the visible light image are converted to the range of 0-1, the infrared thermal imaging data are normalized according to their temperature range, and the laser ranging data are scaled according to the maximum range. The normalized three types of data are integrated into a three-dimensional tensor according to the channel dimension, wherein the visible light data occupy the RGB channel, the infrared data are a separate channel, and the laser ranging data are converted into a depth map form as the fourth channel.

[0065] The multi-dimensional tensor is input into a multi-scale convolutional neural network for processing. The encoding path of the network contains multiple convolutional layers, with shallow networks identifying edge and contour features of the ship and deep networks extracting more abstract features such as ship type and structural information. In the decoding path, high-level features gradually recover spatial details through upsampling operations and are fused with shallow features. The network finally outputs multi-level feature maps containing features of different scales, which not only retain the detailed features of the ship but also contain information at the semantic level. When labeling the generated multi-level feature maps with salient regions, the feature response value of each spatial position is calculated. The feature response value of the ship contour region in the visible light channel is high, the feature response value of the engine heat source region in the infrared channel is significant, and the feature response value of the ship entity region in the laser ranging channel is prominent. By weighting the feature response values of these modalities, a comprehensive feature response map is obtained.

[0066] The preset first and second threshold values of the feature response value divide the ship region into different saliency levels. The engine region of the ship has both obvious heat radiation features and geometric features, and its feature response value exceeds the second threshold value, which is marked as a high saliency area. The deck region of the ship has obvious features in the visible light image but weak heat radiation features, and its feature response value is between the two thresholds, which is marked as a medium saliency area. The sea surface region around the ship has a low feature response value, which is lower than the first threshold value, which is marked as a low saliency area. Different labels are used to label these saliency areas, with the high saliency area marked in red, the medium saliency area marked in yellow, and the low saliency area marked in blue. The labeled feature map clearly shows the importance difference of different regions of the ship, providing a basis for subsequent tracking decisions.

[0067] In embodiment 3, the adaptive tracking decision module deploys a differentiated tracking point distribution pattern within the target spatial domain based on the saliency region labeling results provided by the dynamic feature extraction module. A dense tracking point cloud is deployed in the high saliency area to improve tracking accuracy, a medium density distribution is used in the medium saliency area, and sparse point distribution is used in the low saliency area to optimize computing resources. Different prediction strategies are used for regions of different saliency levels. A recurrent neural network constructed by a gated recurrent unit is used to sequence model the target motion, with the input being a sequence of spatiotemporal coordinates of historical trajectory points and the output being a position prediction at a future time step. Initial trajectory prediction is generated based on the current observation sequence, and the prediction time step is dynamically adjusted according to the region saliency. A shorter prediction step is used in the high saliency area to maintain tracking sensitivity, a medium step is used in the medium saliency area, and a longer prediction step is used in the low saliency area to achieve forward-looking prediction.

[0068] An online state estimation algorithm based on maximum a posteriori estimation is applied to adjust the prediction time step, which adaptively corrects the step size parameter by analyzing the statistical characteristics of the prediction error. A multi-model interaction motion modeling framework is introduced, which includes uniform speed model, uniform acceleration model and cooperative turning model, etc. Various motion modes are weighted and fused through interaction probability. The iterative filtering algorithm optimizes the parameters in the expectation maximization framework, and recalculates the state estimation value and model parameters in each iteration. When the number of iterations reaches the preset maximum value or the parameter variation is less than the convergence threshold, the optimization process is terminated.

[0069] The tracking control command is generated based on the nonlinear model predictive control theory, and the following optimization problem is constructed:

[0070]

[0071] Wherein: represents the prediction time domain length, is the predicted position vector at time k, is the reference position vector at time k, is the state error weight matrix, is the control time domain length, is the control increment vector at time k, is the control increment weight matrix. The optimal control sequence is obtained by solving the optimization problem, and the first control command is extracted and output to the actuator.

[0072] The execution control module establishes a right-handed rectangular coordinate system with the physical mounting point of the photoelectric pod as the coordinate origin, and obtains the intrinsic matrix and extrinsic transformation relationship of the gimbal mechanism through a precise calibration process. The intrinsic matrix includes optical parameters such as focal length, principal point coordinates and distortion coefficients, and the extrinsic transformation represents the rotation and translation relationship between the gimbal coordinate system and the global coordinate system. The tracking control command is converted from the prediction coordinate system to the global coordinate system, and the quaternion interpolation method is used for smooth transition to ensure the continuity and stability during the coordinate system conversion. In the global coordinate system, the converted tracking control command is aligned with the current pose of the gimbal, and the deviation between the current gimbal azimuth angle, pitch angle and the required pose is calculated. When generating the gimbal adjustment command, an incremental control strategy is adopted, and the pose deviation is converted into motor driving signals through a proportional-integral-derivative controller. The pitch motor and azimuth motor of the gimbal mechanism are controlled through high-precision servo drivers, and closed-loop control is used to correct the execution error in real time, realizing accurate and rapid adjustment of the sensor pointing. During the entire execution process, the gimbal load state and motion parameters are continuously monitored, and the protection mechanism is automatically triggered when abnormal vibration or overload is detected, ensuring the reliability and stability of the system operation.

[0073] Embodiment 4: Synchronous acquisition and processing of multiple data sources, the execution parameters of the PTZ mechanism include azimuth angle, elevation angle, angular velocity and acceleration data, which are obtained in real time through the encoders and inertial measurement units installed on the PTZ motor. The actual position data of the target is obtained through a multi-modal sensor fusion algorithm, which integrates the target pixel coordinates in the visible light image, the temperature distribution characteristics of the infrared thermal image, and the spatial point cloud information of the laser ranging, and calculates the three-dimensional position coordinates of the target in the global coordinate system. The deviation calculation between the actual position data and the predicted position data uses the Euclidean distance formula, which performs difference operation on two coordinate points in three-dimensional space to generate a quantitative tracking deviation index value, which reflects the gap between the current tracking accuracy and the expected performance.

[0074] The preset tracking deviation index threshold system contains two key values, the first threshold is set as the upper limit of normal fluctuation within the allowed range, and the second threshold is defined as the critical warning line of system performance. When the calculated tracking deviation index is less than the first threshold, the system determines that the current tracking state is within the stable range, and maintains the existing parameter operation. If the deviation index is between the first and second thresholds, the system generates a judgment conclusion that the state needs to be fine-tuned, and starts the parameter adjustment process. When the deviation index exceeds the second threshold, the system determines that the tracking state is abnormal, triggering the emergency handling mechanism. These threshold values are dynamically configured according to the specific application scenario and target characteristics, for example, higher threshold values are used for high-speed moving targets, and lower threshold values are used for precise tracking scenarios. The process of dynamically optimizing the multi-modal target automatic identification and tracking strategy is based on quantitative evaluation, and the optimization target system contains two core dimensions of tracking accuracy and response speed. The tracking accuracy target requires minimizing position deviation, and the response speed target pursues maximizing system adjustment rate. Limiting conditions include upper limit of computing resource occupation, PTZ mechanical movement limit, and energy consumption constraints, etc. These conditions jointly constitute the boundary constraints of the optimization problem. The heuristic optimization algorithm uses an improved genetic algorithm, which represents different tracking strategy parameter combinations through chromosome coding. Each chromosome contains key parameters such as filter coefficients, prediction step, control gain, etc.

[0075] Randomly generated multiple sets of tracking strategy parameter schemes form the initial population, and each scheme represents a complete set of system parameter configurations. When evaluating each scheme, the system runs the parameter configuration in a simulation environment, collects real-time data on tracking accuracy and response speed, and the accuracy evaluation is completed by calculating the root mean square value of the position deviation, and the speed evaluation measures the time delay from the occurrence of the deviation to the completion of the adjustment. The completion of the two optimization objectives is scored separately on a 100-point scale, the accuracy score is inversely proportional to the deviation value, and the speed score is inversely proportional to the delay time. Finally, according to the application requirements, different weight coefficients are assigned to the two scores, and a weighted sum is obtained to get the comprehensive score. From all the evaluation schemes, the parameter configuration with the highest comprehensive score is selected as the final optimization scheme, and this selection process not only considers the total score, but also analyzes the performance stability of each scheme in a specific scenario. The selected final optimization scheme is applied to the actual running system through the parameter configuration interface, updating the key settings such as the filter parameters of the control algorithm, the confidence coefficients of the motion model, and the time domain parameters of the trajectory prediction. The entire optimization process uses a closed-loop operation mode, periodically re-evaluates system performance and triggers a new round of optimization to ensure that the system can adapt to environmental changes and target characteristic changes. Referring to Table 1, the scheme evaluation data within a certain optimization period is shown, reflecting the performance of different parameter configurations.

[0076] Table 1: Evaluation results of tracking strategy parameter schemes

[0077]

[0078] The weight distribution is 0.6 for accuracy and 0.4 for speed, and the comprehensive score calculation formula is: accuracy score x 0.6 + speed score x 0.4. According to the evaluation results, scheme 002 obtains the highest comprehensive score, and its parameter configuration will be adopted as the new running parameter. After the system executes the parameter update, it continues to monitor the actual performance and records the new deviation data to provide input for the next round of optimization, forming a closed-loop optimization mechanism for continuous improvement.

[0079] Embodiment 5: A preset time period trigger mechanism, which can be dynamically adjusted according to the system running state, is usually set as a fixed time interval or an event trigger based on the amount of data accumulation. When the period condition is met, the module first obtains the historical tracking data of the target from the persistent storage, which includes the historical position coordinate sequence, the historical sensor parameter configuration record and the corresponding timestamp information. The historical position data is stored in the form of three-dimensional coordinates, including azimuth angle, pitch angle and distance value, and the historical sensor parameters include focal length, exposure time, sampling rate and other device configuration information. At the same time, the module obtains real-time tracking data through the data interface, including the real-time position coordinates and real-time sensor working parameters of the target at the current time, which come from the ongoing tracking task of the system and have the latest time mark.

[0080] According to historical tracking data and real-time tracking data, the module constructs a data deviation vector to describe the data change characteristics. Each dimension of the deviation vector corresponds to a data index, for example, the position deviation dimension calculates the Euclidean distance between the historical position and the real-time position, and the parameter deviation dimension calculates the difference value of the sensor parameter configuration. For position data, the deviation calculation considers the difference value of the spatial coordinates; for sensor parameters, the deviation calculation uses the relative change rate or the absolute difference value, depending on the parameter type. After all the deviation components are normalized, they are combined into a multi-dimensional deviation vector, which comprehensively reflects the difference between the historical data and the real-time data. When retrieving a set of deviation modulus values that satisfy the data deviation vector, the module first calculates the deviation modulus value corresponding to each historical data point and the real-time data. The deviation modulus value is obtained by calculating the norm of the deviation vector, usually using the L2 norm or a custom norm weighted according to the application scenario. These modulus values form a time series set, reflecting the law of data deviation change over time. When the variance calculation result of the deviation modulus value set shows that its value is greater than or equal to the preset variance threshold, it indicates that the data fluctuation degree is large, there are more anomalies or outdated data, and cleaning operations are needed. The variance threshold is set according to the storage capacity requirement and the data importance, usually determined by statistical analysis of historical data characteristics.

[0081] An adaptive algorithm based on the variance value is used to calculate the data storage ratio, which defines the percentage of the amount of historical data that needs to be retained in the total data. When the variance is large, the storage ratio is correspondingly reduced, allowing more data to be cleaned; when the variance is small, the storage ratio is increased, retaining more historical data. The ratio calculation considers factors such as current storage space usage, data importance, and system performance requirements, and determines the final ratio value through a weighted function. According to the calculated data storage ratio, the module performs cleaning operations on the historical tracking data. The cleaning process uses a priority strategy, first deleting data records that are older in time and less frequently used, and retaining recently used and key node data. For position data, the trajectory integrity is considered during cleaning to avoid damaging continuous trajectory segments; for sensor parameter data, key node records of parameter changes are retained during cleaning. After cleaning is completed, the module updates the data index and metadata information to ensure the accessibility and consistency of the remaining data. The entire data management process is automatically executed periodically, forming a continuous data maintenance mechanism, which not only ensures the effectiveness of the data, but also optimizes the use efficiency of the storage resources.

[0082] As an example, assume that the system is configured to perform a data management cycle every 24 hours. At the beginning of a cycle, the module retrieves historical tracking data for the past 30 days, containing 100,000 location records and corresponding sensor parameters. Real-time data shows that the current target location is (120.5, 35.8, 1000), and the sensor is configured in a telephoto mode with a high sampling rate. The module calculates the deviation vector for each historical data point from the real-time data, including location deviation and parameter configuration deviation. The calculated deviation modulus set shows a variance value of 0.85, which is higher than the configured 0.7 variance threshold. According to the preset algorithm, the data storage ratio is calculated to be 65%. The module accordingly cleans up 35% of the historical data, preferentially deleting older and more deviated records, while retaining the main data and key node records for the last 7 days. After cleaning, the system optimizes the amount of stored data while maintaining the integrity of important historical information.

[0083] It should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0084] While the embodiments of the application have been illustrated and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and alterations can be made therein without departing from the spirit and scope of the application, which is defined by the following claims and their equivalents.

Claims

1. An opto-electronic pod-based multi-modal target automatic identification and tracking system, characterized in that, The method comprises the following steps: a multi-modal data acquisition module based on an optoelectronic pod integrated with a visible light sensor, an infrared thermal imaging sensor and a laser ranging sensor acquires multi-modal perception data of a target and performs spatio-temporal synchronization calibration on the multi-modal perception data; a dynamic feature extraction module applies a multi-scale convolutional neural network to the multi-modal perception data to generate multi-level feature maps of the target and performs salient region labeling on the multi-level feature maps; an adaptive tracking decision module deploys a recurrent neural network according to the salient region labeling results, predicts the motion trajectory of the target and generates tracking control instructions through an optimization algorithm; an execution control module drives the gimbal mechanism of the optoelectronic pod based on the tracking control instructions to adjust the pointing direction of the sensors to achieve target tracking; a feedback optimization module monitors the target tracking state in real time, calculates tracking deviation indicators, generates tracking strategy adjustment parameters and dynamically optimizes the multi-modal target automatic identification and tracking strategy.

2. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 1, characterized in that, The multi-modal data acquisition module comprises: visible light sensor, infrared thermal imaging sensor and laser ranging sensor are deployed on the optoelectronic pod platform to respectively collect visible light image data, infrared thermal radiation data and laser ranging point cloud data of the target; a multi-sensor collaborative acquisition mechanism is simulated to dynamically adjust the sampling frequency and resolution; based on the target prior information, the target core area is recorded as a high sampling rate area and the target peripheral area is recorded as a low sampling rate area; sensor parameters are dynamically configured through control instructions to obtain the final multi-modal perception data.

3. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 2, characterized in that, The dynamic feature extraction module applies a multi-scale convolutional neural network to the multi-modal perception data to generate multi-level feature maps of the target, comprising: performing mean normalization on the multi-modal perception data and integrating the normalized multi-modal perception data into a multi-dimensional tensor according to the modal dimension; constructing a multi-scale convolutional neural network structure containing an encoding path and a decoding path; in the encoding path, a deep convolutional network is used to extract hierarchical features from the input multi-modal perception data to generate feature maps of different scales; in the decoding path, the high-level feature maps are restored to spatial details through upsampling operation; applying one-dimensional convolution to each output layer of the multi-scale convolutional neural network structure to adjust the feature channel and fuse the feature maps of different scales output by the multi-scale convolutional neural network structure to generate multi-level feature maps of the target.

4. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 3, characterized in that, The dynamic feature extraction module performs salient region labeling on the multi-level feature maps, comprising: calculating the feature response value of each region in the multi-level feature maps by weighting and fusing the visible light image data, infrared thermal radiation data and laser ranging point cloud data to obtain the feature response value; presetting a first threshold value of the feature response value and a second threshold value of the feature response value, comparing the feature response value with the first threshold value of the feature response value and the second threshold value of the feature response value respectively; if the feature response value is less than the first threshold value of the feature response value, the region is marked as a low salient region; if the feature response value is greater than the first threshold value of the feature response value and less than the second threshold value of the feature response value, the region is marked as a medium salient region; if the feature response value is greater than the second threshold value of the feature response value, the region is marked as a high salient region. The multi-level feature map is labeled with different labels to define low saliency regions, medium saliency regions and high saliency regions.

5. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 4, characterized in that, The adaptive tracking decision module generates tracking control instructions through an optimization algorithm, and the tracking control instructions comprise: According to the saliency region labeling result, tracking points with different densities are deployed in the target space domain; different prediction strategies are adopted for different saliency regions, a sequence model of target motion is established through a recurrent neural network for initial trajectory prediction, and the prediction time step is dynamically adjusted according to the saliency region labeling result, wherein the prediction time step of the high saliency region is smaller than that of the medium saliency region, and the prediction time step of the medium saliency region is smaller than that of the low saliency region; A state estimation algorithm is applied to dynamically adjust the prediction time step, a motion model of multi-model interaction is introduced, parameters in the motion model are continuously corrected through an iterative filtering algorithm, and the iterative filtering algorithm is stopped when the maximum number of iterations is reached; and a tracking control instruction is generated based on a path planning algorithm.

6. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 5, characterized in that, The execution control module comprises: A global coordinate system is defined with the photoelectric pod mounting point as the origin; the intrinsic and extrinsic parameters of the holder mechanism are obtained through a calibration process; The tracking control instruction is converted into the global coordinate system to obtain the tracking control instruction in the global coordinate system; the tracking control instruction is aligned with the current pose of the holder mechanism in the global coordinate system to generate a holder adjustment instruction; The holder adjustment instruction is used to control the pitch angle and azimuth angle of the holder mechanism to realize sensor pointing adjustment.

7. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 6, characterized in that, The feedback optimization module comprises: Execution parameters of the holder mechanism are collected, actual position data of the target is synchronously obtained, a deviation between the actual position data and predicted position data is calculated, and a tracking deviation index is generated; A first threshold of the tracking deviation index and a second threshold of the tracking deviation index are preset; if the tracking deviation index is less than the first threshold of the tracking deviation index, it is determined that the tracking state is stable; if the tracking deviation index is greater than the first threshold of the tracking deviation index and less than the second threshold of the tracking deviation index, it is determined that the tracking state needs to be fine-tuned; and if the tracking deviation index is greater than the second threshold of the tracking deviation index, it is determined that the tracking state is abnormal.

8. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 7, characterized in that, The feedback optimization module dynamically optimizes the multi-modal target automatic identification tracking strategy, and the method comprises: Based on the tracking deviation index, an optimization target is established, the optimization target comprises a tracking accuracy target and a response speed target; and a restriction condition is set; A heuristic optimization algorithm is used to randomly generate a plurality of tracking strategy parameter schemes, the completion of the optimization target is evaluated for each scheme, a completion score of the optimization target is obtained, a weighted sum of the completion scores of the optimization target is calculated, a comprehensive score is obtained, and a scheme with the highest comprehensive score is selected from the plurality of schemes as a final optimization scheme; The selected final optimization scheme is applied to the multi-modal target automatic identification tracking strategy.

9. The multi-modal target automatic identification and tracking system based on the optoelectronic pod according to claim 8, characterized in that, The data management module is further configured to: When a preset period is met, historical tracking data of the target is obtained, the historical tracking data comprising historical position data and historical sensor parameters; Real-time tracking data of the target is obtained, the real-time tracking data comprising real-time position data and real-time sensor parameters; A data deviation vector is constructed based on the historical tracking data and the real-time tracking data. Retrieving a deviation modulus value set satisfying a data deviation vector, calculating a data storage ratio when a variance of the deviation modulus value set is greater than or equal to a variance threshold value; Cleaning historical tracking data according to the data storage ratio.

10. A method for optoelectronic pod-based multi-modal target automatic recognition tracking, characterized in that, The multi-modal target automatic identification tracking system based on the optical-electric pod according to any one of claims 1 to 9 performs multi-modal target automatic identification tracking.

Citation Information

Patent Citations

  • Photoelectric pod image tracking method and tracking system based on fusion IMU

    CN110400330A

  • Photoelectric pod target detection system based on abnormal motion detection

    CN120039413A

  • Photoelectric pod target locking system based on multi-target detection tracking

    CN120143661A

  • Pavement defect detection method, device and equipment and storage medium

    CN120163760A

  • Target identification tracking method and system based on multi-source fusion imaging

    CN120182323A

Cited By

  • Azimuth adaptive stability control system for photoelectric pod

    CN121386361A

  • Azimuth adaptive stabilization control system for optoelectronic pods

    CN121386361B

  • Pod cascade aiming method based on prediction-feedforward control

    CN122064119A