Water surface target tracking system and tracking method under incomplete information condition

Abstract

The invention belongs to the technical field of water surface target identification and tracking, and particularly relates to a tracking system and a tracking method for a water surface target under an incomplete information condition, which can be used for solving the problems of incomplete target information and tracking loss caused by wave shielding of an unmanned ship in a marine environment. The multi-modal feature extraction and fusion module is used for improving a convolutional neural network to extract visual features, constructing motion feature vectors in combination with multi-sensor motion information, and performing weighted fusion to obtain multi-modal features; the dynamic target feature library management module stores feature vectors and timestamps through a hash table-linked list structure, and maintains aging by means of a sliding window; when the target is reproduced, the re-identification algorithm module based on space-time joint matching integrates the feature similarity, the position distance and the time interval to determine the ID; the tracking parameter correction module uses Kalman filtering and combines a motion model and an observation value to correct parameters. The method improves the accuracy and stability of target tracking, and has application prospects in the fields of marine rescue, marine monitoring and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of surface target recognition and tracking, and in particular relates to a surface target tracking system and method under incomplete information conditions. Background Art

[0002] With the continuous development and utilization of the ocean, unmanned aerial vehicles (UAVs), as an important ocean exploration and operation tool, have been widely used in a variety of fields, including maritime rescue, ocean monitoring, and military reconnaissance. During the execution of UAV missions, target tracking is a key technology, and its tracking effect directly affects the quality of mission completion.

[0003] In the real world, unmanned aerial vehicles (UAVs) face complex and ever-changing conditions in the ocean. Waves are a significant factor affecting target tracking. When a target is obscured by waves, the UAV's sensors, such as cameras and radar, cannot fully capture the target, and may even temporarily lose sight of the target. This not only prevents the UAV from effectively and continuously tracking the target, but can also impact the progress and accuracy of the entire mission.

[0004] In existing technologies, many unmanned aerial vehicle (UAV) target tracking systems use a single sensor for target monitoring and tracking. For example, they rely solely on cameras to obtain visual information about the target. However, cameras are significantly affected by weather, light, and wave obstruction. When waves obscure the target, the target disappears from the camera's image, making it difficult for the system to continue tracking the target. While radar-based tracking systems can overcome the effects of weather and light to a certain extent, when wave echo signals interfere with the target signal, the radar struggles to accurately identify the target, resulting in target tracking failure.

[0005] Some research has attempted to improve target tracking stability by employing multi-sensor fusion. However, these approaches still have significant shortcomings when dealing with wave occlusion. During brief periods when a target is temporarily obscured by waves, multi-sensor fusion systems struggle to effectively predict and estimate its position and motion. When the target reappears, the system takes a significant amount of time to re-identify and re-lock on it, which is unacceptable in scenarios requiring high real-time tracking performance, such as maritime rescue and military reconnaissance.

[0006] Furthermore, traditional target tracking algorithms are mostly designed based on complete target information. Their performance degrades dramatically when incomplete information is present. For example, the common Kalman filter algorithm cannot accurately predict the target's next position when target information is missing, leading to increased tracking error or even target loss. Furthermore, existing unmanned vehicle target tracking systems lack the ability to adaptively adjust tracking strategies in complex marine environments, making it difficult to dynamically optimize tracking algorithms and parameters based on actual conditions, thus compromising tracking effectiveness.

[0007] In the field of ocean monitoring, unmanned aerial vehicles (UAVs) are required to continuously track and monitor specific marine targets, such as floating debris and marine life. If the target is lost due to wave obstruction, the target's continuous motion trajectory and related information cannot be obtained, affecting the accurate assessment of the marine ecological environment and marine pollution status. In maritime rescue operations, timely and accurate tracking of the target in distress is crucial for carrying out rescue operations. Once the target is lost due to wave obstruction, rescue efforts may be delayed, resulting in irreparable losses.

[0008] In summary, existing unmanned vehicle target tracking technologies have many drawbacks when dealing with the incomplete information problem caused by wave occlusion, and cannot meet the needs of practical applications. Therefore, developing a technology that can achieve continuous tracking of targets by unmanned vehicles under incomplete information conditions is of great practical significance. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the present invention provides a system and method for tracking surface targets under incomplete information conditions. The present invention aims to provide a target tracking technology for unmanned vehicles (UAVs) under incomplete information conditions, addressing the problem of target loss caused by temporary wave obscuration during target tracking, and improving the stability and accuracy of target tracking by UAVs in complex marine environments.

[0010] The present invention provides a surface target tracking system under incomplete information conditions, comprising a multimodal feature extraction and fusion module, a dynamic target feature library management module, a target re-identification algorithm module based on spatiotemporal joint matching, and a tracking parameter correction module;

[0011] The multimodal feature extraction and fusion module is used to extract the visual features and motion features of the target and perform weighted fusion, including:

[0012] A visual feature extraction unit is configured with a camera and an improved convolutional neural network model. The target image captured by the camera is normalized and then input into the improved convolutional neural network model. The improved convolutional neural network model calculates the channel dimension attention weight through the attention mechanism module to extract the visual feature vector;

[0013] a motion feature extraction unit connected to a multi-sensor fusion system for collecting position, velocity, and acceleration data of a target in a geographic coordinate system to construct a motion feature vector;

[0014] A feature fusion unit is used to determine weighted fusion weights based on the application scenario, perform weighted fusion on the visual feature vector and the motion feature vector, and generate a multimodal feature vector;

[0015] Dynamic target feature library management module, used to store and update target feature information, including:

[0016] The feature library storage unit is constructed with a hash table with the target ID as the key value. Each key value corresponds to a linked list for storing multimodal feature vectors and timestamps.

[0017] The feature library update unit is used to add the newly extracted multimodal feature vectors and timestamps to the corresponding linked list when the target is visible, and delete the nodes with earlier timestamps in the linked list through the sliding window mechanism to maintain timeliness;

[0018] The target re-identification algorithm module based on spatiotemporal joint matching is used to determine the target ID when the target reappears, including:

[0019] A similarity calculation unit is used to extract the multimodal feature vector of the reappearing target and calculate its similarity with the feature vector in the linked list corresponding to each target ID in the feature library;

[0020] The spatiotemporal matching unit is used to obtain the location information corresponding to the last feature vector of each target ID in the feature library, calculate the location distance and time interval between it and the newly appeared target, generate a comprehensive matching score based on the similarity, and determine the target ID.

[0021] Tracking parameter correction module, used to correct target tracking parameters, including:

[0022] Kalman filter initialization unit, used to take the position and velocity of the newly appeared target as the initial state vector and set the initial covariance matrix;

[0023] The state update unit is used to predict the state and covariance matrix of the target at the next moment based on the target motion model through the Kalman filter prediction formula, and uses the newly obtained target position and speed as observation values to correct the target state estimation value and covariance matrix through the update formula.

[0024] As a further optimization solution for the above tracking system, the attention mechanism module obtains the channel description vector through global average pooling, generates the channel attention weight through the fully connected layer, and multiplies it with the original feature map to enhance the extraction of target key features.

[0025] As a further optimization solution for the above tracking system, a multi-sensor fusion system integrates multiple sensors for collecting target position, velocity and acceleration data, which may include but are not limited to radar, lidar and inertial measurement unit.

[0026] As a further optimization scheme for the above tracking system, the sliding window mechanism deletes the nodes with earlier timestamps that are out of the window range in the linked list according to the set window size. The window size is dynamically adjusted according to the target movement speed and the ocean environment.

[0027] As a further optimization scheme for the above tracking system, the similarity calculation unit uses a combination of cosine similarity and Euclidean distance to calculate the feature vector similarity. The comprehensive matching score function of the spatiotemporal matching unit comprehensively considers the feature vector similarity, target position distance and time interval. The weight of each factor is determined through experiments or adaptive algorithms.

[0028] As a further optimization scheme for the above-mentioned tracking system, the Kalman filter algorithm of the state update unit includes a prediction step and an update step; the prediction step predicts the target state and covariance through the state transition matrix and the process noise covariance matrix; the update step uses the residual between the observed value and the predicted value to correct the target state estimate and the covariance matrix.

[0029] As a further optimization solution for the above-mentioned tracking system, the feature library storage unit adopts a composite data structure combining a hash table and a linked list, uses the target ID as the hash key value to achieve fast retrieval, and stores multimodal feature vectors and timestamps in chronological order through the linked list, supporting the query and dynamic update of target features.

[0030] As a further optimization scheme for the above tracking system, the weighted fusion weight of the feature fusion unit is dynamically adjusted through an adaptive algorithm. The adaptive algorithm adjusts the fusion ratio of visual features and motion features based on the target motion state (constant speed / acceleration / turning), ocean environment parameters (wave height / current speed) and sensor noise level.

[0031] As a further optimization scheme for the above-mentioned tracking system, when calculating the comprehensive matching score, the spatiotemporal matching unit assigns weights to the feature vectors in the linked list corresponding to the target ID in the feature library according to the distance of the timestamp. The feature vectors closer to the current time have higher weights. The weight function is an exponential decay function or a linear decay function to highlight the contribution of the latest features to target re-identification.

[0032] The present invention further provides a method for tracking a surface target under incomplete information conditions, comprising the following steps:

[0033] (1) Multimodal feature extraction and fusion steps: The target image is captured by a camera, and the visual feature vector of the target is extracted using an improved convolutional neural network. At the same time, the position, velocity, and acceleration data of the target in the geographic coordinate system are obtained through a multi-sensor fusion system to construct a motion feature vector. The weighted fusion weight is determined according to the application scenario, and the visual feature vector and the motion feature vector are weightedly fused to generate a multimodal feature vector.

[0034] (2) Dynamic target feature library management steps: Using the target ID as the index, store the multimodal feature vectors and the corresponding timestamps in chronological order, and delete the feature vectors with earlier timestamps through the sliding window mechanism to maintain the timeliness of the feature library;

[0035] (3) Target re-identification step of spatiotemporal joint matching: When the target reappears, its multimodal feature vector is extracted and the similarity with the feature vectors of each target stored in the feature library is calculated; the target position distance, time interval and similarity are combined to generate a comprehensive matching score to determine the target ID;

[0036] (4) Tracking parameter correction step: Based on the Kalman filter algorithm, the position and velocity of the newly appeared target are used as observation values, and the target state parameters are predicted and corrected in combination with the target motion model to achieve continuous tracking.

[0037] Beneficial effects

[0038] This invention addresses the problem of incomplete target information caused by wave obstruction in unmanned boats in complex marine environments. It achieves stable and accurate target tracking through multi-module collaborative operation, with significant beneficial effects:

[0039] Improve the accuracy of target re-identification: The present invention integrates visual features and motion features through a multimodal feature extraction and fusion method, providing rich and comprehensive information for target recognition. In the visual feature extraction stage, the improved ResNet-50 convolutional neural network introduces an attention mechanism, which enhances the extraction of key features of the target and can more accurately capture the appearance characteristics of the target. The weighted fusion of visual features and motion features takes into account the dynamic information of the target at different times, greatly improving the uniqueness and recognition of feature representation. In addition, the re-identification algorithm based on spatiotemporal joint matching not only calculates the similarity between feature vectors, but also comprehensively considers the motion trajectory and time information of the target, effectively avoiding misidentification due to feature similarity, and greatly improving the accuracy of target re-identification.

[0040] Ensuring the efficiency and timeliness of the feature library: The dynamic target feature library utilizes a structure that combines a hash table and a linked list, using the target ID as the key value. This facilitates rapid querying and updating of target feature information. Furthermore, a sliding window mechanism periodically deletes feature vectors with older timestamps from the linked list. This ensures that valid information is retained in the feature library while preventing uncontrolled data growth and guaranteeing the timeliness of the feature library. This enables matching based on the latest and most relevant feature information during target re-identification, further improving re-identification efficiency and accuracy.

[0041] Enhanced adaptability and stability of the tracking process: After target re-identification, the Kalman filter algorithm is used to update the target state based on the newly appeared target's position and velocity. This allows for rapid and accurate correction of tracking parameters, enabling the UAV to quickly adapt to changes in the target's state and ensuring the stability of the tracking process. Whether the target reappears after being briefly obscured by waves or undergoes a sudden change in its motion state, continuous and stable tracking of the target is achieved, significantly improving the UAV's mission execution capabilities in complex marine environments.

[0042] Expanding the application scope of unmanned boats: The technical solution of this invention effectively solves the problem of target tracking for unmanned boats under incomplete information conditions, enabling unmanned boats to more reliably complete tasks in areas with high target tracking requirements, such as maritime rescue, ocean monitoring, and military reconnaissance. For example, in maritime rescue, accurate target tracking helps to quickly locate targets in distress and improve the success rate of rescue. In ocean monitoring, the movement trajectory and status of marine targets can be continuously monitored, providing more accurate data support for marine ecological research and environmental monitoring. This greatly expands the application scope of unmanned boats and has broad market prospects and social value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the multi-sensor fusion module of the present invention.

[0044] Figure 2 This is a flow chart of the target motion prediction module of the present invention.

[0045] Figure 3 This is a flow chart of the adaptive tracking strategy adjustment module of the present invention.

[0046] Figure 4 This is the flow chart of the target re-identification module of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further illustrated below through specific embodiments. These embodiments are exemplary and are intended to illustrate the problem and explain the present invention, and are not intended to be limiting.

[0048] 1. Multi-sensor fusion module

[0049] Sensor selection and installation: The selection of sensors is determined based on the mission type of the unmanned boat, the target tracking range, and the complexity of the marine environment. For maritime rescue scenarios, it is necessary to quickly locate targets at a long distance, so a telephoto camera with high resolution and a wide field of view is selected to obtain detailed visual information of the target; to cope with severe weather and long-distance detection needs, it is equipped with an S-band radar, which has excellent penetration and anti-interference capabilities; at the same time, it is equipped with a laser radar based on the time of flight (ToF) principle to achieve accurate three-dimensional distance measurement of the target. In terms of installation layout, through spatial geometric analysis, the camera is installed on the top of the unmanned boat in an open field of view to ensure that its field of view covers the detection area of the radar and laser radar, while avoiding mutual obstruction between devices. In addition, electromagnetic compatibility analysis is used to rationally plan the position of the radar and other electronic equipment to prevent signal interference.

[0050] Data acquisition and preprocessing: Each sensor collects data from the target at a set sampling frequency f. The image sequence collected by the camera is represented as Among them I t is the image at time t. The image data is denoised using the Gaussian filter algorithm, and the formula Construct a Gaussian kernel, where σ is the standard deviation of the Gaussian distribution, and perform a convolution operation on the image to remove noise points. Then use the histogram equalization method to enhance the image contrast, and use the cumulative distribution function Redistribute the image grayscale value, where L is the image grayscale, n i is the number of pixels with gray value i, N i The radar collected echo signal is processed by mixing, filtering, etc. to obtain the target distance r, azimuth angle θ and radial velocity v r Information, point cloud data collected by lidar P i =(x i ,y i , z i ) represents the 3D coordinates of the i-th point. A voxel filtering algorithm is used to remove noise points. By dividing the space into voxels, the mean value of the points within each voxel is calculated to represent the point cloud information within that voxel. To ensure temporal consistency of data from different sensors, a timestamp-based synchronization method is used. The acquisition time of each sensor is calibrated using the UAV's system clock as a reference to ensure data synchronization.

[0051] Data fusion algorithm implementation: A data fusion algorithm based on joint probabilistic data association (JPDA) combined with Bayesian network is used to build a multi-sensor data fusion model. Assume that at time k, the observation sets obtained by radar, lidar and camera are where n r 、n l 、n c are the number of observations of radar, lidar and camera at time k respectively. Define the target state vector Where (x k ,y k , z k ) is the target position, is the target speed, is the acceleration of the target. According to the characteristics and error model of each sensor, the observation model of the sensor is established:

[0052] Radar observation model: where h r (X k ) is the radar observation function, is the radar observation noise, which obeys the Gaussian distribution N(0, R r ), R r is the radar observation noise covariance matrix.

[0053] LiDAR observation model: where h l (X k ) is the lidar observation function, is the laser radar observation noise, which obeys the Gaussian distribution N(0, R l ), R l is the lidar observation noise covariance matrix.

[0054] Camera observation model: where h c (X k ) is the camera observation function, is the camera observation noise, which obeys the Gaussian distribution N(0, R c ), R c is the camera observation noise covariance matrix.

[0055] According to the target's motion characteristics, establish a state transfer model: X k+1 =F k X k +G k u k +w k , where F k is the state transfer matrix, G k is the control input matrix, u k is the control input, w k is the process noise, which obeys Gaussian distribution N(0, Q k ), Q kis the process noise covariance matrix. Using the joint probability data association algorithm, the association probability β between each observation and the target state is calculated i，j , represents the degree of association between the i-th target and the j-th observation. Then, based on the Bayesian formula, combined with the observation model and the state transition model, the optimal estimate of the target state is obtained through recursive calculation:

[0056]

[0057] in is the estimated value of the target state assuming that the jth observation is associated with the i-th target, n t is the target number, is the total number of observations at time k. Through this data fusion algorithm, the information of each sensor is fully integrated to improve the accuracy and reliability of target state estimation, effectively addressing the problem of incomplete target information in marine environments.

[0058] Flowchart as Figure 1 shown.

[0059] 2. Target motion prediction module

[0060] Data collection and annotation: During the mission, the multi-sensor fusion system on board the unmanned boat continuously collects the target's motion data at time intervals Δt. This data includes the target's location information in the geographic coordinate system (x t ,y t , z t ), speed information And acceleration information Where t represents the sampling time. Considering the diversity of target motion in the marine environment, in addition to classifying the target motion state into conventional categories such as uniform linear motion, accelerated motion, and turning motion, a wave drift motion category is added for targets that are significantly affected by waves. For each motion sample, its motion category is accurately marked by combining manual labeling with automated algorithms to form a labeled training sample set {(X t , C t )},in is the motion feature vector, C t is the corresponding sports category label.

[0061] Model building and training:

[0062] Model Architecture Design: A Bi-LSTM-Attention model is used with an attention mechanism to capture long-term and short-term dependencies in target motion and highlight key motion features. The Bi-LSTM consists of a forward LSTM and a backward LSTM. The forward LSTM processes the input sequence in chronological order, while the backward LSTM processes it in reverse order. The outputs of the two are concatenated to form a feature representation that contains richer temporal context. The attention mechanism adaptively focuses on information relevant to the current prediction task by calculating attention weights at each time step of the input sequence.

[0063] Model training process: The collected labeled data is divided into training set and test set in a ratio of 8:2. During the training phase, the loss function is defined as the weighted sum of cross entropy loss and mean square error loss:

[0064] L=αL CE +(1-α)L MSE

[0065] in, is the cross entropy loss, which is used to measure the difference between the predicted motion category and the true category, y ij Indicates that the true label of the i-th sample belongs to the j-th category, Predict the probability that the i-th sample belongs to the j-th class for the model; is the mean square error loss, which is used to evaluate the deviation between the predicted motion parameters and the true parameters. is the motion parameter of the i-th sample predicted by the model, X i is the true motion parameter; α is a hyperparameter that balances the two loss terms and its value range is [0,1].

[0066] The Adam optimizer is used to iteratively train the model. The formula for updating the optimizer parameters is:

[0067]

[0068] Among them, m t and v t are the first-order moment estimate and the second-order moment estimate of the gradient, g t is the gradient at the current moment, β1 and β2 are exponential decay rates, and is the revised first-order moment estimate and second-order moment estimate, θ t is the model parameter at the current moment, η is the learning rate, and ε is a small constant to prevent division by zero. During training, the K-fold cross-validation method is used to divide the training set into K mutually exclusive subsets. Each time, one of the subsets is selected as the validation set, and the remaining K-1 subsets are used as the training set. The training is repeated K times, and the average of the K validation results is taken as the model performance indicator to prevent overfitting.

[0069] Target motion prediction: When the target is blocked by waves, the target motion data X before the occlusion is T time steps. t-T：t =[X t-T , X t-T+1 ,…,X t ] is input into the trained Bi-LSTM-Attention model. The model predicts the motion state of the next H time steps by learning the target motion pattern. To improve prediction accuracy, the prediction results are updated every ΔT based on the latest sensor data. If significant changes in ocean environmental parameters (such as wave height and current speed) are detected during the prediction process, the model's prediction step size and parameters are adaptively adjusted to better adapt to these environmental changes. For example, if wave height increases dramatically, the prediction step size H is appropriately reduced to reduce prediction error.

[0070] Flowchart as Figure 2 shown.

[0071] 3. Adaptive tracking strategy adjustment module

[0072] Multi-dimensional information collection and pre-processing: The unmanned boat is equipped with high-precision meteorological sensors, wave sensors and various sensors for monitoring target tracking status. The meteorological sensor collects wind speed V in real time. w 、wind direction θ w , the wave sensor obtains the wave height H s , wave cycle T p At the same time, with the help of multi-sensor fusion module and target motion prediction module, the real-time position of the target (x t ,y t , z t ),speed and the tracking error ε t In order to reduce the impact of sensor noise, the collected information is processed by Kalman filtering. Taking wind speed as an example, the state equation is V w，k =V w，k-1 +w k , the observation equation is Z w，k =V w，k +v k , where V w，k is the wind speed at the kth moment, w k is the process noise, Z w，k is the wind speed observation value at the kth moment, v k is the observation noise. The wind speed is estimated by the Kalman filter algorithm to obtain more accurate wind speed information. Other data are processed similarly.

[0073] Tracking strategy adjustment decision model based on deep reinforcement learning: Construct a decision model based on deep Q network (DQN), which takes processed environmental information and target tracking state information as input and outputs the optimal tracking strategy adjustment parameters. Define the state space S as a multidimensional vector containing weather information, wave information, target position, speed and tracking error, that is, The action space A contains tracking strategy adjustment parameters such as the tracking sensitivity threshold adjustment value Δτ and the target motion prediction update frequency adjustment value Δf.

[0074] Q-network construction: The Q-network employs a multi-layer neural network structure, consisting of an input layer, multiple hidden layers, and an output layer. The input layer receives information from the state space S. The hidden layers extract and transform the input information using nonlinear activation functions (such as the ReLU function: f(x) = max(0, x)). The output layer outputs the Q value corresponding to each action, namely Q(s, a; θ), where s∈S is the current state, a∈A is the action, and θ is the parameter of the Q-network.

[0075] Training process: Use the experience replay mechanism and Q learning algorithm to train the Q network. At each time step t, the unmanned boat is based on the current state s t Select action a t , after executing the action, you get the reward r t and the new state s t+1 , will (s t , a t , r t , s t+1 ) is stored in the experience replay buffer D. A batch of samples (s i , a i , r i , s i+1 ), by minimizing the loss function

[0076] To update the parameters θ of the Q network, where γ is the discount factor, θ - are the parameters of the target Q network, which periodically copies parameters from the main Q network to improve the stability of training.

[0077] Decision-making process: In actual applications, the unmanned boat selects the action with the largest Q value through the Q network according to the current state s This determines the adjustment parameters of the tracking strategy.

[0078] Tracking strategy adjustment implementation: According to the tracking strategy adjustment parameters output by the DQN model, the tracking strategy of the unmanned boat is adjusted in real time. s Large and tracking error ε tWhen the set threshold is exceeded, the tracking sensitivity threshold τ is reduced, and the adjustment formula is τ new =τ old -Δτ, where τ old is the tracking sensitivity threshold before adjustment, τ new is the adjusted tracking sensitivity threshold, and Δτ is the adjustment value output by the DQN model. At the same time, the target motion prediction update frequency f is increased, and the adjustment formula is f new =f old +Δf, where f old is the update frequency before adjustment, f new is the adjusted update frequency, and Δf is the adjustment output by the DQN model. When the target reappears, the tracking sensitivity threshold and target motion prediction update frequency are quickly restored based on the target's new position and motion state, ensuring that the unmanned vehicle can track the target in a timely and accurate manner.

[0079] Flowchart as Figure 3 shown.

[0080] 4. Target Re-Identification Module

[0081] Multimodal feature extraction and fusion

[0082] Visual feature extraction: The improved ResNet-50 convolutional neural network is used to process the target image captured by the camera. The attention mechanism module is introduced into the network structure to enhance the ability to extract key target features by calculating the attention weight in the channel dimension. Input image After a series of convolutional layers, pooling layers, and attention modules, the visual feature vector is obtained. Where H and W are the height and width of the image respectively, d v is the dimension of the visual feature vector.

[0083] Motion feature extraction: According to the position of the target at different times (x t ,y t , z t ),speed and acceleration Information, construct motion feature vector. Define motion feature vector where d m =9,

[0084] Feature fusion: Using weighted fusion, the visual feature vector V and the motion feature vector M are fused to obtain a multimodal feature vector where d f =d v +d m, the fusion formula is F = αV + (1-α)M, α∈[0, 1] is the fusion weight, which is adjusted according to different application scenarios and target characteristics.

[0085] Dynamic target feature library management

[0086] Feature Library Structure Design: The target feature library uses a hash table structure, using the target ID as the key value to store the corresponding multimodal feature vector and timestamp. The feature information of each target is organized in a linked list format to facilitate updating and querying the feature information.

[0087] Feature Library Update Strategy: When a target is visible, multimodal feature vectors are extracted at intervals of Δt and added to a linked list corresponding to the target ID. To prevent the feature library from growing indefinitely, a sliding window mechanism is used to periodically delete feature vectors with older timestamps from the linked list, maintaining the timeliness of the feature library. Assuming the sliding window size is N, only feature vectors from the most recent N time steps are retained.

[0088] Target Re-Identification Algorithm Based on Spatiotemporal Joint Matching

[0089] Similarity calculation: When the target reappears, extract its current multimodal feature vector F new , and calculate the similarity with the feature vector list corresponding to each target ID in the target feature library. The similarity between feature vectors is calculated by combining cosine similarity and Euclidean distance. For the jth feature vector F corresponding to the i-th target ID in the feature library, i，j , the similarity calculation formula is:

[0090]

[0091] Among them, β∈[0,1] is the weight coefficient used to balance the contribution of cosine similarity and Euclidean distance. and F new and F i，j The value of the kth dimension of .

[0092] Spatiotemporal joint matching: To further improve the accuracy of re-identification, the target's motion trajectory and time information are considered. For each target ID, the distance D between the position corresponding to the last feature vector in the feature library and the position of the newly appeared target is calculated. pos , and time interval ΔT. Comprehensive similarity S i，j , position distance D pos And time interval ΔT, construct a comprehensive matching score function:

[0093]

[0094] Among them, w j The jth feature vector is weighted based on the distance of its timestamp, with closer features receiving higher weights. γ1 and γ2 are weight coefficients that balance the effects of similarity, location distance, and time interval. The target ID with the highest overall matching score is selected as the ID of the reappearing target, enabling target re-identification.

[0095] Tracking parameter correction: After target re-identification, the UAV's tracking parameters are corrected based on the newly discovered target's position and motion state. Using the Kalman filter algorithm, the newly discovered target's position and velocity are used as observations to update the target's state, resulting in more accurate estimates of the target's position, velocity, and acceleration, providing a reliable basis for subsequent tracking.

[0096] Flowchart as Figure 4 shown.

[0097] Example

[0098] 1. Multimodal feature extraction and fusion

[0099] 1.1 Visual Feature Extraction

[0100] Equipment Selection and Installation: Select an industrial-grade camera with a resolution of at least 4K and a frame rate of at least 30fps. Mount it on top of the UAV in a location with a wide field of view to ensure that the camera's field of view covers the target tracking area. Calibrate the camera horizontally and vertically by placing standard targets at different distances and adjusting the camera's pitch and azimuth to ensure accurate image capture.

[0101] Data acquisition: Image data is continuously collected at the maximum frame rate of the camera, and the image storage format uses lossless RAW format to preserve the most original image information.

[0102] Feature Extraction: A modified ResNet-50 neural network model is deployed in the computing unit of the unmanned boat. Image data is normalized, mapping pixel values from [0, 255] to [0, 1] before being input into the neural network. During the network's forward propagation, the attention mechanism module calculates attention weights along the channel dimension. Taking the Squeeze-Excitation module as an example, global average pooling is used to compress the feature map of each channel into a single value, resulting in a channel description vector. This is then passed through two fully connected layers to generate an attention weight for each channel. This weight is multiplied with the original feature map to highlight the channel information that plays a key role in target recognition, outputting a visual feature vector of dimension d_v.

[0103] 1.2 Motion Feature Extraction

[0104] Data collection: With the help of the multi-sensor fusion system on the unmanned boat, the position of the target in the geographic coordinate system (x t ,y t , z t ),speed and acceleration data.

[0105] Vector construction: Combine the above 9-dimensional data in order to construct the motion feature vector where d m =9.

[0106] 1.3 Feature Fusion

[0107] Weight determination: The weight α of weighted fusion is determined experimentally based on different application scenarios. For example, in a maritime rescue scenario, where the target moves quickly and motion characteristics are more important for target recognition, α can be set to 0.3. In an ocean monitoring scenario, where the target's appearance characteristics are relatively stable, α can be set to 0.7.

[0108] Fusion calculation: According to the formula F = αV + (1-α)M, the multimodal feature vector is calculated where d f =d v +d m .

[0109] 2. Dynamic target feature library management

[0110] 2.1 Feature library structure construction

[0111] Initialization: Create a hash table in the storage unit of the unmanned vehicle, using the target ID as the key value. Each key value corresponds to a linked list. The linked list nodes are used to store multimodal feature vectors and their corresponding timestamps.

[0112] 2.2 Feature Library Update

[0113] Data addition: When the target is visible, every Δt = 1 second, the extracted multimodal feature vector and its timestamp are added as new nodes to the end of the linked list corresponding to the target ID.

[0114] Data cleaning: Set the sliding window size N = 10. When the number of nodes in the linked list exceeds N, delete the node with the earliest timestamp at the head of the linked list to maintain the timeliness of the feature library.

[0115] 3. Target Re-ID Algorithm Based on Spatiotemporal Joint Matching

[0116] 3.1 Similarity Calculation

[0117] Feature extraction: When the target reappears, the multimodal feature vector F of the current target is extracted according to the above multimodal feature extraction and fusion method. new .

[0118] Similarity calculation: traverse the linked list corresponding to each target ID in the target feature library, and calculate each feature vector F in the linked list. i，j , according to the formula Calculate the similarity. β is determined based on experiments and is set to 0.6.

[0119] 3.2 Joint Spatiotemporal Matching

[0120] Acquisition of position and time information: Obtain the position information corresponding to the last feature vector of each target ID in the feature library, and calculate the distance D between it and the newly appeared target position pos At the same time, the time interval ΔT between the newly appeared target and the last record of the target in the feature library is calculated.

[0121] Comprehensive matching score calculation: According to the formula Calculate the comprehensive matching score. j Determined by the distance of the timestamp. The closer to the current time, the j The larger the value, the range is [0.1,1]; γ1 and γ2 are determined by experiments and are set to 0.5 and 0.2 respectively.

[0122] Target ID determination: The target ID with the highest comprehensive matching score is selected as the ID of the reappearing target.

[0123] 4. Tracking parameter correction

[0124] 4.1 Kalman filter initialization

[0125] Initialization of state vector and covariance matrix: The position and velocity of the newly appeared target are used as the initial state vector, and a reasonable initial covariance matrix is set to represent the uncertainty of the initial state.

[0126] 4.2 Status Update

[0127] Prediction step: Based on the target's motion model, use the Kalman filter prediction formula to predict the target's state and covariance matrix at the next moment.

[0128] Update step: Take the newly acquired target position and velocity as observation values and use the Kalman filter update formula to calculate the target position and velocity.

[0129] Correct the target's state estimate and covariance matrix to obtain more accurate estimates of the target's position, velocity, and acceleration.

[0130] Provide a basis for subsequent tracking.

[0131] The above embodiments are exemplary and intended to illustrate the technical concepts and features of the present invention so that those skilled in the art can understand the present invention and implement it accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A surface target tracking system under incomplete information conditions, characterized in that: It includes multimodal feature extraction and fusion module, dynamic target feature library management module, target re-identification algorithm module based on spatiotemporal joint matching, and tracking parameter correction module; The multimodal feature extraction and fusion module is used to extract the visual features and motion features of the target and perform weighted fusion, including: A visual feature extraction unit is configured with a camera and an improved convolutional neural network model. The target image captured by the camera is normalized and then input into the improved convolutional neural network model. The improved convolutional neural network model calculates the channel dimension attention weight through the attention mechanism module to extract the visual feature vector; a motion feature extraction unit connected to a multi-sensor fusion system for collecting position, velocity, and acceleration data of a target in a geographic coordinate system to construct a motion feature vector; a feature fusion unit, configured to determine a weighted fusion weight according to an application scenario, perform weighted fusion on the visual feature vector and the motion feature vector, and generate a multimodal feature vector; The dynamic target feature library management module is used to store and update target feature information, including: A feature library storage unit is constructed with a hash table using the target ID as a key value, each key value corresponds to a linked list, and is used to store the multimodal feature vector and timestamp; The feature library update unit is used to add the newly extracted multimodal feature vectors and timestamps to the corresponding linked list when the target is visible, and delete the nodes with earlier timestamps in the linked list through the sliding window mechanism to maintain timeliness; The target re-identification algorithm module based on spatiotemporal joint matching is used to determine the target ID when the target reappears, including: A similarity calculation unit is used to extract the multimodal feature vector of the reappearing target and calculate its similarity with the feature vector in the linked list corresponding to each target ID in the feature library; The spatiotemporal matching unit is used to obtain the position information corresponding to the last feature vector of each target ID in the feature library, calculate the position distance and time interval between it and the newly appeared target, generate a comprehensive matching score based on the similarity, and determine the target ID. The tracking parameter correction module is used to correct the target tracking parameters, including: Kalman filter initialization unit, used to take the position and velocity of the newly appeared target as the initial state vector and set the initial covariance matrix; The state update unit is used to predict the state and covariance matrix of the target at the next moment based on the target motion model through the Kalman filter prediction formula, and uses the newly obtained target position and speed as observation values to correct the target state estimation value and covariance matrix through the update formula.

2. The surface target tracking system under incomplete information conditions according to claim 1 is characterized in that: The attention mechanism module obtains the channel description vector through global average pooling, generates the channel attention weight through the fully connected layer, and multiplies it with the original feature map to enhance the extraction of target key features.

3. The surface target tracking system under incomplete information conditions according to claim 1, characterized in that: The multi-sensor fusion system integrates sensors for collecting target position, velocity and acceleration data.

4. The surface target tracking system under incomplete information conditions according to claim 1, characterized in that: The sliding window mechanism deletes nodes with earlier timestamps that are out of the window range in the linked list according to the set window size. The window size is dynamically adjusted according to the target movement speed and the ocean environment.

5. The surface target tracking system under incomplete information conditions according to claim 1, characterized in that: The similarity calculation unit calculates the feature vector similarity by combining cosine similarity and Euclidean distance. The comprehensive matching score function of the spatiotemporal matching unit comprehensively considers the feature vector similarity, target position distance and time interval. The weight of each factor is determined by experiments or adaptive algorithms.

6. The surface target tracking system under incomplete information conditions according to claim 1, characterized in that: The Kalman filter algorithm of the state update unit includes a prediction step and an update step; the prediction step predicts the target state and covariance through the state transfer matrix and the process noise covariance matrix; the update step uses the residual between the observation value and the predicted value to correct the target state estimate and the covariance matrix.

7. The surface target tracking system under incomplete information conditions according to claim 1, characterized in that: The feature library storage unit adopts a composite data structure combining a hash table and a linked list, uses the target ID as the hash key value to achieve fast retrieval, and stores multimodal feature vectors and timestamps in chronological order through the linked list, supporting the query and dynamic update of target features.

8. The surface target tracking system under incomplete information conditions according to claim 1, characterized in that: The weighted fusion weight of the feature fusion unit is dynamically adjusted by an adaptive algorithm, and the adaptive algorithm adjusts the fusion ratio of visual features and motion features based on the target motion state, ocean environment parameters and sensor noise level.

9. The surface target tracking system under incomplete information conditions according to claim 1, characterized in that: When calculating the comprehensive matching score, the spatiotemporal matching unit assigns weights to the feature vectors in the linked list corresponding to the target ID in the feature library according to the distance of the timestamp. The feature vectors closer to the current time have higher weights, and the weight function is an exponential decay function or a linear decay function.

10. A method for tracking a surface target under incomplete information conditions, characterized in that The following steps are involved: (1) Multimodal feature extraction and fusion steps: The target image is captured by a camera, and the visual feature vector of the target is extracted using an improved convolutional neural network. At the same time, the position, velocity, and acceleration data of the target in the geographic coordinate system are obtained through a multi-sensor fusion system to construct a motion feature vector. The visual feature vector and the motion feature vector are weightedly fused to generate a multimodal feature vector. (2) Dynamic target feature library management steps: Using the target ID as the index, store the multimodal feature vectors and the corresponding timestamps in chronological order, and delete the feature vectors with earlier timestamps through the sliding window mechanism to maintain the timeliness of the feature library; (3) Spatiotemporal joint matching target re-identification step: When the target reappears, its multimodal feature vector is extracted and the similarity with the feature vectors of each target stored in the feature library is calculated; the target position distance, time interval and the similarity are combined to generate a comprehensive matching score to determine the target ID; (4) Tracking parameter correction step: Based on the Kalman filter algorithm, the position and velocity of the newly appeared target are used as observation values, and the target state parameters are predicted and corrected in combination with the target motion model to achieve continuous tracking.

Images