Dual-branch fusion target trajectory prediction method combined with motion compensation and related device

Abstract

The application discloses a kind of double-branch fusion target trajectory prediction methods and related devices combined with motion compensation, comprising: obtaining the continuous image frame sequence of target by infrared sensor, and continuously capturing target motion information in target tracking process, to obtain the historical trajectory of target;The continuous image frame sequence of the target is input into the trained motion perception trajectory prediction network with the historical trajectory, and the target trajectory is predicted, wherein the motion perception trajectory prediction network is constructed based on camera motion compensation module, double-branch prediction architecture and adaptive fusion weight network, and the method and related device can realize high-precision, high-robustness single-target trajectory prediction in infrared dynamic scene.

Description

Technical Field

[0001] This invention belongs to the field of infrared target tracking technology, and relates to a dual-branch fusion target trajectory prediction method and related device that combines motion compensation. Background Technology

[0002] Single-target tracking is one of the core tasks in computer vision. Its core requirement is to accurately locate the target and predict its subsequent trajectory in a continuous sequence of image frames, providing support for the continuity and stability of target tracking. In practical applications, camera motion is common (such as in infrared dynamic scenarios like drone tracking and vehicle-mounted camera monitoring), and the target's motion state is diverse. These two factors together pose a serious challenge to existing trajectory prediction methods, making it difficult to meet the high accuracy and robustness requirements of practical applications.

[0003] During tracking, predicting maneuvering targets (i.e., targets with abrupt changes in motion, drastic acceleration changes, or unpredictable trajectories) presents a significant challenge. In real-world tracking scenarios, targets frequently exhibit maneuvering behaviors such as sudden turns, accelerations, and decelerations. However, existing mainstream prediction methods (such as GRU, LSTM, and KF) are mostly based on fixed motion assumptions or general temporal feature extraction patterns, without being specifically optimized for the motion characteristics of maneuvering targets. This makes it difficult to quickly capture abrupt changes in the target's motion state, resulting in significantly higher prediction errors for maneuvering targets and prediction accuracy far from meeting the requirements of practical applications.

[0004] Existing trajectory prediction methods have significant shortcomings in addressing the aforementioned core issues, resulting in insufficient robustness and accuracy in complex scenarios. Specifically: Insufficient accuracy and poor adaptability in predicting maneuvering targets: Existing methods are usually based on the camera being stationary. In actual tracking scenarios where the camera is moving, problems such as discontinuity and jumps in the predicted trajectory are prone to occur. Furthermore, the temporal feature extraction and prediction model are not optimized for the sudden motion characteristics of maneuvering targets. The response to sudden turns, accelerations and other maneuvering behaviors of the target is lagging, and it is impossible to accurately capture the motion pattern of maneuvering targets, resulting in high prediction errors and failure to achieve stable and accurate tracking of maneuvering targets.

[0005] In summary, the core challenge currently facing the field of trajectory prediction is how to solve the problem of discontinuous predicted trajectories caused by camera motion, while improving the model's prediction accuracy for moving targets in infrared images, and achieving high-precision and robust single-target trajectory prediction in dynamic infrared scenes. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a dual-branch fusion target trajectory prediction method and related device that combines motion compensation. This method and related device can achieve high-precision and high-robust single-target trajectory prediction in infrared dynamic scenes.

[0007] To achieve the above objectives, this invention discloses a dual-branch fusion target trajectory prediction method incorporating motion compensation, comprising: The target's continuous image frame sequence is acquired by an infrared sensor, and the target's motion information is continuously captured during the target tracking process to obtain the target's historical trajectory; The continuous image frame sequence and historical trajectory of the target are input into the trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on a camera motion compensation module, a dual-branch prediction architecture and an adaptive fusion weight network.

[0008] Furthermore, the motion-aware trajectory prediction network includes a camera motion compensation module, a motion-aware module, a GRU encoder, a Kalman predictor head, a GRU decoder, an adaptive fusion weight network, and an innovative attention module; The camera motion compensation module establishes signal transmission connections with the motion sensing module, the GRU encoder, and the Kalman predictor head. The output of the camera motion compensation module outputs a stable trajectory, which is connected to the inputs of the motion sensing module, the GRU encoder, and the Kalman predictor head. The outputs of the GRU encoder and the motion sensing module are both connected to the input of the vector stitching module. The output of the vector stitching module is connected to the inputs of the Kalman predictor head and the adaptive fusion weight network. Simultaneously, the output of the GRU encoder is also connected to the input of the GRU decoder, which in turn is connected to the input of the adaptive fusion weight network. The output of the Kalman predictor head is connected to both the input of the adaptive fusion weight network and the input of the innovation attention module. The output of the adaptive fusion weight network is connected to the input of the innovation attention module.

[0009] Furthermore, the loss function of the motion-aware trajectory prediction network during training is constructed based on the main loss, component loss, weight regularization loss, classification loss, and time-series consistency loss.

[0010] Furthermore, the camera motion compensation method is characterized in that it receives two consecutive frames of infrared sequence images through an image input module, extracts depth features from the infrared images using a convolutional neural network, analyzes the extracted image features, outputs six-degree-of-freedom camera motion parameters, and then performs reverse compensation on the historical trajectory according to the camera motion, unifying the historical trajectory to the current frame coordinate system.

[0011] Furthermore, the GRU encoder and GRU decoder capture the temporal dependencies of historical trajectories to predict smooth, continuous normal motion.

[0012] Furthermore, the Kalman predictor incorporates the Kalman filtering concept, dynamically learning the Kalman gain through a neural network to respond to the target's maneuvering behavior.

[0013] This invention discloses a dual-branch fusion target trajectory prediction system incorporating motion compensation, comprising: The acquisition module is used to acquire a continuous sequence of image frames of the target through an infrared sensor, and continuously capture the target's motion information during the target tracking process to obtain the target's historical trajectory; The prediction module is used to input the continuous image frame sequence and historical trajectory of the target into the trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on the camera motion compensation module, the motion-aware trajectory prediction network and the adaptive fusion weight network.

[0014] The present invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dual-branch fusion target trajectory prediction method incorporating motion compensation.

[0015] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the dual-branch fusion target trajectory prediction method incorporating motion compensation.

[0016] The present invention has the following beneficial effects: The dual-branch fusion target trajectory prediction method and related device combining motion compensation described in this invention, in specific operation, inputs the continuous image frame sequence of the target and its historical trajectory into a trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on a camera motion compensation module, a dual-branch prediction architecture, and an adaptive fusion weight network. By employing these components, the method addresses the core pain points of existing trajectory prediction methods, such as discontinuous trajectories and insufficient accuracy in predicting moving targets in dynamic camera scenarios. It achieves high-precision and robust single-target trajectory prediction in infrared dynamic scenes, providing a reliable technical solution for single-target tracking tasks in fields such as video surveillance, drone tracking, and autonomous driving. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a diagram of the architecture of the motion-sensing trajectory prediction network in this invention; Figure 2 This is a flowchart of the training process of the present invention; Figure 3 This is a flowchart illustrating the reasoning process of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0023] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0024] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0026] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0027] Example 1 refer to Figure 1 and Figure 2 The dual-branch fusion target trajectory prediction method combining camera motion compensation described in this invention includes: 1) Obtain a continuous sequence of image frames of the target using an infrared sensor, and continuously capture the target's motion information during target tracking to obtain the target's historical trajectory; 2) Input the continuous image frame sequence of the target and its historical trajectory into the trained motion-aware trajectory prediction network to predict the target trajectory.

[0028] refer to Figure 1 The motion-aware trajectory prediction network includes a camera motion compensation module, a motion-aware module, a GRU encoder, a Kalman predictor head, a GRU decoder, an adaptive fusion weight network, and an innovative attention module. The camera motion compensation module establishes signal transmission connections with the motion sensing module, the GRU encoder, and the Kalman predictor head. The output of the camera motion compensation module is a stable trajectory, which is connected to the inputs of the motion sensing module, the GRU encoder, and the Kalman predictor head. The outputs of the GRU encoder and the motion sensing module are both connected to the input of the vector stitching module; the output of the vector stitching module is connected to the inputs of the Kalman predictor head and the adaptive fusion weight network. Simultaneously, the output of the GRU encoder is also connected to the input of the GRU decoder, and the output of the GRU decoder is connected to the input of the adaptive fusion weight network. The output of the Kalman predictor head is connected to the inputs of the adaptive fusion weight network and the innovation attention module; the output of the adaptive fusion weight network is connected to the input of the innovation attention module, completing the data flow interaction and hierarchical association between the various functional modules.

[0029] Camera motion compensation module: In practical applications, when the camera moves, the target's motion relative to the camera includes the camera's own motion component. This causes a drift in the statistical characteristics of historical trajectories, affecting prediction accuracy. The camera motion compensation module, based on the relative pose transformation of adjacent frames and image visual information, unifies historical trajectories to the current coordinate system. Specific implementation includes: Camera motion estimation: Visual features are extracted from infrared images through a convolutional neural network. A two-layer long short-term memory network is used to perform temporal modeling of the visual features of consecutive frames. A fully connected layer regresses the temporal features and outputs six-degree-of-freedom camera motion parameters [dx,dy,dz,roll,pitch,yaw].

[0030] Trajectory compensation: The historical trajectory is inversely compensated based on the camera movement to unify all historical trajectories into the current frame coordinate system.

[0031] The camera motion compensation module effectively distinguishes between the target's actual motion and the camera's motion through multimodal fusion, significantly improving the prediction accuracy in camera-moving scenarios and its adaptability to infrared images.

[0032] Motion perception module: Extracts rich motion features from historical trajectories. The input is 10 frames of compensated trajectory data in the format (10,4), which includes the target center coordinates [cx,cy] and width and height dimensions [w,h]. The data is normalized before input, mapping the coordinates and dimensions to the range [0,1].

[0033] Used to assist in motion type identification and prediction, the extracted features include 19-dimensional motion features, covering multiple dimensions such as velocity, acceleration, jerk, displacement change, velocity change rate, and directional consistency.

[0034] Adaptive Fusion Weight Network: Located after the two prediction heads, its core function is to dynamically generate fusion weights based on the motion type.

[0035] Network structure: Three-layer fully connected network + Sigmoid.

[0036] Input: A concatenation of the GRU last hidden state, motion type embedding, and motion type probability (146 dimensions). Output: A 2-dimensional weight vector, corresponding to the weights of the GRU prediction and the Kalman prediction, respectively.

[0037] The innovation attention module, located after the adaptive fusion weight network, is used only during inference. Based on the concept of "innovation" in Kalman filtering, this module dynamically adjusts the prediction confidence by analyzing the innovation sequence generated by KalmanNet during prediction.

[0038] Input: The innovation sequence, which is the difference between the state estimate and the predicted state generated by KalmanNet during the multi-frame prediction process (1,4). Output: 1-dimensional attention weights, ranging from 0 to 1, representing the confidence level of the prediction; The training process of the motion-aware trajectory prediction network is as follows: 11) Data loading; During training, the camera motion compensation module and the motion perception prediction network are trained separately. The training data for the camera motion compensation module is provided by the ViViD++ dataset, and a conventional loss function is used for learning. The ViViD++ dataset provides the input infrared image and the camera's precise position and pose (6-DOF pose) at each frame, forming a complete motion trajectory. The training data for the dual-branch prediction architecture and the adaptive fusion weight network are trajectory data loaded from the Anti-UAV410 dataset. The x, y, w, h format trajectories in the txt file are parsed, and training samples (10 historical frames + 1 future frame) are generated through a sliding window. Normalization processing (divided by the image size 640x512) is performed, and optional data augmentation (scale transformation, translation, Gaussian noise, trajectory mixing) can be applied.

[0039] 12) Forward propagation; Historical trajectory and image data are input into the model, which then passes through camera motion compensation, motion type determination, GRU encoding, KalmanNet prediction, GRU decoding, and an adaptive fusion weight network in sequence. The model outputs the predicted trajectory and various intermediate results (GRU prediction, KalmanNet prediction, fusion weights, motion type probability, etc.). 13) Calculate the loss; This invention employs a motion-aware loss function, which is the core of the entire training process and includes the following five loss components. It should be noted that the camera motion compensation module is trained separately, using mean squared error as the loss function. The following description only covers the loss functions used during the training of the dual-branch prediction architecture and the adaptive fusion weight network, and does not elaborate on the conventional loss functions used in the camera motion compensation module: 131) Main Loss; The main loss measures the error between the final predicted result and the true target, and is calculated using SmoothL1Loss. The position loss (prediction error of center coordinates x, y) and the size loss (prediction error of width and height w, h) are calculated separately, with the position loss weighted at 1.0 and the size loss weighted at 0.5, because position prediction is more important for tracking accuracy. These two losses are weighted and averaged using w_pos and w_size generated by a dynamic weighting network to obtain the main loss value.

[0040] 132) Component Loss; Component loss is used to measure the prediction errors of the GRU and KalmanNet prediction heads separately. First, the errors between the GRU and KalmanNet predictions and the ground truth are calculated separately. Then, these errors are weighted using two weights, w_gru and w_kalman, generated by a dynamic weight network. This loss allows the model to automatically adjust the contribution ratio of the two prediction heads to the total loss, achieving sample-level adaptive optimization.

[0041] 133) Weight Regularization Loss; To ensure the fused weights conform to physical laws, a weight regularization loss is introduced. A pre-defined prior distribution is used: during normal motion, the GRU weights account for 0.7 and the KalmanNet weights for 0.3; during abrupt changes in motion, the KalmanNet weights are appropriately increased. The mean squared error loss (MSE) is used to constrain the weights learned by the fused weight network to approximate the pre-defined prior distribution, making the learned weights both flexible and conforming to physical laws.

[0042] 134) Classification Loss; The classification loss is used to train the motion type judgment module. Entropy regularization is employed, calculating the entropy value of the motion type probability distribution and using it as part of the loss. Entropy regularization prevents the model from becoming overconfident in its motion type judgments (overfitting) and improves the model's generalization ability. The loss coefficient is set to 0.05 to balance its importance with other losses.

[0043] 135) Temporal Consistency Loss; The temporal consistency loss is an innovative component of this method, used to constrain the temporal smoothness of the predicted trajectory. It consists of two parts: first, a velocity change penalty (acceleration), which calculates the squared mean of velocity changes between adjacent frames to penalize abrupt changes in the trajectory; second, a scale change penalty, which calculates the squared mean of scale changes between adjacent frames to penalize abrupt changes in size. The two are weighted and summed to obtain the temporal consistency loss. This loss makes the predicted trajectory smoother and more natural, conforming to the laws of physical motion.

[0044] 136) Calculation of total loss; The total loss is a weighted sum of the five loss components mentioned above. The weights of each component are as follows: main loss weight 1.0, component loss weight 0.3, weighted regularization loss weight 0.15, classification loss weight 0.05, and temporal consistency loss weight 0.1. The total loss is then used for backpropagation.

[0045]

[0046] 14) Backpropagation The backpropagation process consists of three steps. First, `optimizer.zero_grad()` is called to clear the gradients calculated in the previous batch, ensuring the correctness of the gradient calculation in the current batch. Then, `losses['total'].backward()` is called to execute the backpropagation algorithm, calculating the gradients of all learnable parameters in the model with respect to the loss function based on the loss value. Finally, `torch.nn.utils.clip_grad_norm_()` is used to clip the calculated gradients, limiting the maximum norm of the gradients to within 1.0 to prevent gradient explosion and ensure the stability of the training process.

[0047] 15) Parameter update; Parameter updates are relatively simple. The `optimizer.step()` function updates the model parameters using gradients calculated via backpropagation, based on the optimizer's (e.g., Adam's) update rules. Hyperparameters such as the learning rate are pre-configured in the training script. This end-to-end training approach allows all modules (including the KalmanNet core and adaptive fusion weight network) to optimize collaboratively, achieving optimal overall performance.

[0048] 16) Output the results; Output the training loss value and the loss of each sub-item (total, main, component, weight_reg, cls, temporal), and calculate the time taken for each stage (data transmission, forward propagation, loss calculation, back propagation).

[0049] The entire training process consists of multiple training epochs. Each epoch begins with an iterative training phase, traversing all training data, executing the five steps mentioned above, and calculating the training loss. Then, an iterative validation phase occurs, traversing all validation data, performing forward propagation and loss calculation, but not backpropagation or parameter updates, to obtain the validation set loss and learning rate. The scheduler determines whether to adjust the learning rate based on the validation set loss. If the validation set loss is lower than the historical best value, the model is saved as the optimal model. The number of training epochs can be set to 50, or training can be stopped early when the validation set loss no longer decreases.

[0050] The optimizer used is Adam, with a learning rate of 0.001 and weight decay set to 1e-5. The batch size is set to 32. Dropout is set to 0.2. The learning rate scheduling uses the ReduceLROnPlateau strategy, with the validation set loss as the monitoring metric. The patience value is 5, meaning that if the validation set loss does not improve after 5 consecutive epochs, the learning rate is adjusted, with an adjustment factor of 0.5, i.e., the learning rate is halved. The maximum norm of gradient clipping is set to 1.0.

[0051] This invention has the following characteristics: To address the core issues of sudden changes in the motion of maneuvering targets, the lag in response of existing methods, and insufficient prediction accuracy, this invention designs a dual-branch fusion prediction network of GRU and KalmanNet to achieve unified modeling of normal motion and maneuvering motion, balancing the continuity of trajectory prediction with the prediction accuracy of maneuvering targets.

[0052] The dual-branch structure employs differentiated functional designs to form complementary advantages: the GRU prediction branch, based on the GRU encoder and decoder, excels at capturing the temporal dependencies of historical trajectories, enabling accurate prediction of smooth and continuous normal motion and ensuring the stability of trajectory prediction in conventional scenarios; the KalmanNet prediction branch incorporates the Kalman filtering concept, dynamically learning the Kalman gain through a neural network to replace the fixed gain of traditional Kalman filtering, enabling rapid response to maneuvering behaviors such as sudden turns, accelerations, and decelerations of targets, significantly improving the prediction accuracy of maneuvering targets.

[0053] To achieve optimal fusion of the two-branch prediction results, an adaptive fusion weight network is designed. Taking the last hidden state of the GRU encoder, motion type embedding, and motion type probability as input, the network outputs dynamic fusion weights for the two branches: increasing the weight of the GRU prediction branch during normal motion to ensure trajectory continuity; and increasing the weight of the KalmanNet prediction branch during maneuvering motion to enhance the response to sudden motion changes. Simultaneously, a novelty attention module is introduced. During the inference phase, the prediction results are dynamically adjusted by analyzing the novelty sequence generated by the KalmanNet, further optimizing the prediction performance for maneuvering targets and completely solving the problem of existing methods struggling to balance trajectory continuity and maneuvering target prediction accuracy.

[0054] Example 2 The dual-branch fusion target trajectory prediction system combining motion compensation described in this invention includes: The acquisition module is used to acquire a continuous sequence of image frames of the target through an infrared sensor, and continuously capture the target's motion information during the target tracking process to obtain the target's historical trajectory; The prediction module is used to input the continuous image frame sequence and historical trajectory of the target into the trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on a camera motion compensation module, a dual-branch prediction architecture, and an adaptive fusion weight network.

[0055] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in each embodiment of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0056] Example 3 A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the dual-branch fusion target trajectory prediction method incorporating motion compensation. For example, the method includes: acquiring a continuous sequence of image frames of a target using an infrared sensor, continuously capturing target motion information during target tracking to obtain the target's historical trajectory; inputting the continuous image frame sequence and the historical trajectory of the target into a trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on a camera motion compensation module, a dual-branch prediction architecture, and an adaptive fusion weight network. The memory may include main memory, such as high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device. The processor, network interface, and memory are interconnected via an internal bus, which can be an industry-standard architecture bus, a peripheral component interconnection standard bus, an extended industry-standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory stores the program; specifically, the program may include program code, which includes computer operation instructions. The memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0057] Example 4 A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the dual-branch fusion target trajectory prediction method incorporating motion compensation. For example, the method includes: acquiring a continuous sequence of image frames of a target using an infrared sensor, continuously capturing target motion information during target tracking to obtain the target's historical trajectory; inputting the continuous image frame sequence and the historical trajectory of the target into a trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on a camera motion compensation module, a dual-branch prediction architecture, and an adaptive fusion weight network. Specifically, the computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0058] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and disclosure of the invention. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0063] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0064] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A dual-branch fusion target trajectory prediction method combining motion compensation, characterized in that, include: The target's continuous image frame sequence is acquired by an infrared sensor, and the target's motion information is continuously captured during the target tracking process to obtain the target's historical trajectory; The continuous image frame sequence and historical trajectory of the target are input into the trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on a camera motion compensation module, a dual-branch prediction architecture and an adaptive fusion weight network.

2. The dual-branch fusion target trajectory prediction method combining motion compensation according to claim 1, characterized in that, The motion-aware trajectory prediction network includes a camera motion compensation module, a motion-aware module, a GRU encoder, a Kalman predictor head, a GRU decoder, an adaptive fusion weight network, and an innovative attention module; The camera motion compensation module establishes signal transmission connections with the motion sensing module, the GRU encoder, and the Kalman predictor head, respectively. The output of the camera motion compensation module outputs a stable trajectory, which is connected to the inputs of the motion sensing module, the GRU encoder, and the Kalman predictor head. The outputs of the GRU encoder and the motion sensing module are both connected to the input of the vector stitching module. The output of the vector stitching module is connected to the input of the Kalman predictor head and the input of the adaptive fusion weight network. Simultaneously, the output of the GRU encoder is also connected to the input of the GRU decoder, and the output of the GRU decoder is connected to the input of the adaptive fusion weight network. The output of the Kalman predictor head is connected to the input of the adaptive fusion weight network and the input of the innovation attention module. The output of the adaptive fusion weight network is connected to the input of the innovation attention module.

3. The dual-branch fusion target trajectory prediction method combining motion compensation according to claim 2, characterized in that, The loss function of the motion-aware trajectory prediction network during training is constructed based on the main loss, component loss, weight regularization loss, classification loss, and time-series consistency loss.

4. The frequency domain adaptive attention camera motion compensation method according to claim 2, characterized in that, The image input module receives two consecutive frames of infrared images, uses a convolutional neural network to extract depth features from the infrared images, analyzes the extracted image features, outputs six-degree-of-freedom camera motion parameters, and then performs reverse compensation on the historical trajectory according to the camera motion, unifying the historical trajectory to the current frame coordinate system.

5. The dual-branch fusion target trajectory prediction method combining motion compensation according to claim 2, characterized in that, The GRU encoder and GRU decoder capture the temporal dependencies of historical trajectories to predict smooth, continuous normal motion.

6. The dual-branch fusion target trajectory prediction method combining motion compensation according to claim 2, characterized in that, The Kalman predictor embeds the Kalman filter concept, dynamically learning the Kalman gain through a neural network to respond to the target's maneuvering behavior.

7. A dual-branch fusion target trajectory prediction system incorporating motion compensation, characterized in that, include: The acquisition module acquires a continuous sequence of image frames of the target through an infrared sensor and continuously captures the target's motion information during the target tracking process to obtain the target's historical trajectory; The prediction module is used to input the continuous image frame sequence and historical trajectory of the target into the trained motion-aware trajectory prediction network to predict the target trajectory. The motion-aware trajectory prediction network is constructed based on a camera motion compensation module, a dual-branch prediction architecture, and an adaptive fusion weight network.

8. The dual-branch fusion target trajectory prediction system with motion compensation according to claim 1, characterized in that, The motion-aware trajectory prediction network includes a camera motion compensation module, a motion-aware module, a GRU encoder, a Kalman predictor head, a GRU decoder, an adaptive fusion weight network, and an innovative attention module; The camera motion compensation module establishes signal transmission connections with the motion sensing module, the GRU encoder, and the Kalman predictor head, respectively. The output of the camera motion compensation module outputs a stable trajectory, which is connected to the inputs of the motion sensing module, the GRU encoder, and the Kalman predictor head. The outputs of the GRU encoder and the motion sensing module are both connected to the input of the vector stitching module. The output of the vector stitching module is connected to the input of the Kalman predictor head and the input of the adaptive fusion weight network. Simultaneously, the output of the GRU encoder is also connected to the input of the GRU decoder, and the output of the GRU decoder is connected to the input of the adaptive fusion weight network. The output of the Kalman predictor head is connected to the input of the adaptive fusion weight network and the input of the innovation attention module. The output of the adaptive fusion weight network is connected to the input of the innovation attention module.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the dual-branch fusion target trajectory prediction method with motion compensation as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the dual-branch fusion target trajectory prediction method with motion compensation as described in any one of claims 1-7.

Images