Ship trajectory prediction method, device and equipment based on scene semantic information

By using scene semantic information and attention calculation in ship trajectory prediction, combined with Transformer network and large language model, the multimodality and inhomogeneity problems of ship trajectory prediction in complex scenarios are solved, and the prediction accuracy and accuracy are improved.

CN118799830BActive Publication Date: 2024-11-26TSINGHUA UNIVERSITY
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411288440.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-26
Estimated Expiration
2044-09-14

AI Technical Summary

Technical Problem

The existing ship trajectory prediction methods cannot accurately predict future trajectories in complex scenarios, especially due to the multimodality and inhomogeneity problems caused by the wideness of the sea surface and the complexity of the external environment, which leads to a large difference between the prediction results and the real trajectory.

Method used

Using a method based on scene semantic information, the scene semantic features and historical ship trajectory are extracted, and the ship-scene and ship-ship attention calculation is carried out, and the future trajectory is predicted by combining the Transformer network and large language model. .

Benefits of technology

The ship trajectory prediction accuracy in complex scenarios is improved, and the impact of environment and ship interaction is taken into account, making the prediction results more reasonable and accurate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799830B_ABST
    Figure CN118799830B_ABST
Patent Text Reader

Abstract

The present application provides a ship trajectory prediction method, device and equipment based on scene semantic information, which relates to the field of ship trajectory prediction technology, by acquiring scene images and historical ship trajectories in scenes represented by scene images; extracting scene semantic features from scene images, and extracting historical ship trajectory features from historical ship trajectories; performing ship-scene attention calculation based on scene semantic features and historical ship trajectory features, and performing ship-ship attention calculation based on historical ship trajectory features; predicting the ship trajectory in the future based on the results of ship-scene attention calculation, the results of ship-ship attention calculation and scene semantic features. By combining scene semantic information, self-attention mechanism network and large language model, the environmental position information of the ship is added to the network calculation, and the impact of the environment on the ship is considered during prediction, so that the predicted trajectory is more reasonable and accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of ship trajectory prediction, and in particular to a method, device and equipment for ship trajectory prediction based on scene semantic information. Background Art

[0002] With the rapid development of artificial intelligence, cloud computing, the Internet of Things and other fields, the shipping industry is undergoing a new industrial revolution. Due to the continuous growth of the shipping industry, the development of maritime activities has attracted widespread attention, and the prediction of the general direction and position of ships over medium and long distances has become the focus of attention, including but not limited to search and rescue, path planning, port congestion avoidance, etc.

[0003] Due to the vastness of the sea and the complexity of the external environment (wind speed and weather conditions, etc.), the future trajectory of the ship shows multimodality, that is, there are many possible routes for the movement of the ship in the future, which is not considered in the current ship trajectory prediction method. And because the ship data shows significant non-uniformity in different geographical regions and environmental conditions, this difference will have a negative impact on the trajectory prediction performance, resulting in a very large difference between the final generated trajectory and the actual trajectory.

[0004] Therefore, how to more accurately predict the future trajectory of a ship in complex scenarios remains a difficult problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a ship trajectory prediction method, device and equipment based on scene semantic information to solve the problem that related trajectory prediction methods cannot accurately predict future trajectories in complex scenarios.

[0006] In a first aspect of an embodiment of the present application, a ship trajectory prediction method based on scene semantic information is provided, the method comprising:

[0007] Acquire a scene image and a historical ship track in a scene represented by the scene image;

[0008] extracting scene semantic features from the scene image, and extracting historical ship trajectory features from the historical ship trajectory;

[0009] performing ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features, and performing ship-ship attention calculation according to the historical ship trajectory features;

[0010] The ship trajectory in the future is predicted based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the semantic features of the scene.

[0011] In an optional embodiment of the present application, extracting historical ship trajectory features from the historical ship trajectory is achieved by a first fully connected module and an encoder, the encoder includes a first large language model and a second large language model, and the input ends of the first large language model and the second large language model are respectively connected to the output end of the first fully connected module;

[0012] The step of extracting historical ship trajectory features from the historical ship trajectory comprises:

[0013] Encoding the two-dimensional coordinate points represented by the historical ship trajectory through the first fully connected module and the first large language model to obtain a first historical ship trajectory feature, wherein the first historical ship trajectory feature is used to perform ship-scene attention calculation with the scene semantic feature;

[0014] The two-dimensional coordinate points representing the historical ship trajectory are encoded through the first fully connected module and the second large language model to obtain the second historical ship trajectory features, and the first historical ship trajectory features and the second historical ship trajectory features are used for ship-ship attention calculation.

[0015] In an optional embodiment of the present application, the prediction of the ship trajectory in the future period of time according to the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic features is achieved by sequentially connecting a splicing module, a second fully connected module and a decoder, and the decoder is a long short-term memory network;

[0016] The method of predicting the ship trajectory in the future based on the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic features includes:

[0017] The splicing module splices the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic feature to obtain a splicing result;

[0018] The splicing result is processed by the second fully connected module, and the processing result is input into the long short-term memory network to obtain the ship trajectory in the future period of time.

[0019] In an optional embodiment of the present application, the method is to process the scene image and the historical ship trajectory through a pre-trained ship trajectory prediction model, thereby predicting the ship trajectory within the future period of time;

[0020] The ship trajectory prediction model includes: a first fully connected module, an encoder, an attention module, a splicing module, a second fully connected module and a decoder, and a feature extractor connected in sequence. The ship trajectory prediction model uses minimizing the mean square error between future predicted trajectory points and historical true trajectory points as a loss function.

[0021] In an optional embodiment of the present application, the ship-scene attention calculation based on the scene semantic features and the historical ship trajectory features is implemented through a Transformer network structure, and the Transformer network structure includes: a self-attention layer, an FFN layer and a cross-attention layer; the cross-attention layer is between the self-attention layer and the FFN layer;

[0022] The ship-scene attention calculation is performed according to the scene semantic features and the historical ship trajectory features, including:

[0023] The historical ship trajectory features are input into the self-attention layer for processing, the processing results output by the self-attention layer and the scene semantic features are input into the cross-attention layer for processing, and the processing results output by the cross-attention layer are input into the FFN layer for processing to obtain the result of the ship-scene attention calculation.

[0024] In an optional embodiment of the present application, the method also includes: using the final displacement error and the average displacement error as evaluation indicators to evaluate the quality of the predicted trajectory of the ship trajectory prediction model; wherein the final displacement error is used to characterize the prediction accuracy of the ship trajectory in the future period of time, and the average displacement error is used to characterize the average predictability of the ship trajectory prediction model along the historical ship trajectory.

[0025] In an optional embodiment of the present application, the acquiring of the scene image includes:

[0026] Obtain ship surveillance video;

[0027] Extracting any frame image from the ship monitoring video as the scene image;

[0028] The training samples of the ship trajectory prediction model are obtained according to the following steps:

[0029] Obtain surveillance videos containing ships collected in different scenarios;

[0030] Using any frame image in the surveillance video containing the ship as a training scene image;

[0031] The ship trajectories at the first time under different training scene images are used as the training samples.

[0032] In a second aspect of an embodiment of the present application, a ship trajectory prediction device based on scene semantic information is provided, the device comprising:

[0033] An acquisition module, used for acquiring a scene image and a historical ship track in a scene represented by the scene image;

[0034] A historical ship track feature extraction module, used to extract scene semantic features from the scene image, and to extract historical ship track features from the historical ship track;

[0035] a calculation module, configured to perform ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features, and perform ship-ship attention calculation according to the historical ship trajectory features;

[0036] The ship trajectory prediction module is used to predict the ship trajectory in the future based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the scene semantic features.

[0037] In a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, a ship trajectory prediction method based on scene semantic information as described in any one of the first aspects above is implemented.

[0038] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the ship trajectory prediction method based on scene semantic information as described in any one of the first aspects above is implemented.

[0039] The present application includes the following advantages: The present application proposes a ship trajectory prediction method, device and equipment based on scene semantic information, by acquiring a scene image and a historical ship trajectory in a scene represented by the scene image; extracting scene semantic features from the scene image, and extracting historical ship trajectory features from the historical ship trajectory; performing ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features, and performing ship-ship attention calculation according to the historical ship trajectory features; predicting the ship trajectory in the future based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the scene semantic features. By combining scene semantic information, a self-attention mechanism network and a large language model, the environmental position information of the ship is added to the network calculation, and the impact of the environment on the ship is considered during the prediction, so that the predicted trajectory is more reasonable and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 It is a schematic diagram of the steps of a method for predicting ship trajectory based on scene semantic information provided by an embodiment of the present application;

[0042] Figure 2 It is a flow chart of a method for predicting ship trajectory based on scene semantic information provided by an embodiment of the present application;

[0043] Figure 3 This is an architecture diagram of a ship trajectory prediction device based on scene semantic information provided by an embodiment of the present application;

[0044] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] With the rapid development of artificial intelligence, cloud computing, the Internet of Things and other fields, the shipping industry is undergoing a new industrial revolution. Due to the continuous growth of the shipping industry, the development of maritime activities has attracted widespread attention, and the prediction of the general direction and position of ships over medium and long distances has become the focus of attention, including but not limited to search and rescue, path planning, port congestion avoidance, etc.

[0047] Due to the vastness of the sea and the complexity of the external environment (wind speed and weather conditions, etc.), the future trajectory of the ship shows multimodality, that is, there are many possible routes for the movement of the ship in the future, which is not considered in the current ship trajectory prediction method. And because the ship data shows significant non-uniformity in different geographical regions and environmental conditions, this difference will have a negative impact on the trajectory prediction performance, resulting in a very large difference between the final generated trajectory and the actual trajectory.

[0048] Traditional trajectory prediction methods mainly use trajectory prediction methods based on kinematic models. Trajectory prediction methods based on kinematic models focus on modeling the motion state of the ship. Commonly used models include Gaussian Mixture Model (GMM) and Kalman Filtering Model (KFM). These methods compare ships to Newtonian entities, and the movement of ships is restricted by physical laws. However, this type of method is only suitable for trajectory prediction of a single ship in simple scenarios. With the continuous development of industry, the prediction of the future trajectory of ships is no longer satisfied with some simple specific scenarios. The prediction of ship trajectories in complex scenarios has become the mainstream. Therefore, how to make more accurate predictions of the future trajectory of ships remains a difficult problem.

[0049] Semantic information such as the ship's intentions, behavior patterns, and interactions with the environment can provide a deeper understanding and prediction capability, thereby enhancing the accuracy and robustness of the prediction model. Therefore, the ship trajectory prediction method based on semantic information has important practical significance for optimizing maritime traffic management and enhancing public safety.

[0050] Based on this, this application proposes a ship trajectory prediction method based on scene semantic information and deep learning to achieve high trajectory prediction accuracy in complex scenarios.

[0051] In a first aspect of an embodiment of the present application, a ship trajectory prediction method based on scene semantic information is provided. Figure 1 , Figure 1 1 is a schematic diagram of the steps of a method for predicting ship trajectory based on scene semantic information provided by an embodiment of the present application, the method comprising the following steps:

[0052] Step 101: Acquire a scene image and a historical ship track in the scene represented by the scene image;

[0053] Step 102: extracting scene semantic features from the scene image, and extracting historical ship track features from the historical ship track;

[0054] Step 103: performing ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features, and performing ship-ship attention calculation according to the historical ship trajectory features;

[0055] Step 104: predicting the trajectory of the ship in the future based on the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic features.

[0056] In order to enable those skilled in the art to better understand the ship trajectory prediction method based on scene semantic information provided by the present application, Figure 2 A schematic diagram of a process flow of a ship trajectory prediction method based on scene semantic information provided by an embodiment of the present application is shown. Figure 2 , the scheme is elaborated in detail:

[0057] When performing the specific implementation of step 101, a large amount of ship trajectory sample data is required when performing trajectory prediction. The data can be captured by a camera from a bird's-eye view, or public ship trajectory data on the Internet can be used. Specifically, the scene image can be obtained by first acquiring a ship monitoring video, which can be captured by a camera, a drone, or other equipment, and then extracting any frame image from the acquired ship monitoring video as a scene image. Then, the historical ship trajectory in the scene represented by the scene image is obtained. For example, the ship trajectory in the ship monitoring video can be collected by visual detection technology, a two-dimensional laser sensor, or the like, to obtain the historical ship trajectory.

[0058] When step 102 is specifically implemented, scene semantic features and historical ship trajectory features are obtained based on the acquired scene image and historical ship trajectory. The scene semantic features are obtained by inputting the scene image into a pre-trained deep learning network (this part corresponds to Figure 2 The feature extractor shown in the figure, CNN is a convolutional neural network), and is extracted based on prior object information. The deep learning network can be ResNet50 (a deep convolutional neural network composed of residual units).

[0059] The above historical ship trajectory features are obtained by the following method: extracting the historical ship trajectory features from the historical ship trajectory is achieved by a first fully connected module and an encoder, the encoder includes a first large language model and a second large language model, and the input ends of the first large language model and the second large language model are respectively connected to the output end of the first fully connected module. The above first fully connected module corresponds to Figure 2 The MLP module between the historical ship trajectory and the encoder in the above-mentioned first language model and the second language model can be a pre-trained large language model (Large Language Model, LLM), such as the BERT (Bidirectional Encoder Representations from Transformers) model and the GPT (Generative Pre-trained Transformer) model. This application introduces a large language model into the field of ship trajectory prediction, and realizes more accurate ship trajectory prediction through the powerful representation ability of the large language model.

[0060] Specifically, the step of extracting historical ship trajectory features from historical ship trajectories includes: first, encoding the two-dimensional coordinate points represented by the historical ship trajectory through the first fully connected module and the first large language model to obtain the first historical ship trajectory features, and mapping the two-dimensional coordinate points to the high-dimensional space through the first fully connected module and the first large language model to fully extract the historical trajectory information, and the first historical ship trajectory features are used to perform ship-scene attention calculation with the scene semantic features. The calculation process of encoding the two-dimensional coordinate points represented by the historical ship trajectory through the first fully connected module and the first large language model is as follows:

[0061] ,

[0062] in, I predict represents the historical ship trajectory characteristics, MLP represents the first fully connected module, LLM represents the first large language model, x history and y history Two-dimensional coordinate points representing historical ship trajectories.

[0063] The deep learning method based on the large language model is used to encode the historical ship trajectories, which can map the two-dimensional coordinates into the high-dimensional space, making the original information richer. Compared with related technologies, it can better extract the potential relationships in the time series. By utilizing the powerful reasoning ability and generalization performance of the large language model, the accuracy of the model's ship trajectory prediction is enhanced.

[0064] Then, the two-dimensional coordinate points representing the historical ship trajectory are encoded through the first fully connected module and the second largest language model to obtain the second historical ship trajectory features. The first historical ship trajectory features and the second historical ship trajectory features are used for ship-ship attention calculation.

[0065] When implementing step 103, considering that the motion trajectory of a ship in a complex scene will be affected by the surrounding ships and the surrounding environment, the interaction between ships and between ships and the environment is modeled, and the ship-scene attention and ship-ship attention are calculated. Figure 2 , the ship-scene attention is calculated by the PE Attention module, and the ship-ship attention is calculated by the PP Attention module. In the embodiment of the present application, the ship-scene attention is calculated according to the above-mentioned scene semantic features and the above-mentioned historical ship trajectory features, and the ship-ship attention is calculated according to the above-mentioned historical ship trajectory features.

[0066] Specifically, the ship-scene attention calculation based on the scene semantic features and the historical ship trajectory features is realized through a Transformer network structure, and the Transformer network structure includes: a self-attention layer, an FFN layer (feed-forward layer) and a cross-attention layer; the cross-attention layer is between the self-attention layer and the FFN layer. The ship-scene attention calculation based on the scene semantic features and the historical ship trajectory features through the Transformer network structure includes: firstly inputting the historical ship trajectory features into the self-attention layer for processing, inputting the processing results output by the self-attention layer and the scene semantic features into the cross-attention layer for processing, and inputting the processing results output by the cross-attention layer into the FFN layer for processing to obtain the result of the ship-scene attention calculation.

[0067] In an optional embodiment of the present application, an attention mechanism is used to calculate the interaction effects between ships. Specifically, for the historical trajectory features of ships in each batch, three different fully connected modules are used to extract the query vector Q, the key vector K and the value vector V respectively, and the model is modeled by calculating the inner integral fraction. The specific calculation formula is as follows:

[0068] ,

[0069] Among them, n represents the dimension of the feature vector, L represents the intermediate hidden vector, and T represents the transposition operation.

[0070] When step 104 is specifically implemented, after obtaining the results of the ship-scene attention calculation and the results of the ship-ship attention calculation, a splicing module is used to splice the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the scene semantic features to obtain splicing features. Based on the splicing features, the coordinates of the ship trajectory in the future are predicted as the final predicted ship trajectory.

[0071] Specifically, the above-mentioned prediction of the ship's trajectory in the future based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the scene semantic features is achieved by sequentially connected splicing modules, second fully connected modules and decoders, and the above-mentioned decoder is a long short-term memory network. The above-mentioned prediction of the ship's trajectory in the future based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the scene semantic features includes: firstly, splicing the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the scene semantic features through the above-mentioned splicing module to obtain a splicing result; then, processing the above-mentioned splicing result through the above-mentioned second fully connected module, and inputting the processing result into the above-mentioned long short-term memory network to obtain the ship's trajectory in the future. The above-mentioned second fully connected module corresponds to Figure 2 The MLP module between the splicing module and the decoder. The above long short-term memory network corresponds to the LSTM network in the decoder in the figure.

[0072] In an optional embodiment of the present application, the ship trajectory prediction method based on scene semantic information is to process the scene image and the historical ship trajectory through a pre-trained ship trajectory prediction model, and then predict the ship trajectory in the future period of time. The ship trajectory prediction model includes: a first fully connected module, an encoder, an attention module, a splicing module, a second fully connected module and a decoder, and a feature extractor connected in sequence. The ship trajectory prediction model takes minimizing the mean square error between the future predicted trajectory point and the historical real trajectory point as the loss function, and the loss function The calculation formula is as follows:

[0073] ,

[0074] in, represents the predicted trajectory point, represents the real trajectory point, i The value range is 1- n , j The value range is 1- m .

[0075] By using the above mean square error as the loss function, the encoder and decoder in the ship trajectory prediction model can be jointly optimized at the same time, which can improve the prediction accuracy of the ship trajectory prediction model.

[0076] In an optional embodiment of the present application, the training samples of the ship trajectory prediction model are obtained according to the following steps: first, surveillance videos containing ships collected in different scenes are obtained; then, any frame image in the surveillance video containing ships is used as a training scene image; and the ship trajectory at the first time under different training scene images is used as the training sample. By inputting the training sample into the ship trajectory prediction model to be trained, the historical ship trajectory at the second time adjacent to the first time is obtained. The loss value between the historical ship trajectory at the second time and the actual historical ship trajectory at the second time is calculated, and the ship trajectory prediction model to be trained is optimized by the loss value to obtain the ship trajectory prediction model.

[0077] In the process of training the ship trajectory prediction model, the first and second language models used are pre-trained through massive text data. The pre-trained large language models have learned rich semantic information and contextual features, which are helpful for understanding the historical trajectory distribution and predicting the future spatial distribution state. If the above large language model is selected as the BERT model, the bidirectional training structure of the BERT model can consider the previous and subsequent information at the same time, providing the model with comprehensive contextual understanding capabilities, which helps the model understand the historical trajectory distribution and predict the future spatial distribution state accordingly. If the above large language model is selected as the GPT model, due to the powerful generation ability of GPT, it is very suitable for time series prediction. It can generate continuous future trajectories based on the existing historical trajectory information.

[0078] In order to better transfer the pre-trained knowledge of the large model to the ship trajectory prediction task without increasing the amount of additional computation, the first large language model and the second large language model are fine-tuned during the training of the ship trajectory prediction model, so that the ship trajectory prediction model can learn highly robust and universal transferable representations. That is, during the training of the ship trajectory prediction model, all parameters of the first large language model and the second large language model are frozen, and only the parameters in the fine-tuning method are updated. The above-mentioned fine-tuning methods include bias fine-tuning, LoRA fine-tuning (Low-Rank Adaptation), full fine-tuning, and freezing all parameters of the large language model. Taking LoRA fine-tuning as an example, LoRA adjusts the weight of the large language model by adding a trainable low-rank matrix to achieve parameter updates of the large language model. This method can significantly reduce the number of parameters that need to be adjusted while maintaining the performance of the large language model.

[0079] In an optional embodiment of the present application, the method further includes: using the final displacement error and the average displacement error as evaluation indicators to evaluate the quality of the predicted trajectory of the ship trajectory prediction model; wherein the final displacement error (Final Displacement Error, FDE) is used to characterize the prediction accuracy of the ship trajectory in the future period of time, and the average displacement error (Average Displacement Error, ADE) is used to characterize the average predictability of the ship trajectory prediction model along the historical ship trajectory. The smaller the values ​​of the final displacement error and the average displacement error, the better the performance of the ship trajectory prediction model.

[0080] Exemplarily, the public ship trajectory datasets E and U are used as processing objects to perform corresponding ship trajectory prediction. The data in the above datasets come from real sea surface monitoring videos, recording the trajectory of ships and various bird's-eye view ship interaction scenes. The datasets E and U contain five sequences (E: E1 and HL, U: U1, Z1 and Z2), recording four different scenes. The E1 dataset contains about 8 minutes of video with about 50 different ships. The H dataset contains videos with a duration of about 13 minutes and about 160 different ships. The Z dataset records ships passing through a specific sea surface point, with a total video duration of 13 minutes and about 200 different ships identified. These scenes are captured in an unconstrained environment. Therefore, the impact of scene restrictions from other physical non-living entities is minimal. Finally, the quality of the predicted trajectory is measured by the two indicators of final displacement error and average displacement error, and the prediction results are shown in Tables 1 and 2.

[0081] Table 1 Results of evaluating different trajectory prediction algorithms using ADE

[0082]

[0083] Table 2 Results of evaluating different trajectory prediction algorithms using FDE

[0084]

[0085] As can be seen from Tables 1 and 2, the trajectory prediction method proposed in this application has achieved better results on the two datasets E and U. Specifically, in the more crowded datasets U and E, the social force model (SF model) and the ship trajectory prediction model proposed in this application both consider the environmental influences such as ships or scenes and perform better than a single LSTM model, indicating that it is meaningful for the method proposed in this application to consider the interaction between ships and scenes in future trajectory prediction. By comparing FDE, it can be concluded that the use of attention mechanism for prediction can better capture this kind of social interaction-like characteristics. Under the HL and Z datasets, by comparing ADE, it can be seen that the LSTM model can better capture time series features.

[0086] The embodiment of the present application proposes a ship trajectory prediction method based on scene semantic information, by acquiring a scene image and a historical ship trajectory in a scene represented by the scene image; extracting scene semantic features from the scene image, and extracting historical ship trajectory features from the historical ship trajectory; performing ship-scene attention calculation based on the scene semantic features and the historical ship trajectory features, and performing ship-ship attention calculation based on the historical ship trajectory features; predicting the ship trajectory in the future based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation, and the scene semantic features. By combining scene semantic information, a self-attention mechanism network, and a large language model, the environmental location information of the ship is added to the network calculation, and the impact of the environment on the ship is considered during the prediction, so that the predicted trajectory is more reasonable and accurate.

[0087] In a second aspect of the present application, a ship trajectory prediction device based on scene semantic information is proposed, referring to Figure 3 , Figure 3 : is an architecture diagram of a ship trajectory prediction device based on scene semantic information provided by an embodiment of the present application, the device comprising:

[0088] An acquisition module 301 is used to acquire a scene image and a historical ship track in a scene represented by the scene image;

[0089] A historical ship track feature extraction module 302, used to extract scene semantic features from the scene image, and to extract historical ship track features from the historical ship track;

[0090] A calculation module 303, configured to perform ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features, and perform ship-ship attention calculation according to the historical ship trajectory features;

[0091] The ship trajectory prediction module 304 is used to predict the ship trajectory in the future period of time according to the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic features.

[0092] The historical ship trajectory feature extraction module is implemented by a first fully connected module and an encoder, the encoder includes a first large language model and a second large language model, and the input ends of the first large language model and the second large language model are respectively connected to the output end of the first fully connected module;

[0093] The historical ship trajectory feature extraction module includes:

[0094] A first encoding submodule, configured to encode the two-dimensional coordinate points represented by the historical ship trajectory through the first fully connected module and the first large language model to obtain a first historical ship trajectory feature, wherein the first historical ship trajectory feature is used to perform ship-scene attention calculation with the scene semantic feature;

[0095] The second encoding submodule is used to encode the two-dimensional coordinate points represented by the historical ship trajectory through the first fully connected module and the second large language model to obtain the second historical ship trajectory features, and the first historical ship trajectory features and the second historical ship trajectory features are used for ship-ship attention calculation.

[0096] The ship trajectory prediction module is implemented by sequentially connecting a splicing module, a second fully connected module and a decoder, and the decoder is a long short-term memory network;

[0097] The ship trajectory prediction module comprises:

[0098] A splicing submodule, used for splicing the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic feature through the splicing module to obtain a splicing result;

[0099] The prediction submodule is used to process the splicing result through the second fully connected module, input the processing result into the long short-term memory network, and obtain the ship trajectory in the future period of time.

[0100] The device further includes a prediction module, which processes the scene image and the historical ship trajectory through a pre-trained ship trajectory prediction model to predict the ship trajectory within the future period of time;

[0101] The ship trajectory prediction model includes: a first fully connected module, an encoder, an attention module, a splicing module, a second fully connected module and a decoder, and a feature extractor connected in sequence. The ship trajectory prediction model uses minimizing the mean square error between future predicted trajectory points and historical true trajectory points as a loss function.

[0102] The computing module is implemented by a Transformer network structure, which includes a self-attention layer, an FFN layer, and a cross-attention layer; the cross-attention layer is between the self-attention layer and the FFN layer;

[0103] The calculation module includes a processing submodule, which is used to input the historical ship trajectory features into the self-attention layer for processing, input the processing results output by the self-attention layer and the scene semantic features into the cross-attention layer for processing, and input the processing results output by the cross-attention layer into the FFN layer for processing to obtain the result of the ship-scene attention calculation.

[0104] The device also includes an evaluation module, which is used to use the final displacement error and the average displacement error as evaluation indicators to evaluate the quality of the predicted trajectory of the ship trajectory prediction model; wherein the final displacement error is used to characterize the prediction accuracy of the ship trajectory in the future period of time, and the average displacement error is used to characterize the average predictability of the ship trajectory prediction model along the historical ship trajectory.

[0105] Wherein, the acquisition module includes:

[0106] The first acquisition submodule is used to acquire the ship monitoring video;

[0107] A scene image determination submodule, used for extracting any frame image from the ship monitoring video as the scene image;

[0108] Wherein, the prediction module also includes:

[0109] The second acquisition submodule is used to acquire surveillance videos containing ships collected in different scenes;

[0110] A training scene image determination submodule, used to use any frame image in the surveillance video containing the ship as a training scene image;

[0111] The training sample determination submodule is used to use the ship trajectory at the first time under different training scene images as the training sample.

[0112] Based on the same application concept, the embodiment of the present application discloses an electronic device, Figure 4 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 4 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement a ship trajectory prediction method based on scene semantic information disclosed in an embodiment of the present application.

[0113] Based on the same application concept, an embodiment of the present application discloses a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, a ship trajectory prediction method based on scene semantic information disclosed in an embodiment of the present application is implemented.

[0114] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0115] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0118] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0119] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0120] The above is a detailed introduction to the ship trajectory prediction method, device and equipment based on scene semantic information provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A ship trajectory prediction method based on scene semantic information, characterized in that: The method comprises: Acquire a scene image and a historical ship track in a scene represented by the scene image; extracting scene semantic features from the scene image, and extracting historical ship trajectory features from the historical ship trajectory; performing ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features, and performing ship-ship attention calculation according to the historical ship trajectory features; The ship trajectory in the future is predicted based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the semantic features of the scene.

2. The ship trajectory prediction method based on scene semantic information according to claim 1 is characterized in that: Extracting historical ship trajectory features from the historical ship trajectory is achieved by a first fully connected module and an encoder, the encoder comprising a first large language model and a second large language model, and input ends of the first large language model and the second large language model are respectively connected to output ends of the first fully connected module; The step of extracting historical ship trajectory features from the historical ship trajectory comprises: Encoding the two-dimensional coordinate points represented by the historical ship trajectory through the first fully connected module and the first large language model to obtain a first historical ship trajectory feature, wherein the first historical ship trajectory feature is used to perform ship-scene attention calculation with the scene semantic feature; The two-dimensional coordinate points representing the historical ship trajectory are encoded through the first fully connected module and the second large language model to obtain the second historical ship trajectory features, and the first historical ship trajectory features and the second historical ship trajectory features are used for ship-ship attention calculation.

3. The ship trajectory prediction method based on scene semantic information according to claim 1, characterized in that: The prediction of the ship trajectory in the future period of time according to the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the semantic features of the scene is achieved by sequentially connecting a splicing module, a second fully connected module and a decoder, wherein the decoder is a long short-term memory network; The predicting of the ship trajectory in the future period of time according to the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic features includes: The splicing module splices the result of the ship-scene attention calculation, the result of the ship-ship attention calculation and the scene semantic feature to obtain a splicing result; The splicing result is processed by the second fully connected module, and the processing result is input into the long short-term memory network to obtain the ship trajectory in the future period of time.

4. The ship trajectory prediction method based on scene semantic information according to claim 1, characterized in that: The method is to process the scene image and the historical ship trajectory through a pre-trained ship trajectory prediction model, and then predict the ship trajectory within the future period of time; The ship trajectory prediction model includes: a first fully connected module, an encoder, an attention module, a splicing module, a second fully connected module and a decoder, and a feature extractor connected in sequence. The ship trajectory prediction model uses minimizing the mean square error between future predicted trajectory points and historical true trajectory points as a loss function.

5. The ship trajectory prediction method based on scene semantic information according to claim 1, characterized in that: The ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features is implemented by a Transformer network structure, and the Transformer network structure includes: a self-attention layer, an FFN layer and a cross-attention layer; the cross-attention layer is between the self-attention layer and the FFN layer; The ship-scene attention calculation is performed according to the scene semantic features and the historical ship trajectory features, including: The historical ship trajectory features are input into the self-attention layer for processing, the processing results output by the self-attention layer and the scene semantic features are input into the cross-attention layer for processing, and the processing results output by the cross-attention layer are input into the FFN layer for processing to obtain the result of the ship-scene attention calculation.

6. The ship trajectory prediction method based on scene semantic information according to claim 1, characterized in that: The method also includes: using a final displacement error and an average displacement error as evaluation indicators to evaluate the quality of the predicted trajectory of the ship trajectory prediction model; wherein the final displacement error is used to characterize the prediction accuracy of the ship trajectory in the future period of time, and the average displacement error is used to characterize the average predictability of the ship trajectory prediction model along the historical ship trajectory.

7. The ship trajectory prediction method based on scene semantic information according to claim 4 is characterized in that: The acquiring of the scene image comprises: Obtain ship surveillance video; Extracting any frame image from the ship monitoring video as the scene image; The training samples of the ship trajectory prediction model are obtained according to the following steps: Obtain surveillance videos containing ships collected in different scenarios; Using any frame image in the surveillance video containing the ship as a training scene image; The ship trajectories at the first time under different training scene images are used as the training samples.

8. A ship trajectory prediction device based on scene semantic information, characterized in that: The device comprises: An acquisition module, used for acquiring a scene image and a historical ship track in a scene represented by the scene image; A historical ship track feature extraction module, used to extract scene semantic features from the scene image, and to extract historical ship track features from the historical ship track; a calculation module, configured to perform ship-scene attention calculation according to the scene semantic features and the historical ship trajectory features, and perform ship-ship attention calculation according to the historical ship trajectory features; The ship trajectory prediction module is used to predict the ship trajectory in the future based on the results of the ship-scene attention calculation, the results of the ship-ship attention calculation and the scene semantic features.

9. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the ship trajectory prediction method based on scene semantic information as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the ship trajectory prediction method based on scene semantic information as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Trajectory prediction method and device

    CN116740664A

  • Multi-source information fused ship trajectory prediction method

    CN117391244A

  • Ship trajectory prediction method and system

    CN117688366A

  • Intelligent ship collision avoidance decision-making method based on pulse attention in complex scene

    CN118334623A

  • Ship image trajectory tracking and prediction method based on ship heading recognition

    US20240013402A1