A method, device, equipment and medium for predicting ship trajectory driven by end point
Through the end-point-driven method, combined with semantic feature encoder and trajectory generation network, the multimodality and inhomogeneity problems of ship trajectory prediction in complex sea surface environments are solved, and more accurate and reliable trajectory prediction is achieved, suitable for a variety of Application scenario.
Patent Information
- Application Number
- CN202411288439.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-14
AI Technical Summary
Existing ship trajectory prediction methods are difficult to accurately predict future trajectories in complex sea surface environments, and due to the inhomogeneity of geographical regions and environmental conditions, the effectiveness of the model in a wider range of applications is limited.
Using the end-point-driven method, by obtaining the scene image to be predicted and the historical ship trajectory, using a semantic feature encoder to extract the scene semantic features, combining the trajectory encoder and end-point generation module to predict the predicted end and end-point features of the ship, using graph convolutional neural network and Transformer structures generate future trajectories to improve prediction robustness and adaptability.
Effectively dealing with multimodality and multi-ship interactions improves the accuracy and reliability of trajectory prediction, allowing the model to better understand and predict ship behavior, and is suitable for trajectory prediction in complex scenarios.
Smart Images

Figure CN118823697B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of ship trajectory prediction, and in particular to an endpoint-driven ship trajectory prediction method, device, equipment and medium. Background Art
[0002] Predicting the approximate direction and position of a vessel over medium to long timescales is central to a variety of applications including, but not limited to, search and rescue, traffic control, trajectory planning, port congestion avoidance, pollution monitoring, and more.
[0003] However, there are great obstacles to ship trajectory prediction methods. First, due to the vastness of the sea surface and the complexity of the external environment (wind speed and weather conditions, etc.), the future trajectory of the ship shows multimodality, that is, there are many possible routes for the movement of the ship in the future. Secondly, since ship data shows significant non-uniformity in different geographical regions and environmental conditions, this difference will have a negative impact on the trajectory prediction performance, thereby limiting the effectiveness of the ship trajectory model in a wider range of applications.
[0004] Based on this, a new ship trajectory prediction method is urgently needed to predict the ship trajectory. Summary of the invention
[0005] The present application provides an endpoint-driven ship trajectory prediction method, device, equipment and medium to solve the above-mentioned problems existing in the existing ship trajectory prediction methods.
[0006] In a first aspect of an embodiment of the present application, a method for predicting a ship trajectory driven by an endpoint is provided, the method comprising:
[0007] Obtaining a scene image to be predicted and a historical ship trajectory of a target ship in the scene;
[0008] Extracting features from the scene image to be predicted to obtain scene semantic features of the scene to be predicted;
[0009] Performing endpoint prediction based on the historical ship trajectory and the scene semantic features to obtain the predicted endpoint and predicted endpoint features of the target ship;
[0010] The trajectory of the target ship is predicted according to the historical ship trajectory and the predicted endpoint characteristics to obtain the future trajectory of the target ship when it reaches the predicted endpoint.
[0011] In an optional embodiment of the present application, feature extraction is performed from the scene image to be predicted to obtain the scene semantic features under the scene to be predicted by a semantic feature encoder, wherein the semantic feature encoder uses a vision transformer network as a backbone network, and the vision transformer network is trained by self-supervised comparative learning using a student network and a teacher network;
[0012] By inputting the scene image to be predicted into the vision transformer network, the semantic information in the scene image to be predicted is segmented, and a classification result is obtained as the scene semantic feature.
[0013] In an optional embodiment of the present application, the destination prediction is performed according to the historical ship trajectory and the scene semantic features, and the predicted destination and predicted destination features of the target ship are obtained by an endpoint prediction network, the endpoint prediction network includes a semantic feature encoder, a trajectory encoder, a splicing module and an endpoint generation module, and the endpoint generation module includes an endpoint encoder, an intermediate feature processing module and an endpoint decoder;
[0014] The endpoint prediction is performed according to the historical ship trajectory and the scene semantic features to obtain the predicted endpoint and predicted endpoint features of the target ship, including:
[0015] Inputting the scene image to be predicted into the semantic feature encoder for processing to obtain scene semantic features;
[0016] Inputting the historical ship trajectory into the trajectory encoder for processing to obtain historical ship trajectory features;
[0017] Inputting the scene semantic features and the historical ship tracks into the splicing module for splicing to obtain splicing features;
[0018] The splicing feature is input into the endpoint encoder in the endpoint generation module for processing, the result of the endpoint encoder is input into the intermediate feature processing module for processing, the result of the intermediate feature processing module is input into the endpoint decoder for decoding processing, and the predicted endpoint and the predicted endpoint feature are obtained.
[0019] In an optional embodiment of the present application, a future trajectory of the target ship arriving at the predicted destination is obtained through a trajectory generation network, the trajectory generation network includes a graph convolutional neural network and a Transformer structure, the Transformer structure includes a self-attention layer, a feedforward layer and a cross-attention layer, and the cross-attention layer is located between the self-attention layer and the feedforward layer;
[0020] Predicting the trajectory of the target ship according to the historical ship trajectory and the predicted destination feature to obtain the future trajectory of the target ship to reach the predicted destination includes:
[0021] The graph convolutional neural network is used to extract the trajectory features of the target ship in the historical ship trajectories to obtain the trajectory features of the target ship;
[0022] Inputting the trajectory feature of the target ship into the self-attention layer to obtain a first auxiliary feature;
[0023] Inputting the first auxiliary feature and the predicted endpoint feature into the cross-attention layer to obtain a second auxiliary feature;
[0024] Inputting the second auxiliary feature into the feedforward layer and performing nonlinear transformation processing to obtain multiple multimodal predicted trajectories of the target ship;
[0025] A final predicted trajectory is determined from the multiple multimodal predicted trajectories of the target ship as the future trajectory of the target ship to reach the predicted end point.
[0026] In an optional embodiment of the present application, determining a final predicted trajectory from a plurality of multimodal predicted trajectories of the target ship as the future trajectory of the target ship to reach the predicted destination includes:
[0027] Get the probability value corresponding to each multimodal prediction trajectory;
[0028] The generated multiple multimodal trajectories are input into the K-means clustering algorithm, and the corresponding multimodal prediction trajectories are divided into a pre-set estimated number of clusters according to the final position of each multimodal prediction trajectory;
[0029] In each cluster, the multimodal predicted trajectory with the highest probability value is selected as the future trajectory of the target ship to reach the predicted destination.
[0030] In an optional embodiment of the present application, the method is to process the scene image to be predicted and the historical ship trajectory through a pre-trained trajectory prediction model, and predict the ship trajectory of the target ship within a period of time in the future based on the predicted endpoint. The trajectory prediction model includes: a semantic feature encoder, a trajectory feature encoder, a splicing module, an endpoint generation network and a trajectory generation network. The trajectory generation network includes a graph convolutional neural network and a Transformer network structure. The Transformer network structure includes a self-attention layer, a feedforward layer and a cross-attention layer.
[0031] In an optional embodiment of the present application, the training process of the trajectory prediction model includes:
[0032] Inputting a plurality of historical ship trajectory samples and corresponding scene image samples into a trajectory prediction model to be trained, obtaining a plurality of predicted trajectory features through a trajectory feature extraction module in a trajectory generation network to be trained, and obtaining a plurality of predicted trajectories through a trajectory prediction module in the trajectory generation network to be trained;
[0033] Using a fully connected network to be trained to classify the multiple predicted trajectory features to obtain multiple trajectory categories;
[0034] Performing dimensionality reduction processing on the multiple predicted trajectory features by using a principal component analysis method, performing standardization processing on the multiple predicted trajectory features after the dimensionality reduction processing, clustering the multiple predicted trajectory features after the standardization processing, and using the clustering results as pseudo-class labels of the multiple predicted trajectory features;
[0035] Calculating a trajectory feature loss based on the pseudo-category labels of the plurality of predicted trajectory features and the plurality of trajectory categories, and calculating a trajectory loss based on the plurality of predicted trajectories and the corresponding plurality of true trajectories;
[0036] Based on the trajectory feature loss and the trajectory loss, the fully connected network to be trained and the trajectory generation network to be trained are updated to obtain a trained trajectory generation network;
[0037] The model parameters of the trained trajectory generation network are frozen, and based on a plurality of historical ship trajectory samples and corresponding scene image samples, the semantic feature encoder, the trajectory feature encoder, the splicing module and the endpoint generation network in the trajectory prediction model to be trained are trained to obtain the trained trajectory prediction model.
[0038] In a second aspect of an embodiment of the present application, a terminal-driven ship trajectory prediction device is provided, the device comprising:
[0039] An acquisition module, used to acquire the scene image to be predicted and the historical ship trajectory of the target ship in the scene;
[0040] A feature extraction module is used to extract features from the scene image to be predicted to obtain scene semantic features under the scene to be predicted;
[0041] A destination prediction module, used to predict the destination according to the historical ship trajectory and the scene semantic features, and obtain the predicted destination and predicted destination features of the target ship;
[0042] The trajectory prediction module is used to predict the trajectory of the target ship according to the historical ship trajectory and the predicted destination feature, so as to obtain the future trajectory of the target ship when it reaches the predicted destination.
[0043] In a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the endpoint-driven ship trajectory prediction method described in any one of the first aspects above.
[0044] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method for predicting the trajectory of a ship driven by an endpoint as described in any one of the first aspects above is implemented.
[0045] The present application includes the following advantages: The present application provides an endpoint-driven ship trajectory prediction method, device, equipment and medium, which obtains the scene image to be predicted and the historical ship trajectory of the target ship in the scene; extracts features from the scene image to be predicted to obtain the scene semantic features of the scene to be predicted; performs endpoint prediction based on the historical ship trajectory and the scene semantic features to obtain the predicted endpoint and predicted endpoint features of the target ship; predicts the trajectory of the target ship based on the historical ship trajectory and the predicted endpoint features to obtain the future trajectory of the target ship reaching the predicted endpoint. The trajectory prediction method based on endpoint drive can better handle multimodality and the interaction between multiple ships, improves the robustness and adaptability of trajectory prediction, enables the model to better understand and predict the behavior of the ship, and thus more reliably performs trajectory prediction in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0047] Figure 1 It is a schematic diagram of the steps of a method for predicting ship trajectory driven by an endpoint proposed in an embodiment of the present application;
[0048] Figure 2 It is an end-point driven ship trajectory prediction device proposed in the embodiment of the present application;
[0049] Figure 3 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0051] Predicting the approximate direction and position of a vessel over medium to long timescales is central to a variety of applications including, but not limited to, search and rescue, traffic control, trajectory planning, port congestion avoidance, pollution monitoring, and more.
[0052] However, there are great obstacles to ship trajectory prediction methods. First, due to the vastness of the sea surface and the complexity of the external environment (wind speed and weather conditions, etc.), the future trajectory of the ship shows multimodality, that is, there are many possible routes for the movement of the ship in the future. Secondly, since ship data shows significant non-uniformity in different geographical regions and environmental conditions, this difference will have a negative impact on the trajectory prediction performance, thereby limiting the effectiveness of the ship trajectory model in a wider range of applications.
[0053] Based on this, the present application introduces the concept of endpoint drive in the ship trajectory measurement method, and models the future trajectory of the ship as a process of reaching the target endpoint. By emphasizing the individual goals of the ship, the model can more specifically predict the future trajectory of the ship.
[0054] In a first aspect of the present application, a method for predicting a ship trajectory driven by an endpoint is provided. Figure 1 , Figure 1 1 is a schematic diagram of the steps of a method for predicting ship trajectory driven by an endpoint according to an embodiment of the present application, the method comprising the following steps:
[0055] Step 101: Acquire a scene image to be predicted and a historical ship trajectory of a target ship in the scene;
[0056] Step 102: extracting features from the scene image to be predicted to obtain scene semantic features of the scene to be predicted;
[0057] Step 103: performing destination prediction according to the historical ship trajectory and the scene semantic features to obtain the predicted destination and predicted destination features of the target ship;
[0058] Step 104: predicting the trajectory of the target ship according to the historical ship trajectory and the predicted destination feature to obtain the future trajectory of the target ship when it reaches the predicted destination.
[0059] When implementing step 101, the present application aims to make the proposed ship trajectory prediction method include a large number of scenarios with high prediction accuracy in complex scenarios. The complex scenarios may include multiple variables, high uncertainty, dynamic changes, and multi-level interactions. At the same time, considering that the historical trajectory information of the ship is an important component information in the study of the ship's navigation process as a special object. Therefore, in the embodiment of the present application, the scene image where the target ship to be predicted is located is first obtained as the scene image to be predicted, and the historical ship trajectory of the target ship contained in the scene is obtained.
[0060] When implementing step 102, considering that in the prediction of ship trajectory, the navigation route of the ship in the scene is restricted by the environment, and the destination of the ship is also affected by the environment, in this application, a semantic feature encoder is used to extract semantic features from the scene image to obtain the scene semantic features of the scene to be predicted. This process uses the scene image to be predicted as the input object, and realizes the semantic feature extraction from low to high layers through the pre-trained semantic feature encoder. The extracted semantic features can provide the following information: 1) Provide scene constraints: provide navigable area and obstacle information, constrain the range and direction of ship movement; 2) Provide semantic information: affect the movement goals and preferences of the ship; 3) Provide contextual environment information: help understand the interaction between the ship and the environment.
[0061] In an optional embodiment of the present application, feature extraction is performed from the above-mentioned scene image to be predicted, and scene semantic features of the scene to be predicted are obtained by a semantic feature encoder, wherein the above-mentioned semantic feature encoder uses a vision transformer network as a backbone network, and the above-mentioned vision transformer network is trained by self-supervised comparative learning using a student network and a teacher network; by inputting the above-mentioned scene image to be predicted into the vision transformer network, the semantic information in the above-mentioned scene image to be predicted is segmented, and a classification result is obtained as the above-mentioned scene semantic feature.
[0062] Exemplarily, the DiNO model is selected as a semantic feature encoder in the embodiment of the present application to extract scene semantic features from the scene image to be predicted. DiNO uses the vision transformer network as the backbone network of the visual feature extractor, and uses two sub-networks with exactly the same model results: the student network and the teacher network, and adopts a self-supervised comparative learning method for training. The model has a relatively good segmentation effect on the speech information in the scene image and can extract more detailed scene semantic features.
[0063] When the specific implementation step 103 is performed: predicting the ship's trajectory, that is, predicting the future time-series trajectory coordinates of the ship, the prediction error of each time step will be passed to the next time step, causing the final trajectory to deviate from the true value, thereby generating a future trajectory outside the distribution, resulting in a serious deviation between the prediction result and the true value. Therefore, the present application adopts an endpoint-driven trajectory prediction method, which can improve the prediction accuracy from the following aspects: 1) Decomposition of the problem: simplifying the ship trajectory prediction problem to predicting the future target position of the ship, and then generating the corresponding trajectory based on the target position; 2) Responding to intentions: The end point of the ship is the intention of the ship, and the method based on the end point can directly reflect the intention of the ship, which can increase the multimodality and interpretability of the prediction; 3) Improving robustness: The end point is more stable than the trajectory, which can reduce prediction errors and noise and improve generalization ability.
[0064] The present application takes into account that the prediction of the ship's destination mainly depends on two aspects, one is the scene environment around the ship, and the other is the historical trajectory of the ship itself. Therefore, the embodiment of the present application mainly involves two types of inputs when predicting the destination: historical ship trajectory and scene semantic features, that is, the destination prediction is performed based on the above-mentioned historical ship trajectory and the above-mentioned scene semantic features to obtain the predicted destination and predicted destination features of the above-mentioned target ship, the predicted destination is the coordinate information, and the predicted destination features are the feature information of the predicted destination.
[0065] In an optional embodiment of the present application, the endpoint prediction is performed according to the above-mentioned historical ship trajectory and the above-mentioned scene semantic features, and the predicted endpoint and predicted endpoint features of the above-mentioned target ship are obtained through an endpoint prediction network. The above-mentioned endpoint prediction network includes a semantic feature encoder, a trajectory encoder, a splicing module and an endpoint generation module, and the above-mentioned endpoint generation module includes an endpoint encoder, an intermediate feature processing module and an endpoint decoder; specifically, the endpoint prediction is performed according to the above-mentioned historical ship trajectory and the scene semantic features to obtain the predicted endpoint and predicted endpoint features of the above-mentioned target ship, including the following process: First, the above-mentioned scene image to be predicted is input into the above-mentioned semantic feature encoder for processing to obtain the scene semantic features; the semantic feature encoder for feature extraction of the scene image to be predicted here has the same network structure as the semantic feature encoder described in the above-mentioned embodiment. Then, the historical ship trajectory is input into the trajectory encoder for processing to obtain the historical ship trajectory features; further, the scene semantic features and the historical ship trajectory are input into the splicing module for splicing to obtain the splicing features; finally, the splicing features are input into the endpoint encoder in the endpoint generation module for processing, the result of the endpoint encoder is input into the intermediate feature processing module for processing, the result of the intermediate feature processing module is input into the endpoint decoder for decoding processing to obtain the predicted endpoint and the predicted endpoint features.
[0066] When step 104 is specifically implemented: after obtaining the historical ship trajectory and the predicted endpoint characteristics of the target ship, the trajectory of the target ship is predicted based on the predicted endpoint characteristics to obtain the future trajectory of the target ship reaching the above-mentioned predicted endpoint.
[0067] In an optional embodiment of the present application, the trajectory of the target ship is predicted by a trajectory generation network, and the future trajectory of the target ship arriving at the predicted destination is obtained by the trajectory generation network. The trajectory generation network includes a graph convolutional neural network and a Transformer structure, and the Transformer structure includes a self-attention layer, a feedforward layer, and a cross-attention layer. The cross-attention layer is located between the self-attention layer and the feedforward layer. By adding a cross-attention layer between the self-attention layer and the feedforward layer, the predicted destination feature can be integrated in the trajectory prediction process.
[0068] Specifically, the trajectory of the target ship is predicted according to the historical ship trajectory and the predicted endpoint to obtain the future trajectory of the target ship reaching the predicted endpoint, including the following processes: first, considering that the trajectory of the target ship will be affected by other surrounding ships, the graph convolutional neural network is used to extract the trajectory features of the target ship in the historical ship trajectory to obtain the trajectory features of the target ship; a graph convolutional neural network (GCN) is a deep learning model that learns on graph structured data. In the trajectory generation features of the target ship, the graph convolutional neural network is used to extract features of the trajectory of the target ship. GCN helps to capture the complex relationship between the trajectory of the target ship and the surrounding ships and accurately extract the trajectory features of the target ship.
[0069] Secondly, the decoder part of the Transformer structure is improved, a cross-attention layer is added between the self-attention layer and the feedforward layer, and prediction is performed in an autoregressive manner. Specifically, in the self-attention layer, in order to prevent information leakage, a mask mechanism is adopted. At each position, only the information before the position can be noticed, which ensures that only the generated part is used when generating the output of the current position. This is very important for sequence generation tasks, because when generating the output at a certain moment, future information should not be used. Based on this, the trajectory feature of the target ship is input into the self-attention layer to obtain the first auxiliary feature, which represents the auxiliary trajectory prediction through the target ship; the decoder also includes a cross-attention layer, which allows the model to pay attention to different parts of the input sequence. The input of the cross-attention layer comes from the output of the self-attention layer, as well as the predicted endpoint feature. The predicted endpoint feature is fused into the first auxiliary feature through the cross-attention layer to obtain the second auxiliary feature, which is used to assist the trajectory prediction through the endpoint.
[0070] Furthermore, by setting a feedforward neural network layer (feedforward layer) after the multi-head attention layer, the second auxiliary feature is subjected to nonlinear transformation processing to obtain multiple multimodal predicted trajectories of the target ship. The feedforward layer can help the model learn more complex feature representations, which is very important for understanding and generating the trajectory of the target ship. Finally, the final predicted trajectory is determined from the multiple multimodal predicted trajectories of the target ship as the future trajectory of the target ship reaching the predicted end point.
[0071] In an optional embodiment of the present application, in order to solve the deviation problem caused by extracting a limited number of samples from the prediction distribution and improve the prediction diversity, a cluster analysis is performed on the generated multiple multimodal prediction trajectories to obtain the final prediction trajectory, which specifically includes the following process:
[0072] First, the probability value corresponding to each multimodal predicted trajectory is obtained. The probability value of each multimodal predicted trajectory reflects the uncertainty and diversity of the trajectory prediction model for future trajectories.
[0073] Then, based on a preset estimated number, i.e., K value, the generated multiple multimodal trajectories are input into the K-means clustering algorithm, and the K-means clustering algorithm divides the corresponding multimodal predicted trajectories into a preset estimated number of clusters according to the final position of each multimodal predicted trajectory; the final position here can be the end point of the multimodal predicted trajectory, indicating different positions that the target ship may reach within the time range predicted by the model, which may be different from the predicted end point.
[0074] Finally, in each cluster, the multimodal predicted trajectory with the highest probability value is selected as the future trajectory of the target ship to reach the above-mentioned predicted end point. Each cluster represents a possible trajectory direction, thereby improving the diversity of the generated results. In this way, not only can deviations be avoided, but also a more comprehensive selection can be provided among multiple possible trajectory schemes. By introducing a clustering mechanism in the multimodal trajectory generation method, the final trajectory prediction results are more diverse and comprehensive, which helps to deal with the uncertainty of the future trajectory of the target ship and provides more expressive generation results, making the trajectory prediction model more suitable for the diversity needs in practical applications.
[0075] In an optional embodiment of the present application, the endpoint-driven ship trajectory prediction method proposed in the present application is to process the above-mentioned scene image to be predicted and the historical ship trajectory through a pre-trained trajectory prediction model, and predict the ship trajectory of the target ship in the future period of time based on the predicted endpoint. The above-mentioned trajectory prediction model includes: a semantic feature encoder, a trajectory feature encoder, a splicing module, an endpoint generation network and a trajectory generation network. The above-mentioned trajectory generation network includes a graph convolutional neural network and a Transformer network structure. The above-mentioned Transformer network structure includes a self-attention layer, a feedforward layer and a cross-attention layer.
[0076] In an optional embodiment of the present application, in the above-mentioned trajectory prediction model, the trajectory generation network in the trajectory prediction model is pre-trained. In the process of training the trajectory generation network, the influence of the characteristics of the navigation trajectory on the ship trajectory is taken into consideration. During the training process, the trajectory generation network learns the trajectory characteristics, so that the trained trajectory generation network can learn features with multimodal information and generate a more accurate predicted trajectory.
[0077] The training process of the above trajectory generation network includes: first, inputting multiple historical ship trajectory samples and corresponding scene image samples into the trajectory prediction model to be trained, obtaining multiple predicted trajectory features through the trajectory feature extraction module in the trajectory generation network to be trained, and obtaining multiple predicted trajectories through the trajectory prediction module in the trajectory generation network to be trained; using the fully connected network to be trained (which can be an MLP multi-layer perceptron) to classify the above multiple predicted trajectory features to obtain multiple trajectory categories; wherein, according to different classification standards, the trajectory categories are different, for example, according to the intention, the trajectory category can be straight, lane change, turn, etc., according to the kinematic details, the trajectory category can be the angle of the turn, the length of the trajectory, etc. Based on the classification of the trajectory, the trajectory generation network trained in this way can learn features with multimodal information. Then, the principal component analysis method is used to reduce the dimension of the above-mentioned multiple predicted trajectory features, and the multiple predicted trajectory features after the dimension reduction processing are standardized, and the multiple predicted trajectory features after the standardization processing are clustered, and the clustering results are used as pseudo-category labels of the above-mentioned multiple predicted trajectory features; based on the pseudo-category labels of the above-mentioned multiple predicted trajectory features and the above-mentioned multiple trajectory categories, the trajectory feature loss is calculated, and the trajectory loss is calculated based on the above-mentioned multiple predicted trajectories and the corresponding multiple real trajectories; in order to reduce the error in the prediction process, the above-mentioned trajectory loss can be calculated by the mean square error; based on the trajectory feature loss and the trajectory loss, the above-mentioned fully connected network to be trained and the trajectory generation network to be trained are updated to obtain a trained trajectory generation network. In the pre-training process of the trajectory generation network, the fully connected network and the principal component analysis method are used for training. When the trained trajectory generation network is actually used, the trained fully connected network and the principal component analysis method are not used to analyze the generated results.
[0078] The training process of the above-mentioned trajectory prediction model includes: freezing the model parameters of the trained trajectory generation network, training the semantic feature encoder, trajectory feature encoder, splicing module and terminal generation network in the trajectory prediction model to be trained based on multiple historical ship trajectory samples and corresponding scene image samples, and obtaining the trained trajectory prediction model. In the process of training the trajectory prediction model, the model parameters of the trajectory generation network are frozen so that the trajectory generation network will not forget the learned trajectory features, so that in the actual prediction process, the learned trajectory features are combined into the predicted trajectory, making the predicted ship trajectory more accurate.
[0079] An endpoint-driven ship trajectory prediction method provided in an embodiment of the present application is implemented by acquiring a scene image to be predicted and the historical ship trajectory of the target ship in the scene; extracting features from the scene image to be predicted to obtain scene semantic features of the scene to be predicted; performing endpoint prediction based on the historical ship trajectory and the scene semantic features to obtain the predicted endpoint and predicted endpoint features of the target ship; predicting the trajectory of the target ship based on the historical ship trajectory and the predicted endpoint features to obtain the future trajectory of the target ship when it reaches the predicted endpoint. The endpoint-driven trajectory prediction method can better handle multimodality and the interaction between multiple ships, improve the robustness and adaptability of trajectory prediction, and enable the model to better understand and predict the behavior of the ship, thereby more reliably performing trajectory prediction in complex scenarios.
[0080] In a second aspect of the present application, a terminal driven ship trajectory prediction device is provided, referring to Figure 2 , Figure 2 The present invention provides an end-point driven ship trajectory prediction device, which includes:
[0081] An acquisition module 201 is used to acquire an image of a scene to be predicted and a historical ship track of a target ship in the scene;
[0082] A feature extraction module 202 is used to extract features from the scene image to be predicted to obtain scene semantic features of the scene to be predicted;
[0083] The destination prediction module 203 is used to perform destination prediction based on the historical ship trajectory and the scene semantic features to obtain the predicted destination and predicted destination features of the target ship;
[0084] The trajectory prediction module 204 is used to predict the trajectory of the target ship according to the historical ship trajectory and the predicted destination feature, and obtain the future trajectory of the target ship to reach the predicted destination.
[0085] The feature extraction module is implemented by a semantic feature encoder, which uses a vision transformer network as a backbone network, and the vision transformer network is trained by self-supervised comparative learning using a student network and a teacher network;
[0086] The feature extraction module also includes:
[0087] The segmentation submodule is used to segment the semantic information in the scene image to be predicted by inputting the scene image to be predicted into the vision transformer network, and obtain the classification result as the scene semantic feature
[0088] The endpoint prediction module is implemented by an endpoint prediction network, which includes a semantic feature encoder, a trajectory encoder, a splicing module and an endpoint generation module. The endpoint generation module includes an endpoint encoder, an intermediate feature processing module and an endpoint decoder. The endpoint prediction module includes:
[0089] A semantic feature extraction submodule, used for inputting the scene image to be predicted into the semantic feature encoder for processing to obtain scene semantic features;
[0090] A trajectory encoding submodule, used for inputting the historical ship trajectory into the trajectory encoder for processing to obtain historical ship trajectory features;
[0091] A splicing submodule, used for inputting the scene semantic features and the historical ship tracks into the splicing module for splicing to obtain splicing features;
[0092] The endpoint encoding submodule is used to input the splicing features into the endpoint encoder in the endpoint generation module for processing, input the result of the endpoint encoder into the intermediate feature processing module for processing, input the result of the intermediate feature processing module into the endpoint decoder for decoding, and obtain the predicted endpoint and the predicted endpoint feature.
[0093] The trajectory prediction module is implemented by a trajectory generation network, which includes a graph convolutional neural network and a Transformer structure. The Transformer structure includes a self-attention layer, a feedforward layer, and a cross-attention layer, and the cross-attention layer is located between the self-attention layer and the feedforward layer.
[0094] The trajectory prediction module comprises:
[0095] A trajectory feature extraction submodule, used to extract the trajectory features of the target ship in the historical ship trajectory using the graph convolutional neural network to obtain the trajectory features of the target ship;
[0096] A first attention submodule, used for inputting the trajectory feature of the target ship into the self-attention layer to obtain a first auxiliary feature;
[0097] A second attention submodule, configured to input the first auxiliary feature and the predicted endpoint feature into the cross-attention layer to obtain a second auxiliary feature;
[0098] A nonlinear transformation submodule, used for inputting the second auxiliary feature into the feedforward layer, performing nonlinear transformation processing, and obtaining a plurality of multimodal predicted trajectories of the target ship;
[0099] The trajectory determination submodule is used to determine a final predicted trajectory from the multiple multimodal predicted trajectories of the target ship as the future trajectory of the target ship to reach the predicted end point.
[0100] Wherein, the trajectory determination submodule includes:
[0101] A probability value acquisition unit, used to obtain a probability value corresponding to each multimodal prediction trajectory;
[0102] A partitioning unit, used for inputting the generated multiple multimodal trajectories into a K-means clustering algorithm, and partitioning the corresponding multimodal prediction trajectory into a preset estimated number of clusters according to the final position of each multimodal prediction trajectory;
[0103] The determination unit is used to select the multimodal predicted trajectory with the highest probability value in each cluster as the future trajectory of the target ship to reach the predicted end point.
[0104] The device processes the scene image to be predicted and the historical ship trajectory through a pre-trained trajectory prediction model, and predicts the ship trajectory of the target ship within a period of time in the future based on the predicted endpoint. The trajectory prediction model includes: a semantic feature encoder, a trajectory feature encoder, a splicing module, an endpoint generation network and a trajectory generation network. The trajectory generation network includes a graph convolutional neural network and a Transformer network structure. The Transformer network structure includes a self-attention layer, a feedforward layer and a cross-attention layer.
[0105] Wherein, the device further includes a training module, and the training module includes:
[0106] A predicted trajectory acquisition submodule, used for inputting a plurality of historical ship trajectory samples and corresponding scene image samples into a trajectory prediction model to be trained, obtaining a plurality of predicted trajectory features through a trajectory feature extraction module in a trajectory generation network to be trained, and obtaining a plurality of predicted trajectories through a trajectory prediction module in the trajectory generation network to be trained;
[0107] A classification submodule, used for classifying the plurality of predicted trajectory features using a fully connected network to be trained to obtain a plurality of trajectory categories;
[0108] a pseudo-label acquisition submodule, configured to perform dimensionality reduction processing on the plurality of predicted trajectory features by using a principal component analysis method, perform standardization processing on the plurality of predicted trajectory features after the dimensionality reduction processing, perform clustering processing on the plurality of predicted trajectory features after the standardization processing, and use the clustering results as pseudo-category labels of the plurality of predicted trajectory features;
[0109] A loss calculation submodule, configured to calculate a trajectory feature loss based on the pseudo-category labels of the plurality of predicted trajectory features and the plurality of trajectory categories, and to calculate a trajectory loss based on the plurality of predicted trajectories and the corresponding plurality of true trajectories;
[0110] An updating submodule, used for updating the fully connected network to be trained and the trajectory generation network to be trained based on the trajectory feature loss and the trajectory loss, so as to obtain a trained trajectory generation network;
[0111] The training submodule is used to freeze the model parameters of the trained trajectory generation network, and train the semantic feature encoder, trajectory feature encoder, splicing module and endpoint generation network in the trajectory prediction model to be trained based on multiple historical ship trajectory samples and corresponding scene image samples to obtain the trained trajectory prediction model.
[0112] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 3 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 3 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, the memory 110 stores a computer program, and the computer program can be run on the processor 120 to implement an end-point driven ship trajectory prediction method disclosed in an embodiment of the present application.
[0113] Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, an end-point driven ship trajectory prediction method disclosed in the embodiment of the present application is implemented.
[0114] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0115] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0116] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0118] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0119] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0120] The above is a detailed introduction to the endpoint-driven ship trajectory prediction method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A terminal driven ship trajectory prediction method, characterized in that: The method comprises: Obtaining a scene image to be predicted and a historical ship trajectory of a target ship in the scene; Extracting features from the scene image to be predicted to obtain scene semantic features of the scene to be predicted; Performing endpoint prediction based on the historical ship trajectory and the scene semantic features to obtain the predicted endpoint and predicted endpoint features of the target ship; Predicting the trajectory of the target ship according to the historical ship trajectory and the predicted endpoint characteristics to obtain a future trajectory of the target ship to reach the predicted endpoint; The future trajectory of the target ship arriving at the predicted destination is obtained through a trajectory generation network, wherein the trajectory generation network includes a graph convolutional neural network and a Transformer structure, wherein the Transformer structure includes a self-attention layer, a feedforward layer, and a cross-attention layer, wherein the cross-attention layer is located between the self-attention layer and the feedforward layer; Predicting the trajectory of the target ship according to the historical ship trajectory and the predicted destination feature to obtain the future trajectory of the target ship to reach the predicted destination includes: The graph convolutional neural network is used to extract the trajectory features of the target ship in the historical ship trajectories to obtain the trajectory features of the target ship; Inputting the trajectory feature of the target ship into the self-attention layer to obtain a first auxiliary feature; Inputting the first auxiliary feature and the predicted endpoint feature into the cross-attention layer to obtain a second auxiliary feature; Inputting the second auxiliary feature into the feedforward layer and performing nonlinear transformation processing to obtain multiple multimodal predicted trajectories of the target ship; A final predicted trajectory is determined from the multiple multimodal predicted trajectories of the target ship as the future trajectory of the target ship to reach the predicted end point.
2. The method for predicting the trajectory of a ship driven by an endpoint according to claim 1, characterized in that: Extracting features from the scene image to be predicted, and obtaining scene semantic features under the scene to be predicted is achieved by a semantic feature encoder, wherein the semantic feature encoder uses a vision transformer network as a backbone network, and the vision transformer network is trained by self-supervised comparative learning using a student network and a teacher network; By inputting the scene image to be predicted into the vision transformer network, the semantic information in the scene image to be predicted is segmented, and a classification result is obtained as the scene semantic feature.
3. The method for predicting ship trajectory driven by endpoint according to claim 1, characterized in that: The destination prediction is performed according to the historical ship trajectory and the scene semantic features, and the predicted destination and predicted destination features of the target ship are obtained by a destination prediction network, wherein the destination prediction network includes a semantic feature encoder, a trajectory encoder, a splicing module and a destination generation module, and the destination generation module includes a destination encoder, an intermediate feature processing module and a destination decoder; The endpoint prediction is performed according to the historical ship trajectory and the scene semantic features to obtain the predicted endpoint and predicted endpoint features of the target ship, including: Inputting the scene image to be predicted into the semantic feature encoder for processing to obtain scene semantic features; Inputting the historical ship trajectory into the trajectory encoder for processing to obtain historical ship trajectory features; Inputting the scene semantic features and the historical ship tracks into the splicing module for splicing to obtain splicing features; The splicing feature is input into the endpoint encoder in the endpoint generation module for processing, the result of the endpoint encoder is input into the intermediate feature processing module for processing, the result of the intermediate feature processing module is input into the endpoint decoder for decoding processing, and the predicted endpoint and the predicted endpoint feature are obtained.
4. The method for predicting the trajectory of a ship driven by an endpoint according to claim 1, characterized in that: Determining a final predicted trajectory from a plurality of multimodal predicted trajectories of the target ship as a future trajectory of the target ship to reach the predicted end point includes: Get the probability value corresponding to each multimodal prediction trajectory; The generated multiple multimodal trajectories are input into the K-means clustering algorithm, and the corresponding multimodal prediction trajectories are divided into a pre-set estimated number of clusters according to the final position of each multimodal prediction trajectory; In each cluster, the multimodal predicted trajectory with the highest probability value is selected as the future trajectory of the target ship to reach the predicted destination.
5. The method for predicting ship trajectory driven by endpoint according to claim 1, characterized in that: The method processes the scene image to be predicted and the historical ship trajectory through a pre-trained trajectory prediction model, and predicts the ship trajectory of the target ship within a period of time in the future based on the predicted endpoint. The trajectory prediction model includes: a semantic feature encoder, a trajectory feature encoder, a splicing module, an endpoint generation network and a trajectory generation network. The trajectory generation network includes a graph convolutional neural network and a Transformer network structure. The Transformer network structure includes a self-attention layer, a feedforward layer and a cross-attention layer.
6. The method for predicting the trajectory of a ship driven by an endpoint according to claim 5, characterized in that: The training process of the trajectory prediction model includes: Inputting a plurality of historical ship trajectory samples and corresponding scene image samples into a trajectory prediction model to be trained, obtaining a plurality of predicted trajectory features through a trajectory feature extraction module in a trajectory generation network to be trained, and obtaining a plurality of predicted trajectories through a trajectory prediction module in the trajectory generation network to be trained; Using a fully connected network to be trained to classify the multiple predicted trajectory features to obtain multiple trajectory categories; Performing dimensionality reduction processing on the multiple predicted trajectory features by using a principal component analysis method, performing standardization processing on the multiple predicted trajectory features after the dimensionality reduction processing, clustering the multiple predicted trajectory features after the standardization processing, and using the clustering results as pseudo-class labels of the multiple predicted trajectory features; Calculating a trajectory feature loss based on the pseudo-category labels of the plurality of predicted trajectory features and the plurality of trajectory categories, and calculating a trajectory loss based on the plurality of predicted trajectories and the corresponding plurality of true trajectories; Based on the trajectory feature loss and the trajectory loss, the fully connected network to be trained and the trajectory generation network to be trained are updated to obtain a trained trajectory generation network; The model parameters of the trained trajectory generation network are frozen, and based on a plurality of historical ship trajectory samples and corresponding scene image samples, the semantic feature encoder, the trajectory feature encoder, the splicing module and the endpoint generation network in the trajectory prediction model to be trained are trained to obtain the trained trajectory prediction model.
7. An endpoint-driven ship trajectory prediction device, characterized in that: The device comprises: An acquisition module, used to acquire the scene image to be predicted and the historical ship trajectory of the target ship in the scene; A feature extraction module, used to extract features from the scene image to be predicted to obtain scene semantic features under the scene to be predicted; A destination prediction module, used to predict the destination according to the historical ship trajectory and the scene semantic features, and obtain the predicted destination and predicted destination features of the target ship; A trajectory prediction module, used to predict the trajectory of the target ship according to the historical ship trajectory and the predicted destination feature, and obtain the future trajectory of the target ship to reach the predicted destination; The trajectory prediction module is implemented by a trajectory generation network, which includes a graph convolutional neural network and a Transformer structure. The Transformer structure includes a self-attention layer, a feedforward layer, and a cross-attention layer, and the cross-attention layer is located between the self-attention layer and the feedforward layer. The trajectory prediction module comprises: A trajectory feature extraction submodule, used to extract the trajectory features of the target ship in the historical ship trajectory using the graph convolutional neural network to obtain the trajectory features of the target ship; A first attention submodule, used for inputting the trajectory feature of the target ship into the self-attention layer to obtain a first auxiliary feature; A second attention submodule, configured to input the first auxiliary feature and the predicted endpoint feature into the cross-attention layer to obtain a second auxiliary feature; A nonlinear transformation submodule, used for inputting the second auxiliary feature into the feedforward layer, performing nonlinear transformation processing, and obtaining a plurality of multimodal predicted trajectories of the target ship; The trajectory determination submodule is used to determine a final predicted trajectory from the multiple multimodal predicted trajectories of the target ship as the future trajectory of the target ship to reach the predicted end point.
8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the endpoint-driven ship trajectory prediction method according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method for predicting the trajectory of a ship driven by an endpoint as described in any one of claims 1-6 is implemented.
Citation Information
Patent Citations
Ship experience route extraction system and method based on AIS historical data
CN114564545A
Dynamic knowledge-driven ship track prediction method and system
CN117474184A
Trajectory prediction method, vehicle, device and storage medium
CN118129756A
Ship image trajectory tracking and prediction method based on ship heading recognition
US20240013402A1
