LSTM multi-ship trajectory prediction method based on state reconstruction

Through the LSTM multi-ship trajectory prediction method based on state reconstruction, combined with the message delivery mechanism of LSTM and graph convolution network, the mutual influence problem of trajectory prediction in multiple ship scenarios is solved, and a more accurate trajectory prediction effect is achieved.

CN118917488APending Publication Date: 2024-11-08SHANGHAI SHIP & SHIPPING RES INST CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411119824.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to consider the mutual influence relationship between ships in multiple ship scenarios, resulting in poor trajectory prediction and ineffective processing of noise and missing data in AIS data.

Method used

The LSTM multi-ship trajectory prediction method based on state reconstruction is adopted, through data preprocessing, feature extraction, state refinement and iterative refinement, combined with the message delivery mechanism of LSTM and graph convolution network, considering the mutual characteristics and social information between ships. Perform trajectory prediction.

Benefits of technology

It realizes trajectory prediction in multiple ship scenarios, has a wide range of application and good prediction effect, can effectively consider the relationship between ships and its own trajectory, and improves the accuracy of prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917488A_ABST
    Figure CN118917488A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of ship trajectory prediction, and provides an LSTM multi-ship trajectory prediction method based on state reconstruction, which combines a message passing mechanism of a graph convolutional network with an LSTM, enables a model to adaptively update the current state of each ship, considers the interaction between ship trajectories, and improves the prediction accuracy of the ship trajectories. The social effect of adjacent ships is adaptively extracted through the social information selection module composed of ship selection attention and a ship moving door, so that the model can pay attention to different individuals and features at the same time, the improved LSTM can realize trajectory prediction in a multi-ship scene, the application range is wide, and the prediction effect is good.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of ship trajectory prediction, and in particular to a method for LSTM multi-ship trajectory prediction based on state reconstruction. Background Art

[0002] With the emergence of large and high-speed ships, especially with the introduction of intelligent and autonomous ships, maritime transportation faces an increasingly complex traffic environment and new safety challenges. Trajectory prediction is widely regarded as one of the key technologies to achieve safe autonomous navigation, and ship trajectory prediction driven by automatic identification system data can effectively help identify abnormal ship behavior and reduce maritime risks such as collision, grounding and contact.

[0003] Automatic Identification System (AIS) provides valuable information for monitoring and surveillance of maritime traffic. AIS data can provide dynamic information of nearby ships (current position represented by latitude and longitude coordinates, current course information relative to the ground, etc.), navigation information, and static information (identification number in the format of Maritime Mobile Service Identity (MMSI number), ship name, etc.). Although AIS can provide a variety of information, AIS data composed of ship trajectories inevitably contain noise or missing data that interfere with the conclusion. To address this problem, an improved kinematic interpolation method is proposed, which combines data preprocessing with interpolation. In the interpolation process, the ship's kinematic information such as the position, velocity and acceleration of the ship is fully considered. For the entangled points appearing in AIS data, an algorithm called graph signal variation detection (GSVD) is proposed to identify and reconstruct the entanglement, preprocess the data, and remove redundant points and stationary points. All of the above shows that it takes a lot of time to process AIS data sets, and there will be some missing points in the data sets that require reconstruction of the ship's position.

[0004] In order to consider the complex spatial dependencies between ships in the marine environment, the use of graph convolutional networks is proposed. Graph convolutional networks are good at capturing the spatial relationships between neighboring ships, enabling the model to consider the impact of surrounding ships on trajectory prediction. Although graph convolutional networks can take into account the relevant impacts between ships, it remains a challenge to fully consider the interactions between ships. In recent years, data-driven learning methods, especially recurrent neural network models, have been favored by researchers because they have shown a strong ability to learn the subtleties of social interactions from large-scale data.

[0005] In addition, ship trajectory prediction also needs to consider that ship trajectories show obvious time correlation. In recent years, some studies have attempted to learn social behaviors based on recurrent neural networks, such as long short-term memory networks (LSTM). In these methods, LSTM is usually used to predict the ship's travel process. The network parameters of all ships are shared by LSTM. For each ship, the LSTM hidden state is used as a feature representation for predicting future trajectories. In order to use mutual trajectory information to predict trajectories, a social pooling is proposed, and the hidden states of adjacent LSTMs are also usually used as pooling. However, due to the long time interval in AIS data, interpolation methods are often used to make up for the missing data in the middle, so the features in the previous time step cannot show their latest state. These features can only represent a slightly delayed trajectory pattern. Summary of the invention

[0006] In order to solve the problem that the prior art can only predict trajectories based on time but cannot consider the mutual influence relationship between multiple ships, and cannot realize trajectory prediction in multi-ship scenarios, the present invention proposes a LSTM multi-ship trajectory prediction method based on state reconstruction. The method can take into account the interaction of trajectories between ships while taking into account the operation trajectory of the ship itself, and finally realize trajectory prediction in multi-ship scenarios, which has a wide range of applications and good prediction effect.

[0007] The technical solution of the present invention is as follows:

[0008] A method for LSTM multi-ship trajectory prediction based on state reconstruction, comprising the following steps:

[0009] S1: Data preprocessing: preprocess the AIS data to be predicted; the AIS data includes: longitude and latitude, time, and ship number information; use interpolation processing method to supplement the missing AIS data and filter to obtain ship track data;

[0010] S2: Feature extraction: The AIS data preprocessed in S1 is input into the LSTM network for feature extraction to obtain ship features including position information and time information;

[0011] S3: State-refined ship features: The ship features initially extracted in S2 are reconstructed into states, and the reconstructed features are input into the LSTM network for trajectory prediction; the trajectories of multiple ships are regarded as a graph using the position information in the predicted ship features, and used as the input of the graph convolutional network; the message passing mechanism in the graph convolutional network is used to extract the mutual features between ships, and finally the ship features after state refinement are obtained through the linear layer;

[0012] S4: Iterative refinement of ship features: The ship trajectory features refined in S3 are further refined using the message passing mechanism of the graph convolutional network;

[0013] S5: Trajectory prediction: The features after iterative state refinement are used as the input of the next LSTM network, input into the input unit of the next LSTM network, and finally output the predicted trajectory.

[0014] Preferably, the specific steps of the AIS data preprocessing process are as follows:

[0015] S11, performing difference processing on the AIS data with a short missing time interval;

[0016] S12, AIS data with a long time interval missing is not considered to be a track;

[0017] S13, filtering the data processed by S11 and S12, and filtering out the AIS data of multiple ships in the same time period.

[0018] Preferably, the specific steps in the state refinement are as follows:

[0019] S31: reconstructing the state of the initially extracted features, and using the LSTM network to predict after reconstruction to obtain feature A; the feature A includes: position information and hidden state; the message passing mechanism of the graph convolutional network is introduced in the state reconstruction;

[0020] S32: Using the position information in feature A, the trajectories of multiple ships are considered as a graph and used as the input of the graph convolutional network;

[0021] S33: The mutual features between ships are extracted by using the message passing mechanism in the graph convolutional network, and finally the features after state refinement are obtained through the linear layer; the mutual features include: the hidden state of the LSTM and the attention value of the corresponding ship; the features after state refinement include: the number, position information and hidden state of adjacent ships.

[0022] Preferably, the state reconstruction step first aligns the ships according to the timestamps, and then takes the three information sources of all ships as input, the three information sources including the current position of the ship in the AIS data, the hidden state in the LSTM and the unit state of the LSTM, and outputs the reconstructed unit state; mathematically, the operation for reconstructing the unit state can be expressed as:

[0023]

[0024] in, represents a message passing function, represents the hidden state of the current target ship LSTM, Representatives in Based on the hidden state of the LSTM after one iteration, Represents the current LSTM unit state, t represents time, and l represents the number of iterations.

[0025] Preferably, the message passing mechanism of the graph convolutional network implements the message passing mechanism of the graph convolutional network through message passing and aggregation between nodes;

[0026] The node is a set of feature representations of the ship, and the feature representation includes the attributes or status of the ship;

[0027] The message passing mechanism of the graph convolutional network is implemented by the following steps: First, for each node, the feature representation of its neighboring nodes is passed as a message. The specific expression is as follows:

[0028]

[0029]

[0030] In the formula, |N j | represents the number of elements in N(i), Represents the hidden state of the LSTM of the neighbor node, the message passing function The hidden state of the LSTM does not depend on the target ship W mp is a linear transformation used to transmit a message from a neighboring node j to a target node i;

[0031] Then, a weighted sum aggregation operation is performed on the received messages to obtain a new feature representation of the node.

[0032] Preferably, the message transmission mechanism introduces a social information selection module, including: ship selection attention and ship movement gate, the ship selection attention is to select the number of ships, the ship movement gate is to select the hidden state, and the specific expression is as follows:

[0033]

[0034] Among them, ⊙ represents the dot product operation, Select attention for ships, that is, the attention between ships. is the selection coefficient of the ship motion gate, i.e. the hidden state, is the relative position size in space, is an intermediate variable, and the specific calculation formula is as follows:

[0035]

[0036]

[0037]

[0038]

[0039] Among them, δ represents the sigmoid activation function, φ r Represents the relative position calculation function, Respectively represent the horizontal and vertical coordinates of the target ship and the adjacent ship, W m 、b m , W T , W aT is the corresponding weight parameter, and the overall hidden state size is calculated through the dot product operation in formula (4).

[0040] Preferably, the refinement iterative method is: the refinement iterative method is: setting the total number of iterations; inputting the ship trajectory features with refined states into the graph convolutional network, using the message passing mechanism of the graph convolutional network to transmit the information of neighboring ships to the target ship, and obtaining new ship trajectory features, and at the same time increasing the number of iterations by 1; if the number of iterations is less than the total number of iterations, using the new ship trajectory features as the input of the next iteration, and repeating the above operation until the number of iterations is equal to the total number of iterations.

[0041] The beneficial effects of the present invention are:

[0042] The present invention constructs a method for LSTM multi-ship trajectory prediction based on state reconstruction, which not only solves the problem that the traditional LSTM network can only predict trajectories according to time but cannot predict the mutual influence relationship between multiple ships, but also considers the movement trajectory of the ship itself and the interaction of the trajectories between ships, and can realize trajectory prediction in multi-ship scenarios, with a wide range of applications and good prediction effect; the present invention is improved on the basis of the traditional LSTM network, and the introduced state refinement module adaptively updates the current state of each ship through the message passing mechanism in the graph convolutional network, so that the model can consider the interaction of ship trajectories; the present invention introduces a social information selection module in the message passing mechanism, so that the model can focus on different individuals as well as different features; the present invention mainly considers the relationship between ships, constructs a model, and can better consider the interaction of ship trajectories.

[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The attached figure is a flow chart of LSTM multi-ship trajectory prediction based on state reconstruction.

[0045] Figure 2 The attached figure is a schematic diagram of the LSTM framework based on state reconstruction.

[0046] Figure 3 The attached drawings are schematic diagrams of single-step mode and multi-step mode.

[0047] Figure 4 The attached figure is a schematic diagram of ship selection attention.

[0048] Figure 5 The attached figure is a schematic diagram of the ship motion door selection mode. Specific implementation methods

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] like Figure 1 As shown, the present invention discloses a method for LSTM multi-ship trajectory prediction based on state reconstruction, comprising the following steps:

[0052] S1: Data preprocessing: preprocess the AIS data to be predicted; the AIS data includes: longitude and latitude, time, and ship number information; use interpolation processing method to supplement the missing AIS data and filter to obtain ship track data;

[0053] S2: Feature extraction: The AIS data preprocessed in S1 is input into the LSTM network for feature extraction to obtain ship features including position information and time information;

[0054] S3: State-refined ship features: The ship features initially extracted in S2 are reconstructed into states, and the reconstructed features are input into the LSTM network for trajectory prediction; the trajectories of multiple ships are regarded as a graph using the position information in the predicted ship features, and used as the input of the graph convolutional network; the message passing mechanism in the graph convolutional network is used to extract the mutual features between ships, and finally the ship features after state refinement are obtained through the linear layer;

[0055] S4: Iterative refinement of ship features: The ship trajectory features refined in S3 are further refined using the message passing mechanism of the graph convolutional network;

[0056] S5: Trajectory prediction: The features after iterative state refinement are used as the input of the next LSTM network, input into the input unit of the next LSTM network, and finally output the predicted trajectory.

[0057] The schematic diagram of the LSTM framework based on state reconstruction proposed in the present invention is as follows: Figure 2 shown.

[0058] Preferably, the specific steps of the AIS data preprocessing process are as follows:

[0059] S11, performing difference processing on the AIS data with a short missing time interval;

[0060] S12, AIS data with a long time interval missing is not considered to be a track;

[0061] S13, filtering the data processed by S11 and S12, and filtering out the AIS data of multiple ships in the same time period.

[0062] Preferably, the specific steps in the state refinement are as follows:

[0063] S31: reconstructing the state of the initially extracted features, and using the LSTM network to predict after reconstruction to obtain feature A; the feature A includes: position information and hidden state; the message passing mechanism of the graph convolutional network is introduced in the state reconstruction;

[0064] S32: Using the position information in feature A, the trajectories of multiple ships are considered as a graph and used as the input of the graph convolutional network;

[0065] S33: The mutual features between ships are extracted by using the message passing mechanism in the graph convolutional network, and finally the features after state refinement are obtained through the linear layer; the mutual features include: the hidden state of the LSTM and the attention value of the corresponding ship; the features after state refinement include: the number, position information and hidden state of adjacent ships.

[0066] Preferably, the state reconstruction step first aligns the ships according to the timestamps, and then takes the three information sources of all ships as input, the three information sources including the current position of the ship in the AIS data, the hidden state in the LSTM and the unit state of the LSTM, and outputs the reconstructed unit state; mathematically, the operation for reconstructing the unit state can be expressed as:

[0067]

[0068] in, represents a message passing function, represents the hidden state of the current target ship LSTM, Representatives in Based on the hidden state of the LSTM after one iteration, Represents the current LSTM unit state, t represents time, and l represents the number of iterations.

[0069] Preferably, the message passing mechanism of the graph convolutional network implements the message passing mechanism of the graph convolutional network through message passing and aggregation between nodes;

[0070] The node is a set of feature representations of the ship, and the feature representation includes the attributes or status of the ship;

[0071] The message passing mechanism of the graph convolutional network is implemented by the following steps: First, for each node, the feature representation of its neighboring nodes is passed as a message. The specific expression is as follows:

[0072]

[0073]

[0074] In the formula, |N j | represents the number of elements in N(i), Represents the hidden state of the LSTM of the neighbor node, the message passing function The hidden state of the LSTM does not depend on the target ship W mp is a linear transformation used to transmit a message from a neighboring node j to a target node i;

[0075] Then, a weighted sum aggregation operation is performed on the received messages to obtain a new feature representation of the node.

[0076] Preferably, the message transmission mechanism introduces a social information selection module, including: ship selection attention and ship movement gate, the ship selection attention is to select the number of ships, the ship movement gate is to select the hidden state, and the specific expression is as follows:

[0077]

[0078] Among them, ⊙ represents the dot product operation, Select attention for ships, that is, the attention between ships. is the selection coefficient of the ship motion gate, i.e. the hidden state, is the relative position size in space, is an intermediate variable, and the specific calculation formula is as follows:

[0079]

[0080]

[0081]

[0082]

[0083] Among them, δ represents the sigmoid activation function, φ r Represents the relative position calculation function, Respectively represent the horizontal and vertical coordinates of the target ship and the adjacent ship, W m 、b m , W T , W aT is the corresponding weight parameter, and the overall hidden state size is calculated by the dot product operation in formula (4);

[0084] further, Figure 4 Figure 1 is a schematic diagram of ship selection attention, with specific explanations as follows: the red dots in the figure are target ships, the blue dots are adjacent ships, the arrows represent the direction of travel of the ships, and the size of the magenta circle represents the attention value of the target ship to the adjacent ships. It can be seen that the target ship's attention to the adjacent ships is dominant, and the reverse attention is less obvious.

[0085] further, Figure 5 The following is a schematic diagram of the ship motion gate. Each row in the figure is associated with a hidden neural feature of the LSTM. Gates with the same features are responsible for roughly similar interaction conditions. When predicting the trajectory of ship i (red), the motion gate carefully selects the features of ship j (blue). These selected features shown in the first column represent the potential trajectories of ship j that may interact with ship i in the future. Figure 5 A total of four motion states are shown: 1) First row: the trajectory pair is very close and when driving close together, the selected hidden features follow the walking direction; 2) Second row: the trajectories are somewhat close, but in opposite directions; 3) Third row: This situation is similar to the second row, and the ship considers the farther ship traveling in the opposite direction; 4) Fourth row: The blue ship is static, and the selected hidden features indicate that the red ship i may notice this static neighbor and drive towards it.

[0086] Preferably, the refinement iteration method is as follows: setting the total number of iterations; inputting the ship trajectory features with refined state into the graph convolutional network, using the message passing mechanism of the graph convolutional network to transmit the information of neighboring ships to the target ship, and obtaining new ship trajectory features, and increasing the number of iterations by 1; if the number of iterations is less than the total number of iterations, using the new ship trajectory features as the input of the next iteration, and repeating the above operation until the number of iterations is equal to the total number of iterations;

[0087] Furthermore, a higher number of iterations means that the distance to the surrounding ships is farther. It does not mean that the higher the number of iterations, the better. Ships that are too far away have less mutual influence. Therefore, when setting the total number of iterations, the total number of iterations shall not exceed 5 times.

[0088] like Figure 3 As shown, this embodiment provides a schematic diagram of a single-step mode and a multi-step mode; the single-step mode is to use multiple position points to predict a single position during training, and use the real position to correct the model, such as Figure 3 (a) shows; the multi-step mode is used in the test phase, using multiple known points to predict multiple future position points, such as Figure 3 (b) as shown.

[0089] It should be noted that the above-described specific implementations can enable those skilled in the art to more fully understand the invention, but do not limit the invention in any way. Therefore, although this specification has described the invention in detail with reference to the drawings and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents. In short, all technical solutions and improvements that do not deviate from the spirit and scope of the invention should be included in the protection scope of the patent for the invention.

Claims

1. A method for LSTM multi-ship trajectory prediction based on state reconstruction, It is characterized in that The following steps are involved: S1: Data preprocessing: preprocess the AIS data to be predicted; The AIS data includes: longitude and latitude, time, and ship number information; an interpolation processing method is used to supplement the missing AIS data and filter out the ship track data; S2: Feature extraction: The AIS data preprocessed in S1 is input into the LSTM network for feature extraction to obtain ship features including position information and time information; S3: State-refined ship features: The ship features initially extracted in S2 are reconstructed into states, and the reconstructed features are input into the LSTM network for trajectory prediction; the trajectories of multiple ships are regarded as a graph using the position information in the predicted ship features, and used as the input of the graph convolutional network; the message passing mechanism in the graph convolutional network is used to extract the mutual features between ships, and finally the ship features after state refinement are obtained through the linear layer; S4: Iterative refinement of ship features: The ship trajectory features refined in S3 are further refined using the message passing mechanism of the graph convolutional network; S5: Trajectory prediction: The features after iterative state refinement are used as the input of the next LSTM network, input into the input unit of the next LSTM network, and finally output the predicted trajectory.

2. According to claim 1, a method for LSTM multi-ship trajectory prediction based on state reconstruction, It is characterized in that The specific steps of the AIS data preprocessing process are as follows: S11, performing difference processing on the AIS data with a short missing time interval; S12, AIS data with a long time interval missing is not considered to be a track; S13, filtering the data processed by S11 and S12, and filtering out the AIS data of multiple ships in the same time period.

3. According to claim 1, a method for LSTM multi-ship trajectory prediction based on state reconstruction, It is characterized in that The specific steps of the state refinement are as follows: S31: reconstructing the state of the initially extracted features, and using the LSTM network to predict after reconstruction to obtain feature A; the feature A includes: position information and hidden state; the message passing mechanism of the graph convolutional network is introduced in the state reconstruction; S32: Using the position information in feature A, the trajectories of multiple ships are considered as a graph and used as the input of the graph convolutional network; S33: The mutual features between ships are extracted by using the message passing mechanism in the graph convolutional network, and finally the features after state refinement are obtained through the linear layer; the mutual features include: the hidden state of the LSTM and the attention value of the corresponding ship; the features after state refinement include: the number, position information and hidden state of adjacent ships.

4. According to claim 3, a method for LSTM multi-ship trajectory prediction based on state reconstruction, It is characterized in that The state reconstruction step first aligns the ships according to the timestamps, then takes the three information sources of all ships as input, the three information sources include the current position of the ship in the AIS data, the hidden state in the LSTM and the cell state of the LSTM, and outputs the reconstructed cell state; mathematically, the operation for reconstructing the cell state can be expressed as: Among them, M j represents a message passing function, represents the hidden state of the current target ship LSTM, Representatives in Based on the hidden state of the LSTM after one iteration, Represents the current LSTM unit state, t represents time, and l represents the number of iterations.

5. According to claim 3, a method for LSTM multi-ship trajectory prediction based on state reconstruction, It is characterized in that The message passing mechanism of the graph convolutional network is realized by message passing and aggregation between nodes. The node is a set of feature representations of the ship, and the feature representation includes the attributes or status of the ship; The message passing mechanism of the graph convolutional network is implemented by the following steps: First, for each node, the feature representation of its neighboring nodes is passed as a message. The specific expression is as follows: In the formula, |N j | represents the number of elements in N(i), Represents the hidden state of the LSTM of the neighbor node, the message passing function M j The hidden state of the LSTM does not depend on the target ship W mp is a linear transformation used to transmit a message from a neighboring node j to a target node i; Then, a weighted sum aggregation operation is performed on the received messages to obtain a new feature representation of the node.

6. The method for LSTM multi-ship trajectory prediction based on state reconstruction according to claim 5, It is characterized in that The message transmission mechanism introduces a social information selection module, including: ship selection attention and ship movement gate. The ship selection attention is to select the number of ships, and the ship movement gate is to select the hidden state. The specific expression is as follows: Among them, ⊙ represents the dot product operation, Select attention for ships, that is, the attention between ships. is the selection coefficient of the ship motion gate, i.e. the hidden state, is the relative position size in space, is an intermediate variable, and the specific calculation formula is as follows: Among them, δ represents the sigmoid activation function, φ r Represents the relative position calculation function, Respectively represent the horizontal and vertical coordinates of the target ship and the adjacent ship, W m , b m , W T , W aT is the corresponding weight parameter, and the overall hidden state size is calculated through the dot product operation in formula (4).

7. The method for LSTM multi-ship trajectory prediction based on state reconstruction according to claim 1, It is characterized in that The refinement iteration method is as follows: setting the total number of iterations; inputting the ship trajectory features with refined states into the graph convolutional network, using the message passing mechanism of the graph convolutional network to transmit the information of neighboring ships to the target ship, and obtaining new ship trajectory features, and increasing the number of iterations by 1; if the number of iterations is less than the total number of iterations, using the new ship trajectory features as the input of the next iteration, and repeating the above operation until the number of iterations is equal to the total number of iterations.