A ship autonomous navigation decision-making method for enhancing scene representation learning

Abstract

The application discloses a kind of ship autonomous navigation decision-making methods for enhancing scene representation learning, break through the bottleneck of applying reinforcement learning to the field of ship autonomous navigation. Firstly, the ship navigation state is vectorized representation;Then build a multi-level Transformer network, encode the multi-modal information in the complex navigation scene, learn the hidden layer representation of the navigation scene;Finally, the ship autonomous navigation decision-making method of enhanced reinforcement learning is proposed, the random strategy network and double Q function network are trained simultaneously, and the optimal strategy of ship autonomous navigation is obtained. The application enhances the scene representation, applies the learned interaction perception between the ship and the surrounding ships and the intention perception between the planned route to the ship autonomous navigation decision-making method based on reinforcement learning, which can greatly improve the safety and reliability of ship autonomous navigation decision-making, and promote the development of intelligent ship autonomous navigation technology.

Description

Technical Field

[0001] This invention relates to the field of intelligent ships, and more specifically to a method for ship autonomous navigation decision-making based on enhanced scene representation learning. Background Technology

[0002] The ever-growing market demand for intelligent ships has spurred the rapid development of intelligent ship technology, with autonomous navigation technology being one of the key technologies. Regulations require that ships meeting the intelligent navigation function criteria must be able to achieve autonomous navigation in various navigation scenarios and complex environmental conditions.

[0003] Most current ship autonomous navigation decision-making technologies assume prior knowledge of the environment and lack autonomous learning capabilities, making them prone to error accumulation when facing unknown environments. Reinforcement learning-based decision-making methods, on the other hand, involve the agent (i.e., the ship) interacting with the environment, setting reward and punishment mechanisms, and automatically learning the optimal strategy. Compared to rule-based decision-making methods, reinforcement learning methods can handle various unknown environments and scenarios.

[0004] However, due to the complexity of the maritime navigation environment and encounter scenarios, the state representation of ship navigation should encompass heterogeneous information, such as the encounter situation with surrounding vessels, the ship's own motion characteristics, and various static or dynamic obstructions within the waterway. A good scenario representation is crucial for intelligent agents to better understand complex environments and improve their decision-making capabilities.

[0005] Therefore, in the face of complex maritime navigation environments, how to construct and encode state representations of ship navigation scenarios is a bottleneck in applying reinforcement learning to the field of autonomous navigation of intelligent ships. In view of this, a new technical solution is needed to address the aforementioned problems. Summary of the Invention

[0006] The purpose of this invention is to provide a ship autonomous navigation decision-making method based on enhanced scene representation learning. By enhancing scene representation, the learned interaction perception between the ship and surrounding ships, as well as the intent perception with the planned route, are applied to a reinforcement learning-based decision-making method, which can significantly improve the safety and reliability of ship autonomous navigation decision-making.

[0007] To achieve the above objectives, the ship autonomous navigation decision-making method based on enhanced scene representation learning of the present invention includes the following steps:

[0008] (S1) Vectorized representation of ship navigation state: navigation state input s at time t t Includes the historical movement trajectory of this ship and surrounding ships. and future candidate trajectories

[0009] (S2) Construct a multi-level Transformer network to learn the hidden layer representation of the navigation scene: learn the interaction perception between the ship and surrounding ships, as well as the intent perception with the planned route, and encode multimodal information using a multi-level structure to represent the navigation scene state s t Mapping to the hidden layer representation h t ;

[0010] (S3) Employing an augmented reinforcement learning-based method for autonomous navigation decision-making: Inputting the hidden layer representation h of the navigation scenario t Simultaneously, it learns a stochastic policy network and a double Q-function network to obtain the optimal strategy for autonomous navigation of the ship. Given the navigation state, it outputs the optimal ship maneuvering action. Specifically:

[0011] In step S1, at time t, the state input s t Includes the historical movement trajectory of this ship and surrounding ships. and future candidate trajectories That is, vectorized representation is

[0012] The vectorized representation of the historical motion trajectories of this ship and surrounding ships is as follows:

[0013]

[0014] in, This indicates the ship's historical trajectory. This represents the historical motion trajectories of n ships surrounding this ship. Each historical motion trajectory specifically includes the current time t and the time T before that. h The sequence of motion states at each moment, i.e. Ship motion state m at time t t Specifically, this includes latitude and longitude location points (x t ,y t ), lateral and longitudinal velocities (v) xt ,v yt ), heading ψ t That is, m t =(x t ,y t ,v xt ,v yt ,ψ t ).

[0015] The vectorized representation of the set of future candidate trajectories of this ship and surrounding ships is:

[0016]

[0017] in, This represents the set of candidate trajectories for this ship. This represents the set of candidate trajectories for n ships surrounding our current vessel. Each candidate trajectory set specifically includes a sequence of candidate routes ahead of our current position. Each candidate trajectory is determined by future time T. K A series of waypoints

[0018] Composition. Waypoint k at time t. t Specifically, this includes latitude and longitude location. With heading angle Right now

[0019] In step S2: the ship learns the interaction perception between itself and surrounding vessels, as well as the intent perception between itself and the planned route. A multi-level structure is used to encode multimodal information, and the navigation scenario state s is then processed. t Mapping to the hidden layer representation h t .

[0020] The multi-level Transformer network structure includes a dynamic layer, a cross-modal layer, an aggregation layer, and an output layer.

[0021] The dynamic layer describes the historical movements of this ship and surrounding ships. and candidate paths Encoding is performed. A single-layer Transformer composed of multi-head attention (MHA), global max pooling (MaxPool), and multilayer perceptron (MLP) is used to encode the historical motion trajectories of the ship and surrounding ships on the time axis, outputting a latent representation of the ship's motion state. To differentiate the vessel from surrounding vessels based on its classification features, the vessel's characteristics (Emb) are embedded into a single-layer Transformer consisting of a multi-head attention mechanism (MHA), a fully connected concat layer, and a multilayer perceptron (MLP). The output is a matching latent set of candidate path points. in, For time mask.

[0022] The cross-modal layer is only relevant to surrounding vessels, and the input is a latent representation with motion characteristics obtained from the dynamic layer. Matching latent set of candidate path points and time mask A single-layer Transformer, composed of multi-head attention (MHA), a fully connected layer concat, and a multilayer perceptron (MLP), outputs the cross-modal characteristics of surrounding ships.

[0023] The aggregation layer encodes the ship's historical trajectory. and the cross-mode of surrounding ships The impact on the ship is aggregated, and a single-layer Transformer composed of multi-head attention (MHA), a fully connected layer concat, and a multilayer perceptron (MLP) is used to output an aggregated interactive scene representation Ag. t .

[0024] Add the ship's future path coding to the output layer Ag represents the aggregation and interaction scenario. t Features are utilized by a single-layer Transformer consisting of multi-head attention (MHA), a fully connected layer (concat), and a multilayer perceptron (MLP) to output a latent representation h of the navigation state. t .

[0025] In step S3: an enhanced reinforcement learning method for ship navigation decision-making is proposed.

[0026] The hidden layer representation h obtained from the input multi-level Transformer navigation scene t Simultaneously, it learns a stochastic policy network and a double Q-function network to obtain the optimal strategy for autonomous navigation of the ship. Given the navigation state, it outputs the optimal ship maneuvering action.

[0027] Given the hidden layer representation h t From the experience replay array Medium-sampled Markov arrays:

[0028] τ=(s t ,a t ,r t ,s t+1 ,γ)

[0029] Among them, s t ,a t ,r t The states, actions, and rewards at time t are respectively, and s is the sum of these three states. t+1 Let γ be the state at time t, and γ be the discount rate.

[0030] Using the current policy π sampling action a′, calculate the time difference objective:

[0031]

[0032] Among them, h t+1 For state s t+1 The hidden layer representation obtained through multiple levels of Transformer, Represents state s t+1 The hidden layer representation obtained by dynamically updating the target network through Polyak averaging at each gradient step.

[0033] Next, based on the mean Bellman squared error, minimize the objective:

[0034]

[0035] Update the double Q-function network parameters θ1, θ2. Simultaneously, minimize the soft-Q function. Update the strategy parameter φ.

[0036] Finally, based on the obtained optimal strategy π for autonomous navigation of the ship... φ Given the navigation state, output the optimal ship handling action.

[0037] Compared with existing ship autonomous navigation decision-making methods, the present invention has the following advantages and effects:

[0038] 1. This invention proposes a multi-level Transformer network for encoding information of heterogeneous scene elements, which can effectively learn the interaction perception between the ship and surrounding ships as well as the intent perception between the ship and the planned route, and generate hidden layer representations of the scene for reinforcement learning.

[0039] 2. This invention designs a novel enhanced scene representation learning framework, which can effectively enhance the application capability of reinforcement learning-based decision-making systems in complex maritime navigation scenarios and improve the safety and interpretability of navigation decisions. Attached Figure Description

[0040] Figure 1 This is a framework diagram of a ship autonomous navigation decision-making method based on enhanced scene representation learning according to the present invention.

[0041] Figure 2 The multi-level Transformer algorithm framework provided by this invention is shown in the diagram. Detailed Implementation

[0042] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0043] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0044] Some exemplary embodiments of the invention have been described for illustrative purposes. It should be understood that the invention may be implemented in other ways not specifically shown in the accompanying drawings.

[0045] like Figure 1 , Figure 2 As shown, the present invention provides a ship autonomous navigation decision-making method based on enhanced scene representation learning, the method comprising the following steps:

[0046] Step 1: Vectorize the ship's navigation state.

[0047] At time t, the state input s t Includes the historical movement trajectory of this ship and surrounding ships. and future candidate trajectories Vectorized representation:

[0048]

[0049] The vectorized representation of the historical motion trajectories of this ship and surrounding ships is as follows:

[0050]

[0051] in, This indicates the ship's historical trajectory. This represents the historical motion trajectories of n ships surrounding this ship. Each historical motion trajectory specifically includes the current time t and the time T before that. h Motion state sequence at time:

[0052]

[0053] Wherein, the ship's motion state m at time t t Specifically, this includes latitude and longitude location points (x... t ,y t ), lateral and longitudinal velocities (v) xt ,v yt ), heading ψ t .

[0054] m t =(x t ,y t ,v xt ,v yt ,ψ t (4) The vectorized representation of the set of future candidate trajectories of this ship and surrounding ships is as follows:

[0055]

[0056] in, This represents the set of candidate trajectories for this ship. This represents the set of candidate trajectories for n ships surrounding our current vessel. Each candidate trajectory set specifically includes a sequence of candidate routes ahead of our current position.

[0057] Each candidate trajectory is determined by future time T. K A series of waypoints Composition. Waypoint k at time t. t Specifically, this includes latitude and longitude location. With heading angle

[0058]

[0059] Step 2: Build a multi-level Transformer network to learn the hidden layer representation of the navigation scene.

[0060] The system learns the interaction perception between the ship and surrounding vessels, as well as the intent perception between the ship and the planned route. A multi-level structure is used to encode multimodal information, representing the navigation scenario state s. t Mapping to the hidden layer representation h t .

[0061] The multi-level Transformer network structure includes a dynamic layer, a cross-modal layer, an aggregation layer, and an output layer.

[0062] The dynamic layer describes the historical movements of this ship and surrounding ships. and candidate paths Encoding is performed. A single-layer Transformer composed of multi-head attention (MHA), global max pooling (MaxPool), and multilayer perceptron (MLP) is used to encode the historical motion trajectories of the ship and surrounding ships on the time axis, outputting a latent representation of the ship's motion state:

[0063]

[0064] To extract the temporal characteristics of candidate paths and distinguish the classification features of the ship from surrounding vessels, the ship characteristic Emb is embedded into a single-layer Transformer consisting of a multi-head attention mechanism (MHA), a fully connected concat layer, and a multilayer perceptron (MLP). The output is a matching latent set of candidate path points.

[0065]

[0066] in, Let i be the time mask, i = 0, 1, 2, ..., n.

[0067] The cross-modal layer is only relevant to surrounding vessels, and the input is a latent representation with motion characteristics obtained from the dynamic layer. Matching latent set of candidate path points and time mask A single-layer Transformer, composed of multi-head attention (MHA), a fully connected layer (concat), and a multilayer perceptron (MLP), outputs the cross-modal characteristics of surrounding ships:

[0068]

[0069] The aggregation layer encodes the ship's historical trajectory. and the cross-mode of surrounding ships The impact on the ship is aggregated, and a single-layer Transformer composed of multi-head attention (MHA), a fully connected layer concat, and a multilayer perceptron (MLP) is used to output an aggregated interactive scene representation:

[0070]

[0071] Add the ship's future path coding to the output layer Ag represents the aggregation and interaction scenario. t Features, utilizing a single-layer Transformer composed of multi-head attention (MHA), a fully connected concat layer, and a multilayer perceptron (MLP), output a latent representation of the navigation state:

[0072]

[0073] Step 3: Propose a method to enhance and reinforce learning-based autonomous navigation decision-making for ships.

[0074] The hidden layer representation h of the input multi-level Transformer navigation scene t Simultaneously, it learns a stochastic policy network and a double Q-function network to obtain the optimal strategy for autonomous navigation of the ship. Given the navigation state, it outputs the optimal ship maneuvering action.

[0075] Given the hidden layer representation h t From the experience replay array Medium-sampled Markov arrays:

[0076] τ=(s t ,a t ,r t ,s t+1 ,γ) (12)

[0077] Among them, s t ,a t ,r t The states, actions, and rewards at time t are respectively: s t+1 Let γ be the state at time t, and γ be the discount rate.

[0078] Calculate the temporal difference objective using the sampling action of the current policy:

[0079]

[0080] Where a′ represents the action sampled from the current policy π, h t+1 For state s t+1 The hidden layer representation obtained through multiple levels of Transformer, Represents state s t+1 The hidden layer representation obtained by dynamically updating the target network through Polyak averaging at each gradient step.

[0081] Next, based on the mean Bellman squared error, minimize the objective:

[0082]

[0083] Update the network parameters θ1 and θ2 of the double Q function.

[0084] Simultaneously, minimized via soft-Q:

[0085]

[0086] Update the strategy parameter φ.

[0087] Finally, based on the obtained optimal strategy π for autonomous navigation of the ship... φ Given the navigation state, output the optimal ship handling action.

[0088] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for ship autonomous navigation decision-making based on enhanced scene representation learning, characterized in that, The method includes the following steps: (S1) Vectorized representation of ship navigation state: Navigation status input at any time Includes the historical movement trajectory of this ship and surrounding ships. and future candidate trajectories ; (S2) Construct a multi-level Transformer network to learn the hidden layer representation of the navigation scene: learn the interaction perception between the ship and surrounding ships, as well as the intent perception with the planned route, and encode multimodal information using a multi-level structure to represent the navigation scene state. Mapping to hidden layer representation ; (S3) Employing reinforcement learning-based autonomous navigation decision-making methods for ships: inputting the hidden layer representation of the navigation scenario. Simultaneously, it learns a stochastic policy network and a double Q-function network to obtain the optimal strategy for autonomous navigation of the ship. Given the navigation state, it outputs the optimal ship maneuvering action. The multi-level structure of the multi-level Transformer network includes a dynamic layer, a cross-modal layer, an aggregation layer, and an output layer. In the dynamic layer of a multi-level Transformer network, a separate temporal Transformer is used to analyze the historical motions of the ship and surrounding vessels. and candidate paths Encode the output and ,in, ; In the cross-modal layer of the multi-level Transformer network, the motion states of surrounding ships are abstracted, and the cross-modal data of surrounding ships are output. ; In the aggregation layer of the multi-level Transformer network, the ship's historical trajectory is encoded. and the cross-mode of surrounding ships Aggregate the impacts on this ship and output an aggregated interactive scenario representation. ; Add the ship's future path encoding to the output layer of the multi-level Transformer network. and aggregation interaction scenario representation Features, the underlying representation of the output navigation state .

2. The ship autonomous navigation decision-making method based on enhanced scene representation learning according to claim 1, characterized in that: Historical movement trajectories of this ship and surrounding vessels Represented as ,in, This indicates the ship's historical trajectory. Indicates the area around this ship The historical motion trajectory of each ship; each historical motion trajectory specifically includes the current moment. Compared to the past Motion state sequence at time 1 Motion state at a single moment Specifically, this includes the ship's latitude and longitude coordinates. Lateral and longitudinal velocities ( ,course .

3. The ship autonomous navigation decision-making method based on enhanced scene representation learning according to claim 1, characterized in that: Set the future candidate trajectories of this ship and surrounding ships Represented as ,in, This represents the set of candidate trajectories for this ship. Indicates the area around this ship A set of candidate trajectories for each vessel; each candidate trajectory set specifically includes a sequence of candidate routes ahead of the current position. Each candidate trajectory is determined by future time. A series of waypoints on the way Waypoints at a single moment Specifically, this includes latitude and longitude coordinates. With heading angle .

4. The ship autonomous navigation decision-making method based on enhanced scene representation learning according to claim 1, characterized in that: In a dual-Q function network, given the hidden layer representation From the experience replay array Medium-sampled Markov arrays ,in, They are respectively The state, actions, and rewards at any given moment The state at any given moment. The discount rate; calculate the time difference target using the sampling action of the current strategy. Finally, based on the mean Bellman squared error, the target is minimized. This leads to the updating of the double Q function network parameters. ; The time difference objective in the double Q function network Represented as: ； in, Indicates the current strategy The sampling action, For state Hidden layer representation obtained through multi-level Transformer Representing state The hidden layer representation obtained by dynamically updating the target network through Polyak averaging at each gradient step.

5. The ship autonomous navigation decision-making method based on enhanced scene representation learning according to claim 4, characterized in that: In a stochastic policy network, soft-Q minimization is used. Update strategy parameters .

Images