Hierarchical ship trajectory prediction method, device and product based on deep reinforcement learning

Through the multi-level multi-agent reinforcement learning method, combined with deep reinforcement learning and meta-evaluation network, the local optimization problem of ship trajectory prediction algorithms in the existing technology during dynamic adjustment of offshore tasks and the problem of insufficient universality of multi-agent strategies. Improves the versatility and robustness of maritime strategies.

CN118798310BActive Publication Date: 2024-11-26TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411288434.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-26
Estimated Expiration
2044-09-14

AI Technical Summary

Technical Problem

Existing ship trajectory prediction algorithms based on deep reinforcement learning are prone to catastrophic forgetting when dynamic adjustment of maritime tasks, resulting in decision-making convergence to local optimality. Multi-agent strategies are not universal in different maritime situations and are less robust. .

Method used

Using multi-level multi-agent reinforcement learning method for multi-tasks, each agent learns strategies for a single maritime task through the first stage of deep reinforcement learning, and conducts game task network training based on the joint state and actions of historical tasks. Obtain the trajectory prediction information of the next maritime task and learn a general strategy through the meta-evaluation network.

Benefits of technology

It improves the universality of maritime strategies in different maritime situations and the robustness of decision-making models, ensuring that each agent can perform new joint actions in new tasks and achieve better ship trajectory prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118798310B_ABST
    Figure CN118798310B_ABST
Patent Text Reader

Abstract

The present invention provides a hierarchical ship trajectory prediction method, device and product based on deep reinforcement learning, which relates to the field of reinforcement learning. It includes: for maritime tasks, N intelligent agents perform deep reinforcement learning based on sample maritime situation information with the goal of maximizing total rewards; the joint state, joint action and total reward of each maritime task that has been executed in the history are input into the game task network to obtain the trajectory prediction information of the next maritime task; the current maritime situation information is input into the N intelligent agents after deep reinforcement learning, the joint state and joint action of the current maritime task are obtained and input into the basic evaluation network to obtain the Q value of the current maritime task; the trajectory prediction information of the next maritime task and the Q value of the current maritime task are input into the meta-evaluation network to obtain the new joint action of the N intelligent agents based on the new predicted trajectory, so as to improve the versatility of the strategy under different maritime situations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of reinforcement learning, and in particular to a hierarchical ship trajectory prediction method, device and product based on deep reinforcement learning. Background Art

[0002] Most of the current research on ship trajectory prediction algorithms based on deep reinforcement learning is a single-model reinforcement structure. When the maritime mission is dynamically adjusted, catastrophic forgetting problems occur. The current maritime mission experience completely covers the learning experience of previous missions, which ultimately leads to the decision-making converging to the local optimum rather than the global optimum.

[0003] Based on this, related technologies have proposed a multi-agent reinforcement learning ship trajectory prediction algorithm that is more suitable for the changing maritime environment. However, since the current multi-agent strategy is mainly trained for a single task, the learned model has weak robustness and lacks versatility under different maritime situations. Summary of the invention

[0004] Based on the above technical problems, the embodiments of the present invention provide a hierarchical ship trajectory prediction method, device and product based on deep reinforcement learning, aiming to improve the versatility of maritime strategies in different maritime situations through a multi-level and multi-agent reinforcement learning method for multiple tasks.

[0005] A first aspect of an embodiment of the present invention provides a hierarchical ship trajectory prediction method based on deep reinforcement learning, the method comprising:

[0006] For the i-th maritime task in the maritime task set, N agents perform the first stage of deep reinforcement learning based on the sample maritime situation information of the i-th maritime task with the goal of maximizing the total reward R. At the end of the first stage of deep reinforcement learning, the N agents each learn a strategy for the i-th maritime task, and the strategy of each agent for the i-th maritime task is: perform action a according to the predicted trajectory in state s, and the reward is r; wherein the total reward R is the sum of the rewards r of the N agents, and the N agents represent N ships;

[0007] The joint state, joint action and total reward of each maritime task that has been executed in the past are input into the game task network to obtain the trajectory prediction information of the next maritime task; the joint action is the sum of the actions a executed by each of the N agents according to the historical trajectory, and the joint state is the sum of the states s corresponding to each of the N agents;

[0008] Input the current maritime situation information into the N agents that have undergone the first stage of deep reinforcement learning to obtain the joint state and joint action of the current maritime task, and input the joint state and joint action of the current maritime task into the basic evaluation network to obtain the Q value of the current maritime task;

[0009] Inputting the trajectory prediction information of the next maritime task and the Q value of the current maritime task into the meta-evaluation network to obtain a new joint action for each maritime task in the maritime task set; the new joint action is the sum of the new actions a performed by each of the N agents according to the new predicted trajectory;

[0010] Based on the new joint action of the current maritime mission, the N agents each perform their own new actions according to the new predicted trajectory.

[0011] Optionally, the performing a first phase of deep reinforcement learning based on the sample maritime situation information of the i-th maritime mission includes:

[0012] Based on the OARE model, the sample maritime situation information of the i-th maritime mission is represented as a sample OARE vector;

[0013] Inputting the sample OARE vector into the trained encoder to obtain the joint state of the i-th marine mission;

[0014] The joint state of the i-th maritime mission is distributed to the N agents to perform the first stage of deep reinforcement learning.

[0015] Optionally, the encoder is trained by the following steps:

[0016] Based on the OARE model, the maritime situation information at multiple moments is represented as multiple OARE vectors;

[0017] Inputting the plurality of OARE vectors into an encoder to be trained, and compressing the plurality of OARE vectors into compressed OARE vectors of a specified length;

[0018] Reconstructing data using the compressed OARE vector to obtain a reconstructed OARE vector;

[0019] The parameters of the encoder to be trained are updated according to the reconstructed OARE vector and the multiple OARE vectors to obtain the trained encoder.

[0020] Optionally, the i-th marine task in the marine task set is any upper-level task in an upper-level task layer;

[0021] Any upper-layer task is composed of multiple lower-layer tasks in the lower-layer task layer;

[0022] Any bottom-level task in the bottom-level task layer is composed of a plurality of basic actions in the basic action layer, and the basic action layer includes at least one of the following: action distance, action direction;

[0023] Any basic action in the basic action layer is an action a performed by any agent in state s according to the predicted trajectory.

[0024] Optionally, the game task network is trained by the following steps:

[0025] The sample joint states, sample joint actions and sample total rewards of multiple historical sample sea tasks are input into the game task network to be trained to obtain the trajectory prediction information of the next sample sea task;

[0026] Based on the trajectory prediction information of the next sample sea mission and the strategies of the actual sample sea missions after the multiple historical sample sea missions, the parameters of the game task network to be trained are updated to obtain the game task network.

[0027] Optionally, the basic evaluation network is trained by the following steps:

[0028] Based on the sample joint state and the sample joint action of the sample sea task, an actual Q value corresponding to the sample sea task is obtained; the actual Q value represents the decision level for the sample sea task;

[0029] Inputting the sample joint state and the sample joint action of the sample marine task into the basic evaluation network to be trained to obtain the predicted Q value corresponding to the sample marine task;

[0030] Based on the predicted Q value and the actual Q value, the parameters of the basic evaluation network to be trained are updated to obtain the basic evaluation network.

[0031] Optionally, the meta-evaluation network is trained by the following steps:

[0032] Based on the sample joint state and sample joint action of the historical sample sea mission, obtaining the sample Q value corresponding to the historical sample sea mission;

[0033] Based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission, an actual sample Q value corresponding to the historical sample sea mission is obtained;

[0034] Inputting the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission into the meta-evaluation network to be trained, and obtaining the predicted sample Q value corresponding to the historical sample sea mission;

[0035] Based on the predicted sample Q value and the actual sample Q value, the parameters of the meta-evaluation network to be trained are updated to obtain the meta-evaluation network.

[0036] A second aspect of an embodiment of the present invention provides a hierarchical ship trajectory prediction device based on deep reinforcement learning, the device comprising:

[0037] A deep reinforcement learning module is used for, for the i-th maritime task in the maritime task set, N intelligent agents take maximizing the total reward R as the goal, based on the sample maritime situation information of the i-th maritime task, to perform the first stage of deep reinforcement learning. At the end of the first stage of deep reinforcement learning, the N intelligent agents each learn a strategy for the i-th maritime task, and the strategy of each intelligent agent for the i-th maritime task is: perform action a according to the predicted trajectory in state s, and the reward is r; wherein the total reward R is the sum of the rewards r of the N intelligent agents, and the N intelligent agents represent N ships;

[0038] The task decision module is used to input the joint state, joint action and total reward of each maritime task that has been executed in the past into the game task network to obtain the trajectory prediction information of the next maritime task; the joint action is the sum of the actions a executed by each of the N intelligent agents, and the joint state is the sum of the states s corresponding to each of the N intelligent agents;

[0039] A basic evaluation module is used to input the current maritime situation information into the N agents that have undergone the first stage of deep reinforcement learning to obtain the joint state and joint action of the current maritime task, and input the joint state and joint action of the current maritime task into the basic evaluation network to obtain the Q value of the current maritime task;

[0040] A meta-evaluation module is used to input the trajectory prediction information of the next maritime task and the Q value of the current maritime task into a meta-evaluation network to obtain a new joint action for each maritime task in the maritime task set; the new joint action is the sum of the new actions a performed by each of the N agents according to the new predicted trajectory;

[0041] The marine action execution module is used for the new joint action based on the current marine task, and each of the N intelligent agents executes their own new action according to the new predicted trajectory.

[0042] Optionally, the deep reinforcement learning module includes:

[0043] An OARE representation module is used to represent the sample maritime situation information of the i-th maritime mission as a sample OARE vector based on the OARE model;

[0044] An encoding module, used for inputting the sample OARE vector into a trained encoder to obtain the joint state of the i-th marine mission;

[0045] A state distribution module is used to distribute the joint state of the i-th maritime task to the N agents to perform the first stage of deep reinforcement learning.

[0046] Optionally, the device further comprises: an encoder training module, configured to train the encoder, wherein the encoder training module comprises:

[0047] A sample OARE representation module, used for representing the maritime situation information at multiple moments as multiple OARE vectors based on the OARE model;

[0048] A compression module, used for inputting the plurality of OARE vectors into an encoder to be trained, and compressing the plurality of OARE vectors into a compressed OARE vector of a specified length;

[0049] A reconstruction module, used for reconstructing data using the compressed OARE vector to obtain a reconstructed OARE vector;

[0050] The encoding training module is used to update the parameters of the encoder to be trained according to the reconstructed OARE vector and the multiple OARE vectors to obtain the trained encoder.

[0051] Optionally, the i-th marine task in the marine task set is any upper-level task in an upper-level task layer;

[0052] Any upper-layer task is composed of multiple lower-layer tasks in the lower-layer task layer;

[0053] Any bottom-level task in the bottom-level task layer is composed of a plurality of basic actions in the basic action layer, and the basic action layer includes at least one of the following: action distance, action direction;

[0054] Any basic action in the basic action layer is an action a performed by any agent in state s according to the predicted trajectory.

[0055] Optionally, the device further comprises: a game task training module, which is used to train the game task network, and the game task training module comprises:

[0056] The sample task decision module is used to input the sample joint states, sample joint actions and sample total rewards of multiple historical sample sea tasks into the game task network to be trained to obtain the trajectory prediction information of the next sample sea task;

[0057] The game task network training module is used to update the parameters of the game task network to be trained based on the trajectory prediction information of the next sample sea task and the strategies of the actual sample sea tasks after the multiple historical sample sea tasks to obtain the game task network.

[0058] Optionally, the device further comprises: a basic evaluation training module, which is used to train the basic evaluation network, and the basic evaluation training module comprises:

[0059] A first determination module is used to obtain an actual Q value corresponding to the sample marine task based on a sample joint state and a sample joint action of the sample marine task; the actual Q value represents a decision level for the sample marine task;

[0060] A first prediction module is used to input the sample joint state and the sample joint action of the sample marine task into the basic evaluation network to be trained to obtain a predicted Q value corresponding to the sample marine task;

[0061] The basic evaluation network training module is used to update the parameters of the basic evaluation network to be trained based on the predicted Q value and the actual Q value to obtain the basic evaluation network.

[0062] Optionally, the device further comprises: a meta-evaluation training module, used for training the meta-evaluation network, the meta-evaluation training module comprising:

[0063] A second determination module is used to obtain a sample Q value corresponding to the historical sample marine mission based on the sample joint state and sample joint action of the historical sample marine mission;

[0064] A third determination module is used to obtain an actual sample Q value corresponding to the historical sample sea mission based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission;

[0065] The second prediction module is used to input the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission into the meta-evaluation network to be trained, so as to obtain the predicted sample Q value corresponding to the historical sample sea mission;

[0066] The meta-evaluation network training module is used to update the parameters of the meta-evaluation network to be trained based on the predicted sample Q value and the actual sample Q value to obtain the meta-evaluation network.

[0067] A third aspect of an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, a hierarchical ship trajectory prediction method based on deep reinforcement learning as described in the first aspect of an embodiment of the present invention is implemented.

[0068] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the hierarchical ship trajectory prediction method based on deep reinforcement learning of the first aspect of an embodiment of the present invention is implemented.

[0069] Through the hierarchical ship trajectory prediction method based on deep reinforcement learning of the embodiment of the present invention, first, in the first stage, with the goal of maximizing the total reward of each maritime task, deep reinforcement learning of multiple maritime tasks and multiple agents is carried out, so that at the end of deep reinforcement learning, multiple agents can learn strategies for each maritime task respectively, wherein the strategy of each agent for each maritime task is: perform action a according to the predicted trajectory under state s, and the reward is r, and N agents represent N ships; thereby learning the joint state, joint action and total reward corresponding to each maritime task; then, the various maritime tasks that have been executed historically and obtained by multiple agents through the first stage of deep reinforcement learning are combined into a single entity. The joint state, joint action and total reward of the task are input into the game task network to obtain the trajectory prediction information of the next maritime task; and the current maritime situation information is input into multiple intelligent agents that have undergone the first stage of deep reinforcement learning to obtain the joint state and joint action of the current maritime task, and the joint state and joint action of the current maritime task are input into the basic evaluation network to obtain the Q value of the current maritime task; finally, the trajectory prediction information of the next maritime task and the Q value of the current maritime task are input into the meta-evaluation network to obtain a new joint action for each maritime task in the maritime task set, and N intelligent agents perform their own new actions according to the new predicted trajectory based on the new joint action of the current maritime task.

[0070] In this implementation, through multi-task and multi-agent deep reinforcement learning, each ship representation agent has multiple types of maritime tasks to handle, so it learns the strategy of a single maritime task, and then based on the strategies of multiple historical maritime tasks and the strategy Q value of the current maritime task, a general strategy is learned through multi-task meta-learning integration of the meta-evaluation network, and new joint actions for each maritime task are learned to complete the prediction of the maritime ship trajectory. Based on the new joint action, each agent performs its own new action according to the new predicted trajectory, which not only improves the versatility of the maritime strategy in different maritime situations, but also improves the robustness of the maritime decision-making model. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0072] Figure 1 It is a schematic diagram of a multi-agent reinforcement learning framework shown in the related art;

[0073] Figure 2 is a flowchart of a hierarchical ship trajectory prediction method based on deep reinforcement learning according to an embodiment of the present invention;

[0074] Figure 3 It is a processing flow diagram of a multi-task multi-level multi-agent reinforcement learning method shown in one embodiment of the present invention;

[0075] Figure 4 It is a structural block diagram of a hierarchical ship trajectory prediction device based on deep reinforcement learning provided by an embodiment of the present invention;

[0076] Figure 5 It is a schematic diagram of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] At present, the implementation process of decision-making problems usually includes four steps: collecting situation information and representing information features, strengthening learning training, making decisions, taking actions and providing feedback, forming an OODA loop. The OODA loop is a decision-making theory that includes four basic elements:

[0079] Observation: This is the process of gathering and analyzing information about the current situation and environment, including identifying and understanding key data, intelligence, trends and patterns, and assessing actions and intentions.

[0080] Orientation: Analyze and interpret observed information to develop an understanding of the situation. This includes identifying patterns, generating hypotheses, evaluating possible solutions, and making decisions.

[0081] Decision: Based on the analysis results of the judgment phase, choose the best course of action. This includes setting goals, action plans, optimizing resource allocation, and conducting risk assessments.

[0082] Action: Based on the decisions that have been made, take specific actions to implement them. This includes executing plans, adjusting actions, collecting feedback, and making appropriate adjustments based on the situation.

[0083] The core of the OODA loop theory is to adapt quickly and flexibly to complex environments. It emphasizes how to gain advantages in dynamic and complex environments (such as maritime missions) through continuous cycles of observation, judgment, decision-making and action. Victory belongs to those individuals or organizations that can complete this cycle fastest and best.

[0084] The multi-intelligence reinforcement learning strategy of related technologies mainly focuses on the reinforcement learning part, which is the process of processing and judging information, and mainly belongs to the judgment (Orientation) and decision (Decision) parts in the OODA loop. Figure 1 , Figure 1 is a schematic diagram of a multi-agent reinforcement learning framework shown in the related art. Figure 1 As shown in the figure, each intelligent agent learns and improves its individual and group decision-making ability in the continuous interaction with the environment. Multiple intelligent agents have the ability to interact with each other, that is, each intelligent agent can reach a "tacit understanding" through a collaborative mechanism. In the process of reinforcement learning, not only its own interests are considered, but also how to maximize the overall interests. Figure 1 In the example, the action set of each agent is {a 1 , a 2 ,…,a n} form a joint action A; the state set {s 1 ,s 2 ,…,sn} form a joint state S; the reward set {r 1 , r 2 ,…,r n} constitute the total return R.

[0085] The entire reinforcement learning process maximizes the total reward R and learns the joint optimal strategy. By setting the reward value and the total threshold, multiple agents are prompted to complete a task together. The task process includes cooperation between one's own side and confrontation with the other side. Figure 1 The coordination mechanism in is responsible for controlling the coordination mode between intelligent agents. The automatic encoder can transform the situation information (i.e. Figure 1 The environment in the environment is encoded into feature information as observation values ​​for multiple agents.

[0086] However, the current multi-agent reinforcement learning algorithms have the following main problems: 1. Situation information is uniformly processed through the autoencoder, and the semantic representation is insufficient. This is because the current information encoding only extracts the main representation of the environment through the autoencoder, and then manually distributes it in chronological order. The granularity is coarse and non-semantic; 2. The current multi-agent strategy is not universal enough in different situations: This is because the current multi-agent deep reinforcement learning method is trained for a single task, and the learned model has weak robustness and is difficult to quickly adapt to different game tasks.

[0087] Therefore, in order to at least partially solve the above-mentioned problems and one or more of other potential problems, an embodiment of the present invention proposes a hierarchical ship trajectory prediction method based on deep reinforcement learning. In this method, the ship is characterized by an agent. Through multi-task and multi-agent deep reinforcement learning, each agent has multiple types of maritime tasks to handle, so as to learn the strategy of a single maritime task. Then, based on the strategies of multiple historical maritime tasks and the Q value of the current maritime task, a general strategy is learned through multi-task meta-learning integration of a meta-evaluation network, and a new joint action for each maritime task is learned to complete the prediction of the maritime ship trajectory. Based on the new joint action, each agent performs its own new action according to the new predicted trajectory, thereby not only improving the versatility of the maritime strategy under different maritime situations, but also improving the robustness of the maritime decision-making model.

[0088] Please refer to Figure 2 , Figure 2 FIG. 1 is a flowchart of a hierarchical ship trajectory prediction method based on deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the method may include steps S11 to S15:

[0089] Step S11: For the ith maritime mission in the maritime mission set, N agents perform the first stage of deep reinforcement learning based on the sample maritime situation information of the ith maritime mission with the goal of maximizing the total reward R. At the end of the first stage of deep reinforcement learning, the N agents each learn a strategy for the ith maritime mission, and the strategy of each agent for the ith maritime mission is: perform action a according to the predicted trajectory in state s, and the reward is r; wherein the total reward R is the sum of the rewards r of the N agents, and the N agents represent N ships.

[0090] In this embodiment, multi-task multi-agent deep reinforcement learning training is first performed. The maritime task set of this embodiment includes multiple maritime tasks, and the maritime tasks can be maritime transportation tasks related to maritime transportation tasks, etc. In addition, this embodiment has multiple agents, that is, there are N agents, and the N agents represent N ships. For each maritime task in the maritime task set, multiple agents aim to maximize the total reward, and based on the sample maritime situation information of each maritime task, the first stage of deep reinforcement learning is performed. Multiple agents make corresponding actions through communication and coordination mechanisms, and then jointly feedback to the maritime environment as a joint action, that is, affecting the maritime situation information. Then, the maritime environment rewards each agent according to the actions of each agent and the reward mechanism, and obtains the final total reward. At the end of the first stage of deep reinforcement learning, multiple agents can each learn the strategy of each maritime task.

[0091] For example, for the i-th maritime task in the maritime task set, i is an arbitrary value, and N agents perform the first stage of deep reinforcement learning based on the sample maritime situation information of the i-th maritime task with the goal of maximizing the total reward R. At the end of the first stage of deep reinforcement learning, the N agents can each learn a strategy for the i-th maritime task, where the strategy of each agent for the i-th maritime task is: perform action a according to the predicted trajectory under state s, and the reward is r. The strategies of each agent for the i-th maritime task constitute a joint strategy for the i-th maritime task, and the joint strategy includes: perform joint action A according to the joint predicted trajectory under the joint state S, and the total reward is R. It can be understood that the joint action A is the sum of the actions a performed by each agent, the joint state S is the sum of the states s corresponding to each agent, the total reward R is the sum of the rewards r of each agent, and the joint predicted trajectory is the sum of the predicted trajectories corresponding to each agent.

[0092] Step S12: Input the joint state, joint action and total reward of each maritime task that has been executed historically into the game task network to obtain the trajectory prediction information of the next maritime task; the joint action is the sum of the actions a executed by each of the N intelligent agents according to the historical trajectory, and the joint state is the sum of the states s corresponding to each of the N intelligent agents.

[0093] In this embodiment, after completing the first stage of deep reinforcement learning, the N agents that have undergone the first stage of deep reinforcement learning can be used to predict strategies for maritime tasks. In this embodiment, the maritime situation information of multiple maritime tasks can be input into the N agents that have undergone the first stage of deep reinforcement learning in chronological order, and the joint state, joint action and total reward of multiple maritime tasks can be obtained respectively. In this embodiment, the joint state, joint action and total reward of the maritime tasks obtained by these N agents that have undergone the first stage of deep reinforcement learning in chronological order based on the maritime situation information are: the joint state, joint action and total reward of each maritime task that has been executed historically.

[0094] This embodiment prepares a game task network in advance, which is used to encode historical maritime task decisions and predict specific maritime tasks. This embodiment can input the joint state, joint action and total reward of each maritime task that has been executed in the past into the game task network to obtain the trajectory prediction information of the next maritime task. Among them, the joint action is the sum of the actions a performed by each of the N agents according to the historical trajectory, the joint state is the sum of the states s corresponding to each of the N agents, and the total reward is the sum of the rewards r of each of the N agents. In this embodiment, the input of the relevant decisions of each historical maritime task is to learn from history and assist in the decision-making of new maritime tasks.

[0095] Step S13: Input the current maritime situation information into the N intelligent agents that have undergone the first stage of deep reinforcement learning to obtain the joint state and joint action of the current maritime mission, and input the joint state and joint action of the current maritime mission into the basic evaluation network to obtain the Q value of the current maritime mission.

[0096] In this embodiment, the current maritime situation information can be input into the N agents that have undergone the first stage of deep reinforcement learning to obtain the joint state and joint action of the current maritime mission. Among them, the current maritime situation information is the maritime situation information corresponding to the current maritime mission, and the current maritime mission is a maritime mission that requires maritime strategy prediction. Among them, the maritime situation information of this embodiment includes but is not limited to: environmental information, personnel information, equipment information, etc. After obtaining the joint state and joint action of the current maritime mission, the joint state and joint action of the current maritime mission can be input into the basic evaluation network to obtain the Q value of the current maritime mission.

[0097] Among them, this embodiment prepares a basic evaluation network in advance, which is used to record the joint state-joint action value of the current maritime task, evaluate the strategy of the current maritime task output by N intelligent agents, and obtain the Q value of the current maritime task, which represents the decision-making level.

[0098] Step S14: Input the trajectory prediction information of the next maritime task and the Q value of the current maritime task into the meta-evaluation network to obtain a new joint action for each maritime task in the maritime task set; the new joint action is the sum of the new actions a performed by each of the N agents according to the new predicted trajectory.

[0099] In this embodiment, the meta-evaluation network is used to extract general knowledge in the maritime mission based on the game history task information and the current maritime situation information, and obtain the common knowledge of the agent completing the task contained in the game confrontation data under different maritime tasks. In other words, the meta-evaluation network of this embodiment is to extract the commonality of similar maritime tasks to avoid retraining the network when new maritime tasks come. After obtaining the trajectory prediction information of the next maritime mission and the Q value of the current maritime mission, the trajectory prediction information of the next maritime mission and the Q value of the current maritime mission can be input into the meta-evaluation network, and the strategy of each maritime mission in the maritime mission set is updated, and at least a new joint action of each maritime mission in the maritime mission set is obtained; wherein the new joint action is the sum of the new actions a performed by each of the N agents according to the new predicted trajectory.

[0100] Step S15: According to the new joint action of the current maritime mission, the N agents each perform their own new actions according to the new predicted trajectory.

[0101] In this embodiment, after obtaining the new joint action of each sea task in the sea task set, the new joint action of each sea task in the sea task set includes the new joint action of the current sea task. Based on the new joint action of the current sea task, N intelligent agents can each execute their own new actions according to the new predicted trajectory to complete the action for the current sea task.

[0102] In an optional implementation, after generating new actions corresponding to each of the N intelligent agents, the N intelligent agents then predict their trajectories according to their corresponding new actions (such as turn left, turn left, go straight, accelerate, etc.) and their corresponding position information (i.e., ship position information), and obtain the predicted trajectories corresponding to each of the N intelligent agents, thereby realizing the trajectory prediction for each of the N ships in the current maritime mission.

[0103] In this embodiment, through multi-task and multi-agent deep reinforcement learning, each agent has multiple types of maritime tasks to handle in order to learn the strategy of a single maritime task, and then based on the strategies of multiple historical maritime tasks and the Q value of the current maritime task, a general strategy is learned through the multi-task meta-learning integration of the meta-evaluation network, and new joint actions for each maritime task are learned to complete the prediction of the maritime ship trajectory. Based on the new joint actions, each agent performs its own new actions according to the new predicted trajectory, which not only improves the versatility of the maritime strategy in different maritime situations, but also improves the robustness of the maritime decision-making model.

[0104] In combination with the above embodiments, in one implementation, the present invention further provides a hierarchical ship trajectory prediction method based on deep reinforcement learning. In the method, the "based on the sample maritime situation information of the i-th maritime mission, performing the first stage of deep reinforcement learning" in the above step S11 may specifically include steps S21 to S23:

[0105] Step S21: Based on the OARE model, the sample maritime situation information of the i-th maritime mission is represented as a sample OARE vector.

[0106] In this embodiment, the OARE model is a target-attribute-relationship-environment model. The OARE model can be defined by a four-tuple: . Where O is a finite set of objects consisting of several different objects: , for example, in a maritime environment, O can be a ship, etc. A is a finite set of attributes that describe and characterize the object's unique attributes: , the attributes of an object can be any information used to describe and distinguish the characteristics of the object, such as the physical, chemical, and semantic attributes of an object, or their classification information, characteristics, and size, or the sensory information that an object gives people, etc. For example, in a marine environment, A can be the latitude and longitude of a ship, its heading, etc. R is a finite set of relations consisting of several relations: This set of relationships is the key and core of the OARE model that distinguishes it from general concept representation. R also contains three types of relationships, namely: the relationship between Object and Attribute, the relationship between Attribute, and the relationship between Object and other Objects. For example, in the marine environment, R can be cooperation, assistance, etc. For example, the relationship between Attributes can be the relationship between the speed and heading of a ship. E is the environment, which is the environment in which the object is located, for example, E is the marine environment.

[0107] In this embodiment, based on the OARE model, the sample maritime situation information of the i-th maritime mission can be represented in the form of OARE to obtain a sample OARE vector, which is an OARE semantic feature vector of the sample maritime situation information.

[0108] Step S22: Input the sample OARE vector into the trained encoder to obtain the joint state of the i-th marine mission.

[0109] In this embodiment, after obtaining the sample OARE vector, the sample OARE vector representing the semantic feature may be input into the trained encoder to obtain the joint state S of the i-th marine mission.

[0110] Step S23: assigning the joint state of the i-th sea mission to the N agents, and performing the first stage of deep reinforcement learning.

[0111] In this embodiment, after obtaining the joint state S of the i-th marine task, the joint state S of the i-th marine task can be divided into N states s according to task classification (such as according to the time information of the task). i , and then N states s i Assign N agents to carry out the first stage of deep reinforcement learning, so that during the first stage of deep reinforcement learning, each agent can make its own actions according to its assigned state s and get its corresponding reward.

[0112] In this embodiment, a semantic representation method based on OARE is proposed, in which the maritime situation information is first represented by the OARE model using OARE semantic features. In this way, the representation accuracy can be improved during the encoding process, thereby solving the problem of insufficient semantic representation of maritime situation information.

[0113] In combination with the above embodiments, in one implementation, the present invention further provides a hierarchical ship trajectory prediction method based on deep reinforcement learning, in which the encoder can be trained by the following steps:

[0114] Step S31: Based on the OARE model, the maritime situation information at multiple moments is represented as multiple OARE vectors.

[0115] In this embodiment, in the training process summary for the encoder, the maritime situation information at multiple moments may be used, and then based on the OARE model, the maritime situation information at multiple moments may be represented as multiple OARE vectors.

[0116] Step S32: input the multiple OARE vectors into the encoder to be trained, and compress the multiple OARE vectors into compressed OARE vectors of a specified length.

[0117] In this embodiment, after obtaining a plurality of OARE vectors, the plurality of OARE vectors may be input into an encoder to be trained, and the plurality of OARE vectors may be compressed to a specified length in the encoder to obtain a compressed OARE vector. The specified length may be set as required, and the encoder may treat the attribute value of each feature (i.e., OARE vector) of the maritime situation information at each moment as an input node to perform feature compression. The attribute value may be, for example, the crew's attribute name Name, the ship's attribute maximum range MaxRang, the attribute start time ST of the climate factor, and the like.

[0118] Step S33: Reconstruct data using the compressed OARE vector to obtain a reconstructed OARE vector.

[0119] In this embodiment, after the compressed OARE vector is obtained, the compressed OARE vector may be used to perform data reconstruction to obtain a reconstructed OARE vector.

[0120] Step S34: updating the parameters of the encoder to be trained according to the reconstructed OARE vector and the multiple OARE vectors to obtain the trained encoder.

[0121] In this embodiment, after obtaining the reconstructed OARE vector, the parameters of the encoder to be trained can be updated according to the reconstructed OARE vector and the multiple OARE vectors to obtain a trained encoder. Specifically, the loss function value can be calculated according to the reconstructed OARE vector and the multiple OARE vectors, the parameters of the encoder to be trained can be updated according to the loss function value, and when the loss function converges, the parameters of the encoder to be trained can be fixed to obtain a trained encoder.

[0122] In this embodiment, data compression and reconstruction are achieved through an automatic encoder, and a very long situation data stream is compressed to a certain length, thereby achieving data compression and feature extraction. This not only saves manpower, but also allows the network structure to be freely adjusted to make feature extraction more intelligent.

[0123] In combination with the above embodiments, the present invention also provides a hierarchical ship trajectory prediction method based on deep reinforcement learning, in which the maritime situation information can be sorted out and divided into three structures: basic action layer, bottom task layer, and upper task layer. Among them, the i-th maritime task in the maritime task set is any upper task in the upper task layer. Any upper task is composed of multiple bottom tasks in the bottom task layer.

[0124] Any underlying task in the underlying task layer is composed of multiple basic actions in the basic action layer, and the basic action layer includes at least one of the following: action distance, action direction; and any basic action in the basic action layer is action a performed by any intelligent agent in state s according to the predicted trajectory.

[0125] In this embodiment, the hierarchical structure of maritime tasks is utilized, and the diversified maritime tasks are actually composed of different permutations and combinations of the underlying general tasks. Therefore, if the agent training process can be reasonably constructed so that the agent learns the underlying tasks, the decision output can be quickly adjusted according to the dynamic maritime situation, thereby reducing the decision space and achieving multi-task optimal decision-making. In this embodiment, the hierarchical representation established in the early stage is used to hierarchize the tasks, and the large-scale complex game confrontation problems are artificially decomposed and abstracted in the state space or action space according to the time domain characteristics, and the divide-and-conquer strategy is adopted to solve the sub-game problems. The hierarchical reinforcement learning proposed in this embodiment can simplify the problem processing by forming hierarchical control through time abstraction, thereby providing an effective method for reinforcement learning to solve complex game confrontation problems.

[0126] In combination with the above embodiments, in one implementation, the present invention further provides a hierarchical ship trajectory prediction method based on deep reinforcement learning. In this method, the game task network can be trained by the following steps:

[0127] Step S41: input the sample joint states, sample joint actions and sample total rewards of multiple historical sample sea tasks into the game task network to be trained to obtain the trajectory prediction information of the next sample sea task.

[0128] In this embodiment, in order to help the agent learn the knowledge between the levels of multiple game tasks (sea tasks), meta-deep reinforcement learning introduces a game task network to encode specific task information. The game task network can encode specific task information. The game task network takes a series of continuous historical trajectories as input and outputs a specific game task encoding. Specifically, during the training process of the game task network, the sample joint state, sample joint action and sample total return of multiple historical sample sea tasks can be input into the game task network to be trained, and the trajectory prediction information of the next sample sea task output by the game task network is obtained.

[0129] The sample joint state, sample joint action and sample total reward of the historical sample maritime mission are the joint state, joint action and total reward obtained by processing the maritime situation information of the historical sample mission through N agents (who have completed the first stage of deep reinforcement learning). The historical sample maritime mission is a maritime mission that has been actually processed in the past and used as a training sample.

[0130] Step S42: Based on the trajectory prediction information of the next sample sea mission and the strategies of the actual sample sea missions after the multiple historical sample sea missions, the parameters of the game task network to be trained are updated to obtain the game task network.

[0131] In this embodiment, it is necessary to determine the strategy of the actual sample sea task after multiple historical sample sea tasks, and the actual sample sea task is the actual sea task that occurs after multiple historical sample sea tasks. After obtaining the trajectory prediction information of the next sample sea task output by the game task network, the parameters of the game task network to be trained can be updated based on the trajectory prediction information of the next sample sea task and the strategy of the actual sample sea task after multiple historical sample sea tasks to obtain the game task network. Specifically, the loss function value of the game task network can be calculated according to the trajectory prediction information of the next sample sea task and the strategy of the actual sample sea task, and the parameters of the game task network to be trained can be updated according to the loss function value. When the loss function of the game task network converges, the parameters of the game task network to be trained are fixed to obtain the game task network.

[0132] In combination with the above embodiments, in one implementation, the present invention further provides a hierarchical ship trajectory prediction method based on deep reinforcement learning. In this method, the basic evaluation network can be trained by the following steps:

[0133] Step S51: based on the sample joint state and sample joint action of the sample sea task, obtain the actual Q value corresponding to the sample sea task; the actual Q value represents the decision-making level for the sample sea task.

[0134] The sample sea mission in this embodiment is a sea mission used as a sample, and the sample joint state and sample joint action of the sample sea mission have been obtained through processing by N intelligent agents, so that based on the sample joint state and sample joint action of the sample sea mission, the actual Q value corresponding to the sample sea mission can be obtained. The actual Q value is the Q value directly calculated based on the sample joint state and the sample joint action, and the actual Q value represents the decision-making level for the sample sea mission.

[0135] Step S52: inputting the sample joint state and sample joint action of the sample offshore task into the basic evaluation network to be trained, and obtaining the predicted Q value corresponding to the sample offshore task.

[0136] In this embodiment, the sample joint state and sample joint action of the sample marine task can be input into the basic evaluation network to be trained to obtain the predicted Q value corresponding to the sample marine task output by the basic evaluation network to be trained. The predicted Q value is the Q value predicted by the basic evaluation network based on the sample joint state and the sample joint action.

[0137] Step S53: Based on the predicted Q value and the actual Q value, the parameters of the basic evaluation network to be trained are updated to obtain the basic evaluation network.

[0138] In this embodiment, after obtaining the actual Q value corresponding to the sample sea task and the predicted Q value corresponding to the sample sea task, the parameters of the basic evaluation network to be trained can be updated based on the predicted Q value and the actual Q value to obtain the basic evaluation network. Specifically, the loss function value of the basic evaluation network can be calculated according to the predicted Q value and the actual Q value, and the parameters of the basic evaluation network to be trained can be updated according to the loss function value. When the loss function of the basic evaluation network converges, the parameters of the basic evaluation network to be trained are fixed to obtain the basic evaluation network.

[0139] In this embodiment, by separating the evaluation of the current maritime task (i.e., the basic evaluation network) from the meta-evaluation network, it is beneficial for the meta-evaluation network to consider the game history task information and the current maritime situation information respectively.

[0140] In combination with the above embodiments, in one implementation, the present invention further provides a hierarchical ship trajectory prediction method based on deep reinforcement learning. In the method, the meta-evaluation network can be trained by the following steps:

[0141] Step S61: Based on the sample joint state and sample joint action of the historical sample sea mission, obtain the sample Q value corresponding to the historical sample sea mission.

[0142] In this embodiment, the sample Q value corresponding to the historical sample maritime mission can be obtained based on the sample joint state and sample joint action of the historical sample maritime mission. Among them, the sample joint state, sample joint action and sample total reward of the historical sample maritime mission are the joint state, joint action and total reward obtained by processing the maritime situation information of the historical sample mission through N agents (the first stage of deep reinforcement learning has been completed). The historical sample maritime mission is a maritime mission that has been actually processed in history and used as a training sample. The sample Q value is a Q value directly calculated based on the sample joint state and sample joint action of the historical sample maritime mission, and the sample Q value represents the decision-making level for the historical sample maritime mission.

[0143] Step S62: Based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission, the actual sample Q value corresponding to the historical sample sea mission is obtained.

[0144] In this embodiment, after obtaining the sample Q value corresponding to the historical sample sea mission, the actual sample Q value corresponding to the historical sample sea mission can be obtained based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission. The next historical sample sea mission after the historical sample sea mission is the actual sample sea mission that occurs after the historical sample sea mission.

[0145] Step S63: input the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission into the meta-evaluation network to be trained to obtain the predicted sample Q corresponding to the historical sample sea mission.

[0146] In this embodiment, the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission can be input into the meta-evaluation network to be trained to obtain the output of the meta-evaluation network to be trained: the predicted sample Q corresponding to the historical sample sea mission.

[0147] The actual sample Q value is the Q value directly calculated based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission, and the predicted sample Q is the Q value predicted by the meta-evaluation network based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission. The Q value of this embodiment represents the decision level.

[0148] Step S64: Based on the predicted sample Q value and the actual sample Q value, the parameters of the meta-evaluation network to be trained are updated to obtain the meta-evaluation network.

[0149] In this embodiment, the parameters of the meta-evaluation network to be trained can be updated based on the predicted sample Q value and the actual sample Q value to obtain the meta-evaluation network. Specifically, the loss function value of the meta-evaluation network can be calculated based on the predicted sample Q value and the actual sample Q value, and the parameters of the meta-evaluation network to be trained can be updated based on the loss function value. When the loss function of the meta-evaluation network converges, the parameters of the meta-evaluation network to be trained are fixed to obtain the meta-evaluation network.

[0150] In this embodiment, the meta-evaluation network uses meta-deep reinforcement learning to solve the game confrontation problem. The core idea is to make full use of the training data of different tasks, find common rules from them, and extract general knowledge. The game confrontation data under different game tasks (sea tasks) contains the common knowledge of the intelligent agent to complete the task. By extracting meta-knowledge, it can not only alleviate the cold start problem of the strategy model at the beginning of the new sea task training, but also improve the learning efficiency and decision-making level Q of the strategy model in the new sea task.

[0151] In one embodiment, if Figure 3 As shown, Figure 3 FIG. 1 is a processing flow diagram of a multi-task multi-level multi-agent reinforcement learning method according to an embodiment of the present invention. In this embodiment, tasks can be layered according to the layered representation established in the early stage. The multi-agent on the left first learns the strategy π of a single task. i , and then learn a general strategy through multi-task meta-learning integration on the right. That is to say, this embodiment proposes a maritime decision-making model, which includes: a multi-agent reinforcement learning network on the left, and a meta-learning neural network on the right. Figure 3 As shown in the figure, first, the maritime environment (maritime situation information) is represented as OARE form, and the OARE semantic features are extracted and entered into the automatic encoder to obtain the joint state S; then the joint state S is classified into multiple states S according to the task. i , and distribute it to n agents; multiple agents make corresponding actions a through coordination mechanism n , and then they are combined into action A to feed back to the marine environment. Then, the marine environment gives each agent a reward ri according to their respective actions and reward mechanisms, and obtains the final total reward R.

[0152] Among them, the multi-agent can obtain the strategy π for each maritime task based on reward training i ,Every time a new strategy is obtained, a new task information will be updated according to the new strategy and task data, and a new maritime task will be generated to enrich the maritime task set.

[0153] Next, the meta-learning neural network on the right is a layered neural network architecture. The top layer establishes a meta-evaluation network that can extract meta-knowledge, and the bottom layer establishes an action network for multiple maritime tasks (i.e., the game tasks in the figure). When new task data is generated, that is, there is a current maritime task, the new task data can be transmitted to the basic evaluation network on the left. The joint state and joint action information (s, a, state value-action) of the current maritime task are used as the input of the basic evaluation network, and the output of the basic evaluation network is Q (s, a).

[0154] Then, the historical information ((s t-1, a t-1 , r t-1 )、(s t-2 , a t-2 , r t-2 ),……,(s t-n , a t- n , r t- n )), that is, the joint state, joint action and total reward of each executed maritime task are taken as the input of the game task network, and the output of the game task network is obtained: the encoding k of the specific task, and the trajectory prediction information of the next maritime task is obtained.

[0155] Finally, the output Q(s, a) of the basic evaluation network is concatenated with the code k of the specific task output by the game task network as the input information of the meta-evaluation network. After being processed by the meta-evaluation network, Q(s, a, k) is obtained. Thus, Q(s, a, k) is passed through multiple action networks to obtain the actions a of each game task i, and the prediction of the trajectory of ships at sea is completed. Among them, in the testing phase of the meta-learning neural network, less interactive data can be used to train the action network of a specific game task. In this process, the parameters of the basic evaluation network, meta-evaluation network and game task network are fixed.

[0156] In this embodiment, in order to enhance the versatility of the multi-agent strategy learned by the algorithm in different situations, a multi-level multi-agent reinforcement learning method for multiple tasks is proposed. By utilizing the hierarchical nature of tasks, each agent has multiple types of tasks to handle. It first learns the strategy of a single task, and then integrates and learns a general strategy through meta-learning, thereby improving the versatility of the maritime strategy in different maritime situations and the robustness of the maritime decision-making model.

[0157] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0158] Based on the same inventive concept, an embodiment of the present invention provides a hierarchical ship trajectory prediction device 500 based on deep reinforcement learning. Figure 4 , Figure 4 1 is a block diagram of a hierarchical ship trajectory prediction device based on deep reinforcement learning provided by an embodiment of the present invention. Figure 4 As shown, the device 500 includes:

[0159] The deep reinforcement learning module 501 is used for, for the i-th maritime task in the maritime task set, N intelligent agents take the maximization of the total reward R as the goal, based on the sample maritime situation information of the i-th maritime task, to perform the first stage of deep reinforcement learning. At the end of the first stage of deep reinforcement learning, the N intelligent agents each learn a strategy for the i-th maritime task, and the strategy of each intelligent agent for the i-th maritime task is: perform action a according to the predicted trajectory in state s, and the reward is r; wherein the total reward R is the sum of the rewards r of the N intelligent agents, and the N intelligent agents represent N ships;

[0160] The task decision module 502 is used to input the joint state, joint action and total reward of each sea task that has been executed in the past into the game task network, and the trajectory prediction information of the next sea task; the joint action is the sum of the actions a executed by each of the N intelligent agents, and the joint state is the sum of the states s corresponding to each of the N intelligent agents;

[0161] The basic evaluation module 503 is used to input the current maritime situation information into the N agents that have undergone the deep reinforcement learning of the first stage, obtain the joint state and joint action of the current maritime task, input the joint state and joint action of the current maritime task into the basic evaluation network, and obtain the Q value of the current maritime task;

[0162] The meta-evaluation module 504 is used to input the trajectory prediction information of the next maritime task and the Q value of the current maritime task into the meta-evaluation network to obtain a new joint action of each maritime task in the maritime task set; the new joint action is the sum of the new actions a performed by each of the N agents according to the new predicted trajectory;

[0163] The marine action execution module 505 is used for the new joint action based on the current marine task, and each of the N intelligent agents executes their own new action according to the new predicted trajectory.

[0164] Optionally, the deep reinforcement learning module 501 includes:

[0165] An OARE representation module is used to represent the sample maritime situation information of the i-th maritime mission as a sample OARE vector based on the OARE model;

[0166] An encoding module, used for inputting the sample OARE vector into a trained encoder to obtain the joint state of the i-th marine mission;

[0167] A state distribution module is used to distribute the joint state of the i-th maritime task to the N agents to perform the first stage of deep reinforcement learning.

[0168] Optionally, the apparatus 500 further includes: an encoder training module, configured to train the encoder, wherein the encoder training module includes:

[0169] A sample OARE representation module, used for representing the maritime situation information at multiple moments as multiple OARE vectors based on the OARE model;

[0170] A compression module, used for inputting the plurality of OARE vectors into an encoder to be trained, and compressing the plurality of OARE vectors into a compressed OARE vector of a specified length;

[0171] A reconstruction module, used for reconstructing data using the compressed OARE vector to obtain a reconstructed OARE vector;

[0172] The encoding training module is used to update the parameters of the encoder to be trained according to the reconstructed OARE vector and the multiple OARE vectors to obtain the trained encoder.

[0173] Optionally, the i-th marine task in the marine task set is any upper-level task in an upper-level task layer;

[0174] Any upper-layer task is composed of multiple lower-layer tasks in the lower-layer task layer;

[0175] Any bottom-level task in the bottom-level task layer is composed of a plurality of basic actions in the basic action layer, and the basic action layer includes at least one of the following: action distance, action direction;

[0176] Any basic action in the basic action layer is an action a performed by any agent in state s according to the predicted trajectory.

[0177] Optionally, the device 500 further includes: a game task training module, which is used to train the game task network, and the game task training module includes:

[0178] The sample task decision module is used to input the sample joint states, sample joint actions and sample total rewards of multiple historical sample sea tasks into the game task network to be trained to obtain the trajectory prediction information of the next sample sea task;

[0179] The game task network training module is used to update the parameters of the game task network to be trained based on the trajectory prediction information of the next sample sea task and the strategies of the actual sample sea tasks after the multiple historical sample sea tasks to obtain the game task network.

[0180] Optionally, the device 500 further includes: a basic evaluation training module, which is used to train the basic evaluation network, and the basic evaluation training module includes:

[0181] A first determination module is used to obtain an actual Q value corresponding to the sample marine task based on a sample joint state and a sample joint action of the sample marine task; the actual Q value represents a decision level for the sample marine task;

[0182] A first prediction module is used to input the sample joint state and the sample joint action of the sample marine task into the basic evaluation network to be trained to obtain a predicted Q value corresponding to the sample marine task;

[0183] The basic evaluation network training module is used to update the parameters of the basic evaluation network to be trained based on the predicted Q value and the actual Q value to obtain the basic evaluation network.

[0184] Optionally, the device 500 further includes: a meta-evaluation training module, which is used to train the meta-evaluation network, and the meta-evaluation training module includes:

[0185] A second determination module is used to obtain a sample Q value corresponding to the historical sample marine mission based on the sample joint state and sample joint action of the historical sample marine mission;

[0186] A third determination module is used to obtain an actual sample Q value corresponding to the historical sample sea mission based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission;

[0187] The second prediction module is used to input the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission into the meta-evaluation network to be trained, so as to obtain the predicted sample Q value corresponding to the historical sample sea mission;

[0188] The meta-evaluation network training module is used to update the parameters of the meta-evaluation network to be trained based on the predicted sample Q value and the actual sample Q value to obtain the meta-evaluation network.

[0189] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the hierarchical ship trajectory prediction method based on deep reinforcement learning as described in any of the above embodiments of the present invention are implemented.

[0190] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 600, such as Figure 5shown. Figure 5 6 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory 602, a processor 601, and a computer program stored in the memory and executable on the processor, and when executed by the processor, the steps in the hierarchical ship trajectory prediction method based on deep reinforcement learning described in any of the above embodiments of the present invention are implemented.

[0191] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0192] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0193] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0194] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0195] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0197] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0198] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0199] The above is a detailed introduction to the hierarchical ship trajectory prediction method, device and product based on deep reinforcement learning provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A hierarchical ship trajectory prediction method based on deep reinforcement learning, It is characterized in that The method comprises: For the i-th maritime task in the maritime task set, N agents perform the first stage of deep reinforcement learning based on the sample maritime situation information of the i-th maritime task with the goal of maximizing the total reward R. At the end of the first stage of deep reinforcement learning, the N agents each learn a strategy for the i-th maritime task, and the strategy of each agent for the i-th maritime task is: perform action a according to the predicted trajectory in state s, and the reward is r; wherein the total reward R is the sum of the rewards r of the N agents, and the N agents represent N ships; The joint state, joint action and total reward of each maritime task that has been executed in the past are input into the game task network to obtain the trajectory prediction information of the next maritime task; the joint action is the sum of the actions a executed by each of the N agents according to the historical trajectory, and the joint state is the sum of the states s corresponding to each of the N agents; Input the current maritime situation information into the N agents that have undergone the first stage of deep reinforcement learning to obtain the joint state and joint action of the current maritime task, and input the joint state and joint action of the current maritime task into the basic evaluation network to obtain the Q value of the current maritime task; Inputting the trajectory prediction information of the next maritime task and the Q value of the current maritime task into the meta-evaluation network to obtain a new joint action for each maritime task in the maritime task set; the new joint action is the sum of the new actions a performed by each of the N agents according to the new predicted trajectory; Based on the new joint action of the current maritime mission, the N agents each perform their own new actions according to the new predicted trajectory.

2. The hierarchical ship trajectory prediction method based on deep reinforcement learning according to claim 1, It is characterized in that The first stage of deep reinforcement learning based on the sample maritime situation information of the i-th maritime mission includes: Based on the OARE model, the sample maritime situation information of the i-th maritime mission is represented as a sample OARE vector; Inputting the sample OARE vector into the trained encoder to obtain the joint state of the i-th marine mission; Assigning the joint state of the i-th maritime task to the N agents to perform the first stage of deep reinforcement learning; The OARE model is an object-attribute-relationship-environment model, and the OARE model is defined as: ; Among them, O is a finite object set consisting of several different objects, A is a finite attribute set consisting of the unique attributes of the description and characterization objects, R is a finite relationship set consisting of several relationships, and E is the environment in which the object is located.

3. The hierarchical ship trajectory prediction method based on deep reinforcement learning according to claim 2, It is characterized in that The encoder is trained by the following steps: Based on the OARE model, the maritime situation information at multiple moments is represented as multiple OARE vectors; Inputting the plurality of OARE vectors into an encoder to be trained, and compressing the plurality of OARE vectors into compressed OARE vectors of a specified length; Reconstructing data using the compressed OARE vector to obtain a reconstructed OARE vector; The parameters of the encoder to be trained are updated according to the reconstructed OARE vector and the multiple OARE vectors to obtain the trained encoder.

4. The hierarchical ship trajectory prediction method based on deep reinforcement learning according to any one of claims 1 to 3, It is characterized in that The i-th marine task in the marine task set is any upper-level task in the upper-level task layer; Any upper-layer task is composed of multiple lower-layer tasks in the lower-layer task layer; Any bottom-level task in the bottom-level task layer is composed of a plurality of basic actions in the basic action layer, and the basic action layer includes at least one of the following: action distance, action direction; Any basic action in the basic action layer is an action a performed by any agent in state s according to the predicted trajectory.

5. The hierarchical ship trajectory prediction method based on deep reinforcement learning according to any one of claims 1 to 3, It is characterized in that The game task network is trained by the following steps: The sample joint states, sample joint actions and sample total rewards of multiple historical sample sea tasks are input into the game task network to be trained to obtain the trajectory prediction information of the next sample sea task; Based on the trajectory prediction information of the next sample sea mission and the strategies of the actual sample sea missions after the multiple historical sample sea missions, the parameters of the game task network to be trained are updated to obtain the game task network.

6. The hierarchical ship trajectory prediction method based on deep reinforcement learning according to any one of claims 1 to 3, It is characterized in that The basic evaluation network is trained by the following steps: Based on the sample joint state and the sample joint action of the sample sea task, an actual Q value corresponding to the sample sea task is obtained; the actual Q value represents the decision level for the sample sea task; Inputting the sample joint state and the sample joint action of the sample marine task into the basic evaluation network to be trained to obtain the predicted Q value corresponding to the sample marine task; Based on the predicted Q value and the actual Q value, the parameters of the basic evaluation network to be trained are updated to obtain the basic evaluation network.

7. The hierarchical ship trajectory prediction method based on deep reinforcement learning according to any one of claims 1 to 3, It is characterized in that The meta-evaluation network is trained by the following steps: Based on the sample joint state and sample joint action of the historical sample sea mission, obtaining the sample Q value corresponding to the historical sample sea mission; Based on the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission, an actual sample Q value corresponding to the historical sample sea mission is obtained; Inputting the sample Q value corresponding to the historical sample sea mission and the trajectory prediction information of the next historical sample sea mission after the historical sample sea mission into the meta-evaluation network to be trained, and obtaining the predicted sample Q value corresponding to the historical sample sea mission; Based on the predicted sample Q value and the actual sample Q value, the parameters of the meta-evaluation network to be trained are updated to obtain the meta-evaluation network.

8. A hierarchical ship trajectory prediction device based on deep reinforcement learning, It is characterized in that The device comprises: A deep reinforcement learning module is used for, for the i-th maritime task in the maritime task set, N intelligent agents take maximizing the total reward R as the goal, based on the sample maritime situation information of the i-th maritime task, to perform the first stage of deep reinforcement learning. At the end of the first stage of deep reinforcement learning, the N intelligent agents each learn a strategy for the i-th maritime task, and the strategy of each intelligent agent for the i-th maritime task is: perform action a according to the predicted trajectory in state s, and the reward is r; wherein the total reward R is the sum of the rewards r of the N intelligent agents, and the N intelligent agents represent N ships; The task decision module is used to input the joint state, joint action and total reward of each maritime task that has been executed in the past into the game task network to obtain the trajectory prediction information of the next maritime task; the joint state is the sum of the actions a executed by each of the N intelligent agents, and the joint state is the sum of the states s corresponding to each of the N intelligent agents; A basic evaluation module is used to input the current maritime situation information into the N agents that have undergone the first stage of deep reinforcement learning to obtain the joint state and joint action of the current maritime task, and input the joint state and joint action of the current maritime task into the basic evaluation network to obtain the Q value of the current maritime task; A meta-evaluation module is used to input the trajectory prediction information of the next maritime task and the Q value of the current maritime task into a meta-evaluation network to obtain a new joint action for each maritime task in the maritime task set; the new joint action is the sum of the new actions a performed by each of the N agents according to the new predicted trajectory; The marine action execution module is used for the new joint action based on the current marine task, and each of the N intelligent agents executes their own new action according to the new predicted trajectory.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the computer program is executed by the processor, the hierarchical ship trajectory prediction method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the hierarchical ship trajectory prediction method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.