A ship agent intelligent control method and device based on reinforcement learning

Abstract

The present invention relates to a ship agent intelligent control method and device based on reinforcement learning, including the following: S1, initializing the ship agent; S2, obtaining the current state of the ship agent, the current state including the ship agent at the current heading angle #imgabs0# speed v and geographical location; S3, combining the current state and the reward function, selecting the corresponding rudder angle in the action space through the strategy, and taking actions to update the current state; S4, based on the updated current state, calculating the reward value through the reward function; S5, updating the value of Q-table (s, a) based on the updated current state and the calculated reward value; S6, judging whether the preset termination condition is met, if so, outputting the final reward function and strategy, i.e. the updated Q-table, otherwise returning to S2. The present invention improves the problem of the traditional ship channel reinforcement learning that tends to select a large rudder angle in extreme environments, and can better control the ship agent.

Description

Technical Field

[0001] The present invention relates to the technical field related to ship automatic driving, and in particular to a ship intelligent agent intelligent control method and device based on reinforcement learning. Background Art

[0002] Due to the advantages of unmanned ships such as flexibility, remote monitoring and reconnaissance, coastal patrol and anti-submarine warfare, more than a dozen countries have developed and deployed smart ships and are continuously increasing their research and development efforts on this technology.

[0003] Currently, unmanned ships on the market mainly obtain more accurate information about the surrounding environment by fusing information from multiple sensors, and use deep learning algorithms to analyze and process the acquired data to obtain more accurate navigation information. However, although there has been some progress in the research on autonomous driving of unmanned ships based on reinforcement learning, most of the research only focuses on the application of reinforcement learning in path planning, collision avoidance, etc. There is little research on actual autonomous driving using reinforcement learning, and there is also a lack of research on how to combine reinforcement learning with ship maneuverability optimization.

[0004] Moreover, in complex virtual ocean environments, unmanned ships tend to steer at large rudder angles. However, in real ocean environments, if a ship steers at large angles, it will capsize, which poses a huge risk. Summary of the invention

[0005] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a method and device for intelligent control of a ship agent based on reinforcement learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Specifically, a ship agent intelligent control method based on reinforcement learning is proposed, including the following:

[0008] S1. Initialize the state space and action space of the ship agent, initialize the reward function and strategy of the reinforcement learning algorithm, and establish Q-table (s, a). Each entry Q (s, a) in the Q-table represents the expected cumulative reward obtained by taking action a in state s;

[0009] S2. Obtain the current state of the ship agent, wherein the current state includes the current heading angle of the ship agent. Speed ​​v and geographic location;

[0010] S3, combining the current state and reward function, selects the corresponding steering angle in the action space through the strategy, and takes action to update the current state;

[0011] S4. Calculate the reward value through the reward function based on the updated current state;

[0012] S5. Update the value of Q-table (s, a) based on the updated current state and the calculated reward value;

[0013] S6. Determine whether the preset termination condition is met. If so, output the final reward function and strategy, i.e. the updated Q-table. Otherwise, return to S2.

[0014] Further, specifically, the action space is,

[0015] Action={-35°, -15°, 0°, 15°, 35°}

[0016] That is, the selection range of the rudder angle is any value in Action.

[0017] Further, specifically, the process of calculating the reward value through the reward function includes the following:

[0018] The reward function is,

[0019]

[0020] A is the absolute value of the difference between the last action and this action and 10. (s-1) is the last rudder angle, a s is the rudder angle selected this time,

[0021] A=|10-|a (s-1) -a s ||

[0022] △φ is the difference between the heading angle before and after this action, specifically:

[0023] △φ=φ k-1 -φ k

[0024] φ r =0.01|△φ|

[0025] The unmanned ship selects the rudder angle a s When the absolute value difference between the front and rear rudder angles is not less than 10 degrees, that is, when |a s-1 -a s |≥10,

[0026] r a =0.36e 0.01A +φ r

[0027] When the absolute value of the front and rear rudder angle is less than 10 degrees, that is, when |as-1 -a s |<10

[0028] r a =0

[0029] The final reward value r = -r a .

[0030] Further, specifically, the updating method of Q-table(s, a) includes:

[0031] Q(s,a)=Q(s,a)+α*[r+γ*max(Q(s',a'))-Q(s,a)]

[0032] Q(s,a) is the state-action value function, which represents the action value of taking action a in state s, s is the current state, a is the selected action, α is the learning rate, γ is the discount factor, r is the reward obtained by the ship agent for performing action a in state s, and s' is the next state.

[0033] Further, specifically, the preset termination condition is whether the ship agent reaches the target position.

[0034] The present invention also proposes a ship agent intelligent control device based on reinforcement learning, comprising:

[0035] The initialization module is used to initialize the state space and action space of the ship agent, initialize the reward function and strategy of the reinforcement learning algorithm, and establish a Q-table (s, a). Each entry Q (s, a) in the Q-table represents the expected cumulative reward obtained by taking action a in state s;

[0036] The data acquisition module is used to obtain the current state of the ship intelligent body, which includes the current heading angle of the ship intelligent body. Speed ​​v and geographic location;

[0037] The state update module is used to combine the current state and the reward function, select the corresponding steering angle in the action space through the strategy, and take actions to update the current state;

[0038] A reward value calculation module, used to calculate the reward value through a reward function based on the updated current state;

[0039] Q-table update module, used to update the value of Q-table (s, a) based on the updated current state and the calculated reward value;

[0040] The termination condition judgment module is used to determine whether the preset termination condition is met. If so, the final reward function and strategy, i.e. the updated Q-table, are output. Otherwise, the module returns to the data acquisition module to continue execution.

[0041] The beneficial effects of the present invention are:

[0042] The present invention proposes a method and device for intelligent control of a ship agent based on reinforcement learning. By modifying the reward function, the present invention can provide more rewards than the native function, which means that in the system or algorithm using the present invention, the user or agent can obtain a higher reward value, thereby increasing the efficiency and success rate of task completion. In addition, in the present invention, the range of the rudder angle can be effectively limited to ensure that the operation is performed within a safe and controllable range. This improvement not only improves the stability and reliability of the system, but also helps to reduce operational errors and accident risks. Due to the improvement of the reward function and the limitation of the rudder angle, the present invention can significantly improve the performance of the system. The present invention optimizes the behavior and decision-making of the system, making the system more intelligent, efficient and reliable, and can solve the problems in previous systems where the system may be inefficient, the task cannot be completed, or the operation is inaccurate due to unreasonable reward function design or excessive rudder angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other features of the present disclosure will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar sampling monitoring points. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0044] Figure 1 Shown is a flow chart of a ship intelligent agent intelligent control method based on reinforcement learning of the present invention;

[0045] Figure 2 Shown is a schematic diagram of a ship agent intelligent control method based on reinforcement learning according to the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0047] Example 1, reference Figure 1 as well as Figure 2 The present invention proposes a ship agent intelligent control method based on reinforcement learning, comprising the following:

[0048] S1. Initialize the state space and action space of the ship agent, initialize the reward function and strategy of the reinforcement learning algorithm, and establish Q-table (s, a). Each entry Q (s, a) in the Q-table represents the expected cumulative reward obtained by taking action a in state s;

[0049] S2. Obtain the current state of the ship agent, wherein the current state includes the current heading angle of the ship agent. Speed ​​v and geographic location;

[0050] S3, combining the current state and reward function, selects the corresponding steering angle in the action space through the strategy, and takes action to update the current state;

[0051] S4. Calculate the reward value through the reward function based on the updated current state;

[0052] S5. Update the value of Q-table (s, a) based on the updated current state and the calculated reward value;

[0053] S6. Determine whether the preset termination condition is met. If so, output the final reward function and strategy, i.e. the updated Q-table. Otherwise, return to S2.

[0054] In this embodiment 1, by modifying the reward function, the present invention can provide more rewards than the native function, which means that in the system or algorithm using the present invention, the user or agent can get a higher reward value, thereby increasing the efficiency and success rate of task completion. In addition, in the present invention, the range of the rudder angle can be effectively limited to ensure that the operation is performed within a safe and controllable range. This improvement not only improves the stability and reliability of the system, but also helps to reduce operational errors and accident risks. Due to the improvement of the reward function and the limitation of the rudder angle, the present invention can significantly improve the performance of the system. The present invention optimizes the behavior and decision-making of the system, making the system more intelligent, efficient and reliable, and can solve the problems in previous systems that the system may be inefficient, the task cannot be completed, or the operation is inaccurate due to unreasonable reward function design or excessive rudder angle.

[0055] As a preferred embodiment of the present invention, specifically, the action space is:

[0056] Action={-35°, -15°, 0°, 15°, 35°}

[0057] That is, the selection range of the rudder angle is any value in Action.

[0058] As a preferred embodiment of the present invention, specifically, the process of calculating the reward value by using the reward function includes the following, where the reward function is:

[0059]

[0060] A is the absolute value of the difference between the last action and this action and 10. (s-1) is the last rudder angle, a s is the rudder angle selected this time,

[0061] A=|10-|a (s-1) -a s ||

[0062] △φ is the difference between the heading angle before and after this action, specifically:

[0063] △φ=φ k-1 -φ k

[0064] φ r =0.01|△φ|

[0065] The unmanned ship selects the rudder angle a s When the absolute value difference between the front and rear rudder angles is not less than 10 degrees, that is, when |a s-1 -a s |≥10,

[0066] r a =0.36e 0.01A +φ r

[0067] When the absolute value of the front and rear rudder angle is less than 10 degrees, that is, when |a s-1 -a s |<10

[0068] r a =0

[0069] The final reward value r = -r a .

[0070] In this preferred embodiment, a specific process of calculating the reward value by the reward function is given. The reward value can be accurately calculated through the above process. The specific derivation process of the reward function is given below.

[0071] A=|10-|a (s-1) -a s ||

[0072] △φ=φ k-1 -φ k

[0073] φ r =0.01|△φ|

[0074] Introducing hyperparameter F

[0075]

[0076] Let 0≤φ r +Fe 0.01A ≤1

[0077] beg

[0078] The maximum rudder angle difference is 70, and the maximum value of A is 60

[0079] The maximum heading angle difference is 360, φ r The maximum value is 0.36

[0080] The hyperparameter F value is 0.36

[0081] How to get reward:

[0082]

[0083] As a preferred embodiment of the present invention, specifically, the updating method of Q-table (s, a) includes:

[0084] Q(s,a)=Q(s,a)+α*[r+γ*max(Q(s',a'))-Q(s,a)]

[0085] Q(s,a) is the state-action value function, which represents the action value of taking action a in state s, s is the current state, a is the selected action, α is the learning rate, γ is the discount factor, r is the reward obtained by the ship agent for performing action a in state s, and s' is the next state.

[0086] As a preferred embodiment of the present invention, specifically, the preset termination condition is whether the ship intelligent agent reaches the target position.

[0087] The present invention also proposes a ship agent intelligent control device based on reinforcement learning, comprising:

[0088] The initialization module is used to initialize the state space and action space of the ship agent, initialize the reward function and strategy of the reinforcement learning algorithm, and establish a Q-table (s, a). Each entry Q (s, a) in the Q-table represents the expected cumulative reward obtained by taking action a in state s;

[0089] The data acquisition module is used to obtain the current state of the ship intelligent body, which includes the current heading angle of the ship intelligent body. Speed ​​v and geographic location;

[0090] The state update module is used to combine the current state and the reward function, select the corresponding steering angle in the action space through the strategy, and take actions to update the current state;

[0091] A reward value calculation module, used to calculate the reward value through a reward function based on the updated current state;

[0092] Q-table update module, used to update the value of Q-table (s, a) based on the updated current state and the calculated reward value;

[0093] The termination condition judgment module is used to determine whether the preset termination condition is met. If so, the final reward function and strategy, i.e. the updated Q-table, are output. Otherwise, the module returns to the data acquisition module to continue execution.

[0094] The ship intelligent agent intelligent control device based on reinforcement learning proposed by the present invention has the following advantages when used due to the application of the method proposed by the present invention:

[0095] (1) Providing more precise reward signals: By considering the magnitude of the rudder angle to determine the reward, a more specific and fine-grained reward signal can be provided, enabling the learning algorithm to better identify and learn good rudder angle decisions.

[0096] (2) Strengthening attention to important rudder angles: A larger rudder angle means a larger heading adjustment. Therefore, determining rewards based on the size of the rudder angle can strengthen the understanding of the importance of rudder angle decisions and make the learning algorithm pay more attention to important rudder angle changes.

[0097] (3) Improve learning efficiency: By explicitly considering the size of the rudder angle, the reward function can guide learning more accurately, thereby accelerating the convergence of the learning algorithm and improving learning efficiency. A method and device for optimizing rewards based on coupled dynamic factors of reinforcement learning.

[0098] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0099] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0100] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

[0101] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. A ship agent intelligent control method based on reinforcement learning, characterized in that: Includes the following: S1. Initialize the state space and action space of the ship agent, initialize the reward function and strategy of the reinforcement learning algorithm, and establish a Q-table (s, a). Each entry Q(s, a) in the Q-table represents the expected cumulative reward obtained by taking action a in state s; S2. Obtain the current state of the ship agent, wherein the current state includes the current heading angle of the ship agent. , speed v and geographical location; S3, combining the current state and reward function, selects the corresponding steering angle in the action space through the strategy, and takes action to update the current state; S4. Calculate the reward value through the reward function based on the updated current state; S5. Update the value of Q-table (s, a) based on the updated current state and the calculated reward value; S6, determine whether the preset termination condition is met, if so, output the final reward function and strategy, i.e. the updated Q-table, if not, return to S2; Specifically, the process of calculating the reward value through the reward function includes the following: The reward function is, ； A is the absolute value of the difference between the last action and this action and 10. is the last rudder angle, is the rudder angle selected this time, A= - ； is the difference between the heading angle before and after this action, specifically: ； = ； Unmanned ship choosing rudder angle When the absolute value difference between the front and rear rudder angles is not less than 10 degrees, that is, when , = ； When the absolute value of the front and rear rudder angle is less than 10 degrees, , ； The final reward value r=- .

2. According to the reinforcement learning-based intelligent control method for a ship agent according to claim 1, it is characterized in that: Specifically, the action space is: Action= ; That is, the selection range of the rudder angle is any value in Action.

3. According to the reinforcement learning-based intelligent control method for a ship agent according to claim 1, it is characterized in that: Specifically, the update methods of Q-table (s, a) include: Q(s, a) = Q(s, a) + α * [r+ γ * max(Q(s', a')) - Q(s, a)]; Q(s, a) is the state-action value function, which represents the action value of taking action a in state s, s is the current state, a is the selected action, α is the learning rate, γ is the discount factor, r is the reward obtained by the ship agent for performing action a in state s, and s' is the next state.

4. According to the reinforcement learning-based intelligent control method for a ship agent according to claim 1, it is characterized in that: Specifically, the preset termination condition is whether the ship agent reaches the target position.

5. A ship agent intelligent control device based on reinforcement learning, characterized in that: The method for intelligent control of a ship agent based on reinforcement learning as claimed in claim 1 is applied, and the device comprises: The initialization module is used to initialize the state space and action space of the ship agent, initialize the reward function and strategy of the reinforcement learning algorithm, and establish a Q-table (s, a). Each entry Q(s, a) in the Q-table represents the expected cumulative reward obtained by taking action a in state s; The data acquisition module is used to obtain the current state of the ship agent, which includes the current heading angle of the ship agent. , speed v and geographical location; The state update module is used to combine the current state and the reward function, select the corresponding steering angle in the action space through the strategy, and take actions to update the current state; A reward value calculation module, used to calculate the reward value through a reward function based on the updated current state; Q-table update module, used to update the value of Q-table (s, a) based on the updated current state and the calculated reward value; The termination condition judgment module is used to determine whether the preset termination condition is met. If so, the final reward function and strategy, i.e. the updated Q-table, are output. Otherwise, the module returns to the data acquisition module to continue execution.

Images