Automatic ship berthing control method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a ship automatic berthing control method and system, electronic equipment and a storage medium, and belongs to the technical field of ship control. According to the method, motion state data of a current ship and a first relative distance between the current ship and a target berth are obtained, the motion state data and the first relative distance are input into a control strategy model for automatic berthing of the ship, and the ship control quantity at the next moment is obtained. The control strategy model is obtained through training based on a reinforcement learning algorithm, that is, environment interaction data is obtained, a target function is maximized, a value network is trained according to the environment interaction data, a strategy network is trained according to the value network and the environment interaction data, and the target function represents an accumulated reward value of an action output by the strategy network; the motion represents the ship control quantity, and then a control strategy model is determined according to the trained strategy network. According to the method, the workload of ship automatic berthing and leaving algorithm modeling can be reduced, the modeling precision is improved, and the safety of the ship automatic berthing process is improved.

Description

Technical Field

[0001] The present application relates to the field of ship control technology, and in particular to a ship automatic berthing control method, system, electronic equipment and storage medium. Background Art

[0002] The research on automatic berthing technology for ships began in the 1970s. With the development of the shipbuilding industry and the complexity of shipping tasks, the demand for automatic berthing technology has become increasingly high. The automatic berthing system can improve the safety and efficiency of ships, reduce human operating errors and unnecessary downtime, and relieve the work pressure of crew members. Therefore, it has become an indispensable part of ship control systems. The research on automatic berthing technology for ships mainly involves multiple fields such as ship dynamics, control theory, and computer technology. The design and implementation of the automatic berthing system need to be combined with the characteristics of the ship and the actual application requirements, and at the same time select appropriate control algorithms and control strategies. In the berthing process of inland container ships, a lot of situations need to be considered. On the one hand, due to the influence of wind, waves, flow and variable effects, complex dynamic behaviors will be generated, making it difficult to accurately control the operation of the ship, and experienced crew members are required to complete the berthing process. On the other hand, the low-speed navigation during the berthing process makes it difficult to accurately model the ship. At present, the research on automatic ship berthing system uses ship berthing data as training data to train artificial neural networks, and uses the two trained models to perform trajectory planning and control respectively. This method requires a large amount of training data and a lot of training time. The workload of the automatic ship berthing and unberthing algorithm is large and the modeling accuracy is low, resulting in a high risk of collision during the automatic berthing process of ships. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method, system, electronic device and storage medium for automatic berthing control of ships, so as to reduce the workload of modeling the algorithm for automatic berthing and unberthing of ships, improve the modeling accuracy, and improve the safety of the automatic berthing process of ships.

[0004] To achieve the above object, one aspect of an embodiment of the present application provides a method for controlling automatic berthing of a ship, comprising the following steps:

[0005] Acquire motion state data of the current ship and a first relative distance from the target berth, wherein the motion state data includes the ship speed;

[0006] Inputting the motion state data and the first relative distance into a control strategy model for automatic berthing of a ship to obtain a ship control variable at the next moment, wherein the ship control variable includes a rudder angle and a rotation speed of the ship;

[0007] The control strategy model is trained based on a reinforcement learning algorithm, and the training process of the control strategy model includes the following steps:

[0008] Get environmental interaction data;

[0009] By maximizing an objective function, a value network is trained according to the environmental interaction data, and a strategy network is trained according to the value network and the environmental interaction data, wherein the objective function represents a cumulative reward value of an action output by the strategy network, and the action represents a control amount of the ship;

[0010] Determine the control strategy model based on the trained strategy network.

[0011] In some embodiments, the step of obtaining the motion state data of the current ship and the first relative distance to the target berth comprises the following steps:

[0012] The second relative distance between the ship and the shore is collected by a microwave radar array arranged on the ship;

[0013] The position of the ship is collected by a positioning device installed on the ship;

[0014] Determining a first relative distance according to the first relative distance, the ship position and the target berth position;

[0015] The motion status data is collected through the electronic compass installed on the ship.

[0016] In some embodiments, the step of obtaining the environment interaction data comprises the following steps:

[0017] Acquiring berthing task information, wherein the berthing task information includes an environment map, a target berth, and a reward function;

[0018] Constructing a reinforcement learning environment under a subprocess according to a ship mathematical model and the berthing task information;

[0019] Based on the reinforcement learning environment, the control strategy of the main process is used to guide the behavior of the intelligent agent, and the environmental feedback data of the sub-process is obtained;

[0020] The environmental interaction data is determined based on the environmental feedback data under each sub-process.

[0021] In some embodiments, the objective function is expressed as follows:

[0022]

[0023]

[0024] Among them, R(s k , a k ) represents the defined reward function, T is the set docking mission duration, s k =[uk , v k ,φ k , X k , Y k , ψ k ] T is the state vector, X k , Y k Indicates the ship's position, u k , v k They represent the longitudinal speed and the transverse drift speed respectively, ψk represents the ship's heading, φk represents the bow angular velocity, and the action represents the ship control quantity, The thrust of the propeller, represents the rudder moment; s I represents the initial state of the ship, s F represents the target terminal state achieved under the control of the policy network; s min and max Respectively represent the minimum state and the maximum state, a min and a max Represent the minimum and maximum values ​​of the action respectively.

[0025] In some embodiments, the reward function is expressed as:

[0026]

[0027] Among them, β represents the weight factor, α is a coefficient greater than zero, and s k Indicates the current state, s F represents the target termination state, and C represents a constant.

[0028] In some embodiments, the step of training a value network according to the environment interaction data by maximizing an objective function, and training a policy network according to the value network and the environment interaction data, comprises the following steps:

[0029] Training a value network based on the current batch of training data in the environment interaction data;

[0030] Inputting the states in the current batch of training data into the strategy network to obtain the action distribution at the next moment;

[0031] The value network is used to select an action for the action distribution at the next moment to obtain a target action;

[0032] Determining the value of the objective function according to the state corresponding to the target action;

[0033] Determine whether the maximum value of the objective function is currently found according to the value of the objective function;

[0034] When the maximum value of the objective function is not found, the parameters of the policy network are updated, and the value network and the policy network are continuously trained according to the next batch of training data in the environment interaction data;

[0035] When the maximum value of the objective function is found, the current policy network is determined as the trained policy network.

[0036] In some embodiments, updating the parameters of the policy network comprises the following steps:

[0037] Determine the current number of training sessions;

[0038] Selecting a corresponding proxy target according to the number of training times to guide the parameter update of the policy network, wherein when the number of training times is less than a preset number of times, selecting a first proxy target; when the number of training times is greater than or equal to the preset number of times, selecting a second proxy target;

[0039] The first proxy target is expressed as follows:

[0040]

[0041] The second proxy target is expressed as follows:

[0042]

[0043] Among them, ∈ represents the truncation hyperparameter, clup() represents the truncation function, and rt represents the ratio of the new and old strategies. represents the advantage function.

[0044] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a ship automatic berthing control system, comprising:

[0045] The first module is used to obtain the motion state data of the current ship and the first relative distance to the target berth, wherein the motion state data includes the speed of the ship;

[0046] The second module is used to input the motion state data and the first relative distance into a control strategy model for automatic berthing of a ship to obtain a ship control quantity at the next moment, wherein the ship control quantity includes a rudder angle and a rotation speed of the ship;

[0047] The control strategy model is trained based on a reinforcement learning algorithm, and the training process of the control strategy model includes the following steps:

[0048] Get environmental interaction data;

[0049] By maximizing an objective function, a value network is trained according to the environmental interaction data, and a strategy network is trained according to the value network and the environmental interaction data, wherein the objective function represents a cumulative reward value of an action output by the strategy network, and the action represents a control amount of the ship;

[0050] Determine the control strategy model based on the trained strategy network.

[0051] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program implements the method described in the above embodiment when executed by the processor.

[0052] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the above embodiment.

[0053] The automatic berthing control method, system, electronic device and storage medium proposed in the present application obtain the motion state data of the current ship and the first relative distance from the target berth, input the motion state data and the first relative distance into the control strategy model for automatic berthing of the ship, and obtain the ship control amount at the next moment. The control strategy model is trained based on the reinforcement learning algorithm, that is, the environmental interaction data is obtained, and the value network is trained according to the environmental interaction data by maximizing the objective function, and the strategy network is trained according to the value network and the environmental interaction data, wherein the objective function represents the cumulative reward value of the action output by the strategy network, and the action represents the ship control amount, and then the control strategy model is determined according to the trained strategy network. The present application uses reinforcement learning to make the berthing task process no longer rely on the ship motion model to control the movement of the ship, and does not require a large amount of training data. The reinforcement learning algorithm can reduce the workload of the ship automatic berthing and unberthing algorithm modeling, improve the modeling accuracy, and thus improve the safety of the ship automatic berthing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a method for controlling automatic berthing of a ship provided in an embodiment of the present application;

[0055] Figure 2 It is a schematic diagram of the structure of the automatic docking algorithm provided in the embodiment of the present application;

[0056] Figure 3 is a flow chart of a training method for a control strategy model provided in an embodiment of the present application;

[0057] Figure 4 is a flow chart of the control strategy model training process provided by an embodiment of the present application;

[0058] Figure 5 This is a schematic diagram of a ship berthing process provided by an embodiment of the present application;

[0059] Figure 6 is a schematic diagram of the reinforcement learning process provided by an embodiment of the present application;

[0060] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] The automatic berthing control method, system, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the automatic berthing control method of a ship in the embodiments of the present application is described.

[0065] The automatic berthing control method for ships provided in the embodiments of the present application relates to the field of ship control technology. The automatic berthing control method for ships provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the automatic berthing control method for ships, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] Figure 1 is an optional flow chart of the ship automatic berthing control method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S102.

[0068] Step S101, obtaining motion state data of the current ship and a first relative distance from a target berth, wherein the motion state data includes the ship's speed;

[0069] Step S102, inputting the motion state data and the first relative distance into a control strategy model for automatic berthing of a ship, and obtaining a ship control variable at the next moment, wherein the ship control variable includes a rudder angle and a rotation speed of the ship.

[0070] In some embodiments, in step S101 to step S102, please refer to Figure 2The automatic berthing algorithm structure of this embodiment is divided into an upper scheduling algorithm and a lower motion control algorithm. The upper scheduling algorithm updates the distance between the ship and the berth shore in real time through the microwave radar array on the ship (such as an inland container ship), and the ship-borne electronic compass transmits the current ship's speed in real time. The above sensors combine GPS data to determine the position of the ship relative to the berth. In addition to providing the real-time posture and state of the ship to the control algorithm, the scheduling method can also provide data for the reward function in the lower control algorithm, so that the reward is closer to the ship's berthing situation. For example, the closer the ship is to the target berth, the greater the reward given. At the same time, if the ship's rudder angle change rate is smaller during the ship's journey in nearshore waters, the reward will also increase further. The lower motion control algorithm adopts a control strategy model for automatic berthing of ships. It determines the next state (target heading angle and target speed) according to the current state of the ship and the distance from the berth, and then uses the berthing controller to calculate the corresponding ship control quantity at the next moment, including the ship's rudder angle and speed.

[0071] In some embodiments, step S101 may include but is not limited to steps S201 to S205:

[0072] Step S201, collecting a second relative distance between the ship and the shore by means of a microwave radar array arranged on the ship;

[0073] Step S202, collecting the position of the ship through a positioning device installed on the ship;

[0074] Step S203, determining the first relative distance according to the first relative distance, the ship position and the target berth position;

[0075] Step S204, collecting motion status data through an electronic compass installed on the ship.

[0076] In this embodiment, the ship is provided with a microwave radar array, a positioning device (such as GPS) and an electronic compass. In the process of collecting the first relative distance between the ship and the target berth, the second relative distance between the ship and the shore is collected by the microwave radar array on the ship, the ship position is collected by the positioning device, the second relative distance is preliminarily determined according to the ship position and the target berth position, and the second relative distance is corrected by the first relative distance detected by the radar, so as to improve the accuracy of the ship positioning, and the electronic compass is used to realize the real-time collection of the ship's speed, heading angle and other motion state data.

[0077] According to some embodiments of the present application, the control strategy model in step S102 is obtained based on the training of the reinforcement learning algorithm. Figure 3 , the training process of the control strategy model may include but is not limited to the following steps:

[0078] Step S301, obtaining environmental interaction data;

[0079] Step S302, training the value network according to the environmental interaction data by maximizing the objective function, and training the strategy network according to the value network and the environmental interaction data, wherein the objective function represents the cumulative reward value of the action output by the strategy network, and the action represents the control amount of the ship;

[0080] Step S303: determine the control strategy model according to the trained strategy network.

[0081] In this embodiment, the control strategy model of the embodiment of the present application is obtained based on an improved proximal strategy optimization, that is, by interacting with a given environment, the optimal control strategy is found, and the strategy network is trained to take appropriate actions to execute a predetermined goal. The action of this embodiment is to control the rudder angle and the rotation speed, and the goal is to reach the designated berth. The proximal strategy optimization model based on reinforcement learning enables the model to achieve optimal control of the rudder angle and the rotation speed, achieve stable and good robust control, and can train the neural network to reach the berthing target point without the need for a large amount of actual data obtained as training data.

[0082] In step S301 of some embodiments, the environmental interaction data refers to the ship state transition sequence and corresponding actions collected in the actual environment or test environment, the ship state includes heading angle, speed and position, etc., and the action refers to the control quantity of the execution device, such as propeller speed, ship rudder angle, etc. The ship takes certain actions in the current state and switches to the next state. At this time, the next state can be regarded as the environmental feedback data of the action.

[0083] For details, please refer to Figure 4 Step S301 may include but is not limited to steps S401 to S404:

[0084] Step S401, obtaining berthing task information, wherein the berthing task information includes an environment map, a target berth and a reward function;

[0085] Step S402, constructing a reinforcement learning environment under the sub-process according to the ship mathematical model and the berthing task information;

[0086] Step S403, based on the reinforcement learning environment, the control strategy of the main process is used to guide the behavior of the intelligent agent, and the environmental feedback data of the sub-process is obtained;

[0087] Step S404, determining the environment interaction data according to the environment feedback data of each sub-process.

[0088] In this embodiment, the proximal policy optimization algorithm (PP0) will have the problem of strong correlation between each set of training data in actual application, which affects the convergence speed. Therefore, the PPO algorithm is optimized by adopting the parallel idea, and a multi-core working mode is adopted to realize multiple intelligent agents (i.e., intelligent ships) working simultaneously in different processes, and sub-processes are added. During training, multiple sub-processes are used to collect training data in the test environment at the same time, wherein all sub-processes share the global strategy (i.e., control strategy) of the main process to guide the behavior of the intelligent agent in exploring the environment, and after obtaining their respective training data, they are stored in the global data storage area, which stores the environmental feedback results obtained by multiple sub-processes, and uses the above results to update the strategy network and value network of the main process, suspending data collection during the update, and continuing to update the data with the updated new strategy after the update.

[0089] In some real-time, the mathematical model of the ship adopted in the reinforcement learning environment is as follows:

[0090] In a fixed coordinate system, based on Newtonian mechanics analysis and the law of angular momentum, the following mathematical model can be used to describe the forces acting on a ship during plane motion using the MMG model:

[0091]

[0092] Where X, Y, and N represent the forces received by the ship on the x-axis and y-axis and the moment around the z-axis; the subscripts H, P, R, wind, wave, and C represent the effects of the bare hull, oars, rudders, wind, waves, and currents; m is the mass of the ship, m x is the additional longitudinal mass of the ship, m y is the lateral additional mass, J ZZ is the additional moment of inertia, I ZZ is the moment of inertia, Yx C is the ship gravity center moment correction. From the above formula, it can be seen that the ship MMG model decomposes the fluid dynamics and moments acting on the propagation into the fluid dynamics acting on the bare hull, propeller and rudder, and the fluid dynamics and moments interfering with each other.

[0093] Among them, the state space of the ship's berthing and unberthing is 6-dimensional, and the action space is 2-dimensional, as follows:

[0094] s t =[n,u,e,v,p i , r] T ;

[0095] a t =[delta,rps] T ;

[0096] Where n represents the x-axis of the ship, u represents the x-axis speed, e represents the y-axis of the ship, v represents the y-axis speed, and pi represents the heading angle, r is the heading angular velocity of the ship, delta is the rudder angle, and rps is its rotation speed.

[0097] In step S302 of some embodiments, please continue to refer to Figure 4 After obtaining and storing the environmental feedback data from each sub-process and forming the environmental interaction data, the environmental interaction data is used as training data for the control strategy (i.e., policy network) training of the main process. Specifically, in each batch of training, the value network (i.e., Critic) is first trained using the training data. The value network can be used to evaluate the action distribution output by the policy network (Actor). Then, the policy network is trained using the value network and the training data. The policy network outputs the action distribution from the current state to the next state. After each training is completed, the obtained control strategy model can be evaluated, that is, it is determined whether the intelligent agent equipped with the control strategy model can complete the berthing task. If it can, the model is output as the optimal deterministic strategy, and the optimal strategy is used to control the ship to complete automatic berthing. If not, continue to collect environmental data for intensive training.

[0098] In some embodiments, step S302 may include but is not limited to steps S501 to S507:

[0099] Step S501, training a value network according to the current batch of training data in the environment interaction data;

[0100] Step S502, inputting the states in the current batch of training data into the strategy network to obtain the action distribution at the next moment;

[0101] Step S503, using the value network to select an action for the action distribution at the next moment to obtain a target action;

[0102] Step S504, determining the value of the objective function according to the state corresponding to the target action;

[0103] Step S505, determining whether the maximum value of the objective function is currently found according to the value of the objective function;

[0104] Step S506, when the maximum value of the objective function is not found, the parameters of the policy network are updated, and the value network and the policy network are continuously trained according to the next batch of training data in the environment interaction data;

[0105] Step S507: When the maximum value of the objective function is found, the current policy network is determined as the trained policy network.

[0106] In this embodiment, the ship berthing process is as follows: Figure 5As shown in the figure, the berthing path planning divides the entire ship berthing process into two parts. First, when the ship enters the turning circle, the ship needs to turn in the area and drive the ship to the guide line on the left. In the second part, the ship needs to drive along the guide line to approach the berth. When entering the deceleration circle, it needs to slowly decelerate to stop. There should be a safe distance S0 between the target berthing point and the berth shore. This distance can be 1.5 times the length of the ship. If the ship exceeds this safe distance, it should immediately decelerate to zero and turn around and drive away from the berth shore. The distance between the microwave radar array on the ship and the berth is S. The turning circle can be set to 12 times the length of the ship, and the deceleration circle can be 5 times the length of the ship. When the ship enters the turning circle, the constrained ship speed should be reduced to a certain value (such as 5KM / H), and slowly approach the guide line, and try to drive close to the guide line, which is conducive to the ship berthing. When the ship enters the deceleration circle along the guide line, the ship speed should be slowly reduced to a certain value (such as 1KM / H) so that the ship slowly enters the berth.

[0107] The automatic berthing problem of underactuated ships can be described as the following nonlinear optimal control problem with constraints, that is, the objective function to be optimized J(π θ ),as follows:

[0108]

[0109]

[0110] Among them, R(s k , a k ) is the reward function defined according to the task objective, and the objective function to be optimized J(π θ ) is consistent with maximizing the expected cumulative reward. According to J(π θ ) can be seen from the definition of it and the state value function are essentially the same, the decision variables of both are parameterized policies π θ The environmental dynamics are described deterministically. k+1 =f(s k , a k ). T is the set mission duration, as ships usually have time limits when berthing. k =[u k , v k ,φ k , X k , Y k ,ψ k ] T is the state vector, X k , Y k is the vessel position of the underdriven vessel, u k , v k is the longitudinal velocity and the lateral drift velocity, ψ kis the state vector of all subsystems such as heading, φ k is the yaw angular velocity, represents the control input of the system, is the propeller thrust, is the rudder torque. The system starts from a specific initial state s during the berthing process. I Start, and finally transfer to the target terminal state s under the control of the agent strategy F The state and action of the ship during navigation are subject to the constraints of boundary values. As mentioned above, the boundary value s of the state min and max , restraining the ship so that it will not hit the port and the shore, and cannot exceed the established speed and heading angular velocity, thereby ensuring the safety of the ship's movement.

[0111] In this embodiment, the ship berthing problem in the related art needs to first calculate a reasonable berthing trajectory by the motion planning layer, and then design a trajectory tracking controller to complete the berthing task. The layered berthing system divides the task into two independent subsystems, which are completed by two different algorithms. The improved proximal strategy optimization algorithm used in this embodiment is an end-to-end method. It only needs to provide a training environment and a corresponding reward function. The reinforcement learning algorithm can complete a series of functions such as optimal berthing path search, path optimization, and path tracking control. After multiple iterative training, the algorithm will optimize the intelligent agent based on historical training data. After the iterative training is completed, the intelligent agent can replace the planning layer and the control layer in the traditional method to complete the ship berthing alone.

[0112] The proximal policy optimization algorithm used in the model training process is a strategy for taking actions for a given state. As the training iterations continue, the strategy is continuously updated to maximize the reward, and finally the optimal strategy that maximizes the reward is reached. t In the case of deciding action a t , generating the next state s t+1 and the reward signal r from the environment t The above objective function is consistent with maximizing the expected cumulative reward. The specific meaning is: let the return of a certain period of time after time t be Gt, where γ is the discount rate, 0≤γ≤1, and is the weighting factor between the current accumulated reward and the accumulated reward in the future time step, then:

[0113]

[0114]

[0115] Among them, J(π) represents the objective function of maximizing the expected return, which means that the agent follows the behavior choice of the strategy, interacts with the environment by observing the state, takes action, and obtains the reward accumulated as G(τ).

[0116] Please refer to Figure 6 The model training process based on the proximal policy optimization algorithm uses two neural networks, Actor and Critic. The two networks have one thing in common, the input state S: one output strategy, responsible for selecting actions, this network is called Actor; the other is responsible for calculating the score of each action, this network is called Critic. The current state of the ship is input into the Actor network, and the output is the parameters mu and sigma that describe the probability distribution of the action. Then the V value output in the Critic network represents the value of the selected action. A normal distribution is constructed based on the selected mu and sigma, and then the action, namely the rudder angle and propeller speed, is randomly selected based on the probability distribution. The action is input to the ship, and the ship starts to run according to the instruction. It runs until the next moment and repeats the above operation until it slowly moves to the target berth and decelerates to 0.

[0117] In some embodiments, generally, finding a strategy that satisfies the constraints and can maximize the problem requires solving partial differential equations, which makes the problem difficult to solve. Some classical methods use numerical solutions to the Hamilton-Jacobi-Bellman equations, which usually involve complex numerical calculations. However, this embodiment uses data to solve nonlinear optimal control problems under the framework of the Markov decision process, which is more efficient. In order to ensure that the ship can berth as accurately as possible under the state constraints, this embodiment adopts the following reward function R(s k , a k ):

[0118]

[0119] Among them, β is the weight factor, α is a decimal greater than zero to ensure that the logarithmic function variable is not equal to zero, and the penalty term is a small negative number. When the state of the ship satisfies the constraints, maximizing the cumulative reward function can encourage the agent to minimize the current state s k and the target state s f When the ship's state violates the constraint, it will be punished.

[0120] When the ship state satisfies the constraint, the first term of the reward function will decrease as the state error decreases. When the ship distance approaches the target state, this term will approach zero. If the reward function only has this term, the ship will lack sufficient reward value feedback in the neighborhood of the target state. At this time, the second term of the function is needed to motivate the agent to berth accurately. If the ship state violates the constraint, the reward function will feedback a negative number to the agent as a penalty term. This situation mainly occurs when the ship speed is too fast or collides with the map boundary and the shore. Since the goal of reinforcement learning is to maximize the cumulative reward, when the state of the ship is always outside the feasible area, it will continue to receive rewards less than zero. The accumulated reward value will tend to negative infinity over time. Then the improved proximal policy optimization algorithm DDPG updates the agent according to the reward value to keep it away from the boundary. In addition, since the penalty value is generally set to a negative number far less than zero, even if the system violates the constraint and quickly returns to the feasible domain, the agent will be punished to a certain extent.

[0121] In some embodiments, the concept of value function is commonly used in reinforcement learning algorithms, by t Start following the actions of policy π to describe the estimated value of the payoff

[0122] Advantage function Represents the deviation of behavior a from the mean in state s.

[0123]

[0124] Improved proximal policy optimization uses multi-process learning. The agent not only interacts with the environment to generate gradients that are aggregated to the global network, but also learns from these data. After the agent aggregates the gradients and applies them to the parameters of the global network, the global network will directly give the learned latest version of the parameters to the agent, thereby speeding up training. When the proximal policy optimization algorithm updates the behavior strategy, it always uses the same stage method to process the proxy target. The initial truncated proxy target is:

[0125]

[0126] Among them, ∈ represents the truncation hyperparameter, which is generally set to 0.2; clip() represents the truncation function, which is responsible for limiting the ratio r t In the interval [1-∈, 1+∈], to ensure convergence; finally L clip Use the min function to select the smaller value between the untruncated and truncated targets to form the target lower limit. If the advantage function is positive, it is necessary to increase the ratio of the new and old strategies r t , however, when r t > 1+∈, no additional incentive will be provided: If the advantage function is negative, it is necessary to reduce the ratio of the new and old strategies rt , but in r t <1-∈, no additional incentive is provided, which limits the difference between the new and old strategies to a reasonable range.

[0127] Most reinforcement learning algorithms used for ship berthing and unberthing have difficulty in choosing the learning rate when dealing with continuous action spaces such as berthing. If the learning rate is too small, it will lead to poor convergence of reinforcement learning and a situation where training cannot be completed. If the learning rate is too large, it will lead to inconsistent data when iterating between the old and new strategies, resulting in large learning fluctuations or local oscillations. The PPO algorithm used in this embodiment alternates between sampling data and using the stochastic gradient ascent method to optimize the target function, which can achieve small batch updates. During iterative updates, the probability π(a of the action taken by the agent in state s at time t is observed by observing the current strategy. t |s t ), and the probability of the action taken by the previous strategy π θold (a t |s t ), calculate the probability ratio to control the update amplitude of the new strategy, the ratio r t Denoted as:

[0128] Through a large number of analyses of the training results of the inland container ship automatic berthing scenario, it can be seen that after a certain number of training rounds (i.e. a certain number of training times), the reward values ​​of subsequent training rounds of the actual algorithm will gradually stabilize in a higher range, indicating that the behavior network and evaluation network of the algorithm have gradually become perfect and gradually adapted to the current training scenario. Based on this embodiment, an enhanced truncation function is proposed, which can increase the subsequent learning rate increase rate to increase the current algorithm model. The improved stage proxy target is:

[0129]

[0130] Due to the use of e r -1 replaces the original r, increases the slope of the truncation function in the truncation interval, and in order to avoid problems such as overestimation and avoid having a higher learning rate outside the truncation interval, this embodiment changes the domain of the truncation interval from the original [1-∈, 1+∈] to [0.69-∈, 0.69+∈], so that the slope of the improved truncation function increases within the domain and the range remains unchanged. It can be understood that the domain of the truncation interval in this embodiment can be [0.69-∈, 0.69+∈], or other numerical intervals, as long as it is smaller than the range of [1-E, 1+∈].

[0131] In step S303 of some embodiments, please continue to refer to Figure 4At the beginning, the berthing task information of the ship will be obtained, such as the coordinate points of the ship, map information, target berth, reward function and other information, and the reinforcement learning environment will be constructed through the above information. During training, the Actor neural network forward propagates according to the state of the current environment to calculate the current action to be taken. Then the environment will transfer to the next state under the action and feedback to the agent for immediate reward. The above data will be used to train the agent, among which the Critic network provides the gradient information necessary for the Actor network to update. As learning continues, the agent's strategy will tend to the optimal strategy. When it is actually applied or the training effect is tested, the trained Actor network can be directly deployed as a control strategy model to control the ship's berthing.

[0132] According to some embodiments of the present application, the present embodiment adopts the automatic berthing of the ship based on the proximal strategy optimization design of reinforcement learning, interacts with the given environment, and iterates through continuous learning. The reward for each learning training is formed according to the reward function until the total cumulative reward of the strategy reaches the maximum value, and the control strategy model for actual berthing is obtained. The training optimization model based on reinforcement learning adopted in this implementation does not need to obtain a large amount of training data in advance, but obtains data by interacting with the given environment. The model training process of this embodiment is not restricted by having a large amount of training data, and the model optimization can be achieved as long as the environmental interaction continues. In addition, the embodiment of the present application can achieve the desired maneuvering behavior without any predictability of the ship's maneuverability through the careful definition of the reward function. The results show that the strategy control model can learn how to robustly maneuver the ship from various initial positions to the berthing target point. The method of the embodiment of the present application can increase the safety of the ship's berthing and unberthing process, so that the ship can be docked at a reasonable berthing position, thereby allowing the surface ship to travel in a standardized manner and reduce the safety accident rate.

[0133] The present application also provides a ship automatic berthing control system, comprising:

[0134] The first module is used to obtain the motion state data of the current ship and the first relative distance to the target berth, wherein the motion state data includes the speed of the ship;

[0135] The second module is used to input the motion state data and the first relative distance into a control strategy model for automatic berthing of a ship, and obtain the ship control quantity at the next moment, wherein the ship control quantity includes the ship rudder angle and rotation speed;

[0136] The control strategy model is trained based on the reinforcement learning algorithm, and the training process of the control strategy model includes the following steps:

[0137] Get environmental interaction data;

[0138] By maximizing the objective function, the value network is trained according to the environmental interaction data, and the policy network is trained according to the value network and the environmental interaction data, wherein the objective function represents the cumulative reward value of the action output by the policy network, and the action represents the control amount of the ship;

[0139] Determine the control strategy model based on the trained strategy network.

[0140] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0141] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned ship automatic berthing control method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0142] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0143] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0144] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the ship automatic berthing control method of the embodiment of this application;

[0145] Input / output interface 703, used to implement information input and output;

[0146] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0147] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0148] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0149] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned ship automatic berthing control method.

[0150] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0151] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0152] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0153] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0154] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0155] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0156] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0158] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0161] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for controlling automatic berthing of a ship, characterized in that: The following steps are involved: Acquire motion state data of the current ship and a first relative distance from the target berth, wherein the motion state data includes the ship speed; Inputting the motion state data and the first relative distance into a control strategy model for automatic berthing of a ship to obtain a ship control variable at the next moment, wherein the ship control variable includes a rudder angle and a rotation speed of the ship; The control strategy model is trained based on a reinforcement learning algorithm, and the training process of the control strategy model includes the following steps: Get environmental interaction data; By maximizing an objective function, a value network is trained according to the environmental interaction data, and a strategy network is trained according to the value network and the environmental interaction data, wherein the objective function represents a cumulative reward value of an action output by the strategy network, and the action represents a control amount of the ship; Determine the control strategy model based on the trained strategy network.

2. The ship automatic berthing control method according to claim 1, characterized in that: The step of obtaining the motion state data of the current ship and the first relative distance to the target berth comprises the following steps: The second relative distance between the ship and the shore is collected by a microwave radar array arranged on the ship; The position of the ship is collected by a positioning device installed on the ship; Determining a first relative distance according to the first relative distance, the ship position and the target berth position; The motion status data is collected through the electronic compass installed on the ship.

3. The ship automatic berthing control method according to claim 1, characterized in that: The step of obtaining the environment interaction data comprises the following steps: Acquiring berthing task information, wherein the berthing task information includes an environment map, a target berth, and a reward function; Constructing a reinforcement learning environment under a subprocess according to a ship mathematical model and the berthing task information; Based on the reinforcement learning environment, the control strategy of the main process is used to guide the behavior of the intelligent agent, and the environmental feedback data of the sub-process is obtained; The environmental interaction data is determined based on the environmental feedback data under each sub-process.

4. The ship automatic berthing control method according to claim 1, characterized in that: The objective function is expressed as follows: Among them, R(s k ,a k ) represents the defined reward function, T is the set docking mission duration, s k =[u k ,v k ,ψ k ,X k ,Y k ,ψ k ] T is the state vector, X k , Y k Indicates the ship's position, u k ,v k They represent the longitudinal velocity and the lateral drift velocity, ψ k Indicates the ship's heading, φ k Indicates the yaw angular velocity, action represents the ship control quantity, The thrust of the propeller, represents the rudder moment; s I represents the initial state of the ship, s F represents the target terminal state achieved under the control of the policy network; s min and max Respectively represent the minimum state and the maximum state, a min and a max Represent the minimum and maximum values ​​of the action respectively.

5. The ship automatic berthing control method according to claim 4, characterized in that: The reward function is expressed as: Among them, β represents the weight factor, α is a coefficient greater than zero, and s k Indicates the current state, s F represents the target termination state, and C represents a constant.

6. The ship automatic berthing control method according to claim 1, characterized in that: The method of training a value network according to the environment interaction data by maximizing the objective function and training a strategy network according to the value network and the environment interaction data comprises the following steps: Training a value network based on the current batch of training data in the environment interaction data; Inputting the states in the current batch of training data into the strategy network to obtain the action distribution at the next moment; The value network is used to select an action for the action distribution at the next moment to obtain a target action; Determining the value of the objective function according to the state corresponding to the target action; Determine whether the maximum value of the objective function is currently found according to the value of the objective function; When the maximum value of the objective function is not found, the parameters of the policy network are updated, and the value network and the policy network are continuously trained according to the next batch of training data in the environment interaction data; When the maximum value of the objective function is found, the current policy network is determined as the trained policy network.

7. The ship automatic berthing control method according to claim 6, characterized in that: The updating of the parameters of the policy network comprises the following steps: Determine the current number of training sessions; Selecting a corresponding proxy target according to the number of training times to guide the parameter update of the policy network, wherein when the number of training times is less than a preset number of times, selecting a first proxy target; when the number of training times is greater than or equal to the preset number of times, selecting a second proxy target; The first proxy target is expressed as follows: The second proxy target is expressed as follows: Among them, ∈ represents the truncation hyperparameter, clip() represents the truncation function, r t represents the ratio of the new and old strategies, represents the advantage function.

8. A ship automatic berthing control system, characterized in that: include: The first module is used to obtain the motion state data of the current ship and the first relative distance to the target berth, wherein the motion state data includes the speed of the ship; The second module is used to input the motion state data and the first relative distance into a control strategy model for automatic berthing of a ship to obtain a ship control quantity at the next moment, wherein the ship control quantity includes a rudder angle and a rotation speed of the ship; The control strategy model is trained based on a reinforcement learning algorithm, and the training process of the control strategy model includes the following steps: Get environmental interaction data; By maximizing an objective function, a value network is trained according to the environmental interaction data, and a strategy network is trained according to the value network and the environmental interaction data, wherein the objective function represents a cumulative reward value of an action output by the strategy network, and the action represents a control amount of the ship; Determine the control strategy model based on the trained strategy network.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 7.

Images