Intelligent matching method of navigation control parameters of unmanned ships based on deep reinforcement learning
Through the combination of deep reinforcement learning and fractional-order PID controllers, the navigation parameters of unmanned ships are automatically adjusted, which solves the problem of manual parameter adjustment in the existing technology that relies on experience, and improves the navigation efficiency and control accuracy of unmanned ships in complex environments. sex.
Patent Information
- Application Number
- CN202311333638.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-10-16
AI Technical Summary
The adjustment of the navigation control parameters of unmanned ships is affected by a variety of factors. The existing methods rely on experience to manually adjust parameters, which is time-consuming and labor-intensive, making it difficult to achieve efficient navigation in complex environments.
The intelligent matching method of unmanned ship navigation control parameters based on deep reinforcement learning is adopted. By obtaining the actual navigation status measurement data, state variables are constructed and pre-trained reinforcement learning network model are input, control strategies are generated, and navigation parameters are adjusted using fractional-order PID controller. Combined with the reward function, optimize the control strategy model.
It has achieved the improvement of navigation efficiency and generalization capabilities of unmanned ships in complex environments, automated parameter adjustment, adapted to different ship types and environments, and improved the accuracy and efficiency of navigation control.
Smart Images

Figure CN117170389B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of unmanned ships, and in particular to an intelligent matching method for navigation control parameters of unmanned ships based on deep reinforcement learning. Background Art
[0002] Unmanned ships are a typical application of automatic control and information technology in the field of ships. Compared with manned ships, unmanned ships have the advantages of flexibility, all-weather operation, strong environmental adaptability, etc. They are widely used in engineering fields such as maritime patrol, mine detection and anti-submarine, environmental monitoring, and marine resource exploration. Navigation control technology is the core key technology of unmanned ships and the basis for them to complete various complex tasks.
[0003] The complexity of the navigation environment brings many challenges to the navigation control of unmanned ships. First, the movement of the unmanned ship itself has the characteristics of strong inertia and high hysteresis, and its motion modeling is relatively difficult; secondly, the water surface environment is complex and changeable, and factors such as wind, waves and currents often cause coupling interference to the movement of unmanned ships. This puts higher requirements on the navigation control of unmanned ships. Currently, the commonly used navigation control methods can be divided into model-based methods and error-based methods. Model-based methods rely on the accuracy of modeling and have poor effects in engineering applications. Error-based methods are usually simple in structure, model-independent, and easy to implement, so they are widely used in various industrial control systems.
[0004] However, in the error-based navigation control method of unmanned ships, the adjustment of navigation control parameters is affected by many factors, such as the ship type coefficient, the complexity of the navigation waters, the navigation mission objectives, the power system constraints, etc. Most of the existing methods are manual parameter adjustment based on experience, which is time-consuming and labor-intensive. This greatly affects the navigation efficiency and application scope of unmanned ships in complex environments. Summary of the invention
[0005] In response to the above problems and technical requirements, the applicant proposes an intelligent matching method for unmanned ship navigation control parameters based on deep reinforcement learning. The technical solution of this application is as follows:
[0006] A method for intelligently matching navigation control parameters of an unmanned ship based on deep reinforcement learning, the method comprising:
[0007] Obtaining measured data on the navigation status of the unmanned ship to be controlled and its environment;
[0008] The state variables are constructed based on the measured data of the navigation state, and the state variables are input into the control strategy model to obtain the output control strategy. The control strategy model is a reinforcement learning network model based on the proximal strategy optimization method obtained by pre-training;
[0009] The navigation control parameters of the unmanned ship to be controlled are obtained based on the control strategy, and the unmanned ship to be controlled is controlled to navigate along the target track according to the navigation control parameters.
[0010] A further technical solution is that the control strategy model outputs a control strategy according to the input state variables, including:
[0011] Determine the current navigation phase of the unmanned ship to be controlled according to the real-time position and real-time speed of the unmanned ship to be controlled included in the measured navigation status data, combined with the target track and target speed of the unmanned ship to be controlled;
[0012] Determine and output the control strategy corresponding to the state variable according to the strategy mapping relationship corresponding to the navigation phase;
[0013] Among them, different navigation stages correspond to different strategy mapping relationships, the same state variables correspond to different control strategies in different strategy mapping relationships, and a set of control strategies corresponding to the strategy mapping relationship of each navigation stage are used to control the unmanned ship to be controlled to adjust the navigation state to give priority to meeting the control requirements corresponding to the navigation stage.
[0014] A further technical solution is that the navigation phase of the unmanned ship to be controlled includes a navigation adjustment phase, a stable tracking phase and a navigation end phase:
[0015] When the deviation value E between the real-time position of the unmanned ship to be controlled and the end point of the target track dis When it is not greater than the terminal deviation threshold, it is determined that the unmanned ship to be controlled is in the end stage of navigation;
[0016] When the track end point deviation value E between the real-time position of the unmanned ship to be controlled and the track end point of the target track dis The position deviation value E between the actual position of the unmanned ship to be controlled and the target track is greater than the endpoint deviation threshold, and the position deviation value E between the actual position of the unmanned ship to be controlled and the target track pos The speed deviation value E between the speed of the unmanned ship to be controlled and the target speed is less than the track deviation threshold. vel When the speed is less than the speed deviation threshold, it is determined that the unmanned ship to be controlled is in the stable tracking stage;
[0017] Otherwise, it is determined that the unmanned ship to be controlled is in the navigation adjustment stage.
[0018] A further technical solution is to obtain the navigation control parameters of the unmanned ship to be controlled based on the control strategy, including:
[0019] A fractional-order PID controller is constructed according to the control strategy. The navigation error is determined according to the measured data of the navigation state of the unmanned ship to be controlled and the target track. Based on the navigation error, the fractional-order PID controller is used to obtain the navigation control parameters of the unmanned ship to be controlled.
[0020] A further technical solution is that the navigation control parameter u(t) at any time t = K p e(t)+K i Ie(t) λ +K d De(t) μ ;
[0021] Among them, e(t) is the navigation error between the real-time position at time t contained in the actual navigation state data and the corresponding target position in the target track, Ie(t) is the integral of the navigation error e(t), and De(t) is the differential of the navigation error e(t); the parameter array [K p ,K i ,K d ,λ,μ] is generated according to the control strategy, K p is the proportional term of the constructed fractional-order PID controller, K i is the integral term of the constructed fractional-order PID controller, K d is the differential term of the constructed fractional-order PID controller, λ is the order of the integral term of the constructed fractional-order PID controller, and μ is the order of the differential term of the constructed fractional-order PID controller.
[0022] Its further technical solution is that the method further comprises:
[0023] Establish a reinforcement learning network model based on the proximal strategy optimization method and initialize the network parameters;
[0024] Obtaining a sample state variable of the sample unmanned ship, the sample state variable including measured data of the navigation state of the sample unmanned ship;
[0025] The sample state variables of the sample unmanned ship are input into the constructed reinforcement learning network model to obtain the control strategy;
[0026] Based on the control strategy, the navigation control parameters of the sample unmanned ship are obtained and the movement of the unmanned ship to be controlled is controlled to change the measured data of the navigation state, and the reward value is calculated based on the measured data of the navigation state before and after the change of the sample unmanned ship according to the reward function;
[0027] Repeat the steps of converting the sample state variables of the sample unmanned ship to construct multiple sets of sample data, each set of sample data including the sample state variables, the control strategy and the corresponding reward value;
[0028] The reinforcement learning network model is trained using multiple groups of sample data until a control strategy model is obtained. The control strategy model reflects the strategy mapping relationship between the state variables and the control strategy that optimizes the reward value.
[0029] Its further technical solution is that the reward function is calculated based on multiple evaluation indicators of the sample unmanned ship, each evaluation indicator is calculated based on the measured data of the navigation status of the sample unmanned ship before and after the change, different navigation stages correspond to different reward functions, and the reward value is calculated according to the reward function based on the measured data of the navigation status of the sample unmanned ship before and after the change, including: the reward value is calculated based on the measured data of the navigation status of the sample unmanned ship before and after the change according to the reward function corresponding to the current navigation stage of the sample unmanned ship; the trained control strategy model reflects the strategy mapping relationship between the state variables and the control strategy that makes the reward value optimal in each navigation stage.
[0030] A further technical solution is that the reward functions of different navigation phases include different evaluation indicators, and the evaluation indicators included in the reward function of each navigation phase match the control requirements of the navigation phase.
[0031] Its further technical solution is that the evaluation indicators used by the reward function corresponding to the navigation adjustment stage include position deviation value and speed deviation value, the position deviation value is the distance deviation value between the real-time position included in the navigation state measured data and the target track, and the speed deviation value is the deviation value between the real-time speed included in the navigation state measured data and the target speed;
[0032] The evaluation indicators used in the reward function corresponding to the stable tracking stage include the heading deviation angle and the rudder angle. The heading deviation angle is the deviation value between the heading included in the measured data of the navigation state and the heading of the target track;
[0033] The evaluation index used by the reward function corresponding to the end stage of navigation includes the track end point deviation value, which is the distance deviation value between the real-time position included in the measured data of the navigation status and the track end point of the target track.
[0034] Its further technical solution is that the reward function corresponding to the navigation adjustment stage is E pos (t-1) is the position deviation value calculated based on the measured data of the navigation status before the change, E vel (t-1) is the speed deviation value calculated based on the measured data of the navigation state before the change, E pos (t) is the position deviation value calculated based on the measured data of the changed navigation state, E vel (t) is the speed deviation value calculated based on the measured data of the changed navigation state;
[0035] The reward function corresponding to the stable tracking stage is E phi (t-1) is the heading deviation angle calculated based on the measured data of the navigation state before the change, δ(t-1) is the rudder angle included in the measured data of the navigation state before the change, Ephi (t) is the heading deviation angle calculated based on the measured data of the changed navigation state, and δ(t) is the rudder angle included in the measured data of the changed navigation state;
[0036] The reward function corresponding to the end of the voyage is: E dis (t-1) is the track end point deviation value calculated based on the measured data of the navigation state before the change, E dis (t) is the track end point deviation value calculated based on the measured data of the changed navigation status.
[0037] The beneficial technical effects of this application are:
[0038] The present application discloses an intelligent matching method for navigation control parameters of an unmanned ship based on deep reinforcement learning. This method aims to solve the problem that the navigation control parameters of the unmanned ship are difficult to adjust. By designing a PPO reinforcement learning network model, the control strategies under different state variables are automatically learned using test data to obtain the navigation control parameters. This method can realize adaptive matching of navigation control parameters for different ship types and different environments, and directly copy the parameter adjustment experience to other ship types or environments, thereby effectively improving the generalization ability of the navigation controller and the navigation efficiency of the unmanned ship in complex environments.
[0039] When learning and training the control strategy model, this method also sets a hierarchical reward function, so as to meet different navigation control requirements, thereby adapting to the adaptive matching of navigation control parameters for different navigation tasks, and the matched navigation control parameters are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a method flow chart of an unmanned ship navigation control parameter intelligent matching method according to an embodiment of the present application.
[0041] Figure 2 It is a flow chart of a method for training a control strategy model in one embodiment of the present application. DETAILED DESCRIPTION
[0042] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.
[0043] This application discloses an intelligent matching method for unmanned ship navigation control parameters based on deep reinforcement learning. Please refer to Figure 1 The method flow chart shown in the figure comprises the following steps:
[0044] Step 1: Obtain the measured data of the navigation status of the unmanned ship to be controlled.
[0045] The measured data of the navigation status is used to reflect the real-time measured data of the unmanned ship to be controlled and its environment, and can be obtained through various sensors carried on the unmanned ship to be controlled. In one embodiment, the measured data of the navigation status of the unmanned ship to be controlled includes the navigation data of the unmanned ship to be controlled and the environmental wind and wave data. The navigation data of the unmanned ship to be controlled includes at least the real-time position, speed, heading, rudder angle and rotation speed of the unmanned ship. The environmental wind and wave data includes the interference force and interference torque of wind and waves on the controlled unmanned ship, which can be calculated through the measured parameters such as wind speed, wind direction, wave height, wave direction and so on acting on the unmanned ship to be controlled.
[0046] Step 2: construct state variables based on the measured data of navigation status, and input the state variables into the control strategy model to obtain the output control strategy.
[0047] When the unmanned ship to be controlled completes a navigation task, its target track is also known in advance, indicating the moving path from the starting position to the end position.
[0048] The control strategy model is a pre-trained reinforcement learning network model based on the proximal strategy optimization method. The control strategy model reflects the strategy mapping relationship between state variables and control strategies. Therefore, after the state variables are input into the control strategy model, the corresponding control strategy can be mapped.
[0049] Step 3, based on the control strategy, obtain the navigation control parameters of the unmanned ship to be controlled, and control the unmanned ship to be controlled to navigate along the target track according to the obtained navigation control parameters. When the unmanned ship to be controlled completes a navigation task, its target track is known in advance, which can be set manually according to the actual task, or automatically generated using a route planning algorithm.
[0050] Obtaining navigation control parameters of the unmanned ship to be controlled based on the control strategy includes: constructing a fractional-order PID controller according to the output control strategy: generating a parameter array [K p ,K i ,K d ,λ,μ],K p is the proportional term of the constructed fractional-order PID controller, K i is the integral term of the constructed fractional-order PID controller, K d is the differential term of the constructed fractional-order PID controller, λ is the order of the integral term of the constructed fractional-order PID controller, and μ is the order of the differential term of the constructed fractional-order PID controller.
[0051] The navigation error e(t) is determined according to the measured data of the navigation state of the unmanned ship to be controlled and the target track. Then, the navigation control parameters of the unmanned ship to be controlled are obtained by using the fractional-order PID controller based on the navigation error. The navigation control parameter u(t) at any time t is K p e(t)+K i Ie(t) λ +K d De(t) μ , e(t) is the navigation error between the real-time position at time t contained in the measured navigation status data and the corresponding target position in the target track, Ie(t) is the integral of the navigation error e(t), and De(t) is the differential of the navigation error e(t).
[0052] The method of the present application requires the use of a control strategy model in the above step 2, so the method also includes the step of pre-training the control strategy model, please refer to Figure 2 The flowchart shown includes the following steps:
[0053] (1) Establish a reinforcement learning network model based on the proximal strategy optimization method and initialize the network parameters.
[0054] (2) Obtaining sample state variables of the sample unmanned ship. The sample state variables include the main parameters of the hull of the sample unmanned ship, the measured data of the navigation state and the target track, which are similar to the meanings of the various parameters of the unmanned ship to be controlled mentioned above.
[0055] (3) The sample state variables of the sample unmanned boat are input into the constructed reinforcement learning network model to obtain the control strategy.
[0056] (4) Based on the control strategy, the navigation control parameters of the sample unmanned ship are obtained and the movement of the unmanned ship to be controlled is controlled to change the measured data of the navigation state. The method for obtaining the navigation control parameters of the sample unmanned ship based on the control strategy is as described in step 3 above, and this embodiment will not be repeated here.
[0057] (5) The reward value is calculated based on the measured data of the navigation status of the sample unmanned boat before and after the change according to the reward function.
[0058] The reward function is calculated based on multiple evaluation indicators of the sample unmanned ship, and each evaluation indicator is calculated based on the measured data of the navigation state of the sample unmanned ship before and after the change. In one embodiment, the evaluation indicators of the sample unmanned ship include the navigation error between the actual position and the target track and the navigation stability.
[0059] (6) Repeat the steps of converting the sample state variables of the sample unmanned boat to construct multiple groups of sample data, each group of sample data includes the sample state variables, the control strategy and the corresponding reward value.
[0060] (7) Use multiple sets of sample data to train the reinforcement learning network model and adjust the network parameters of the reinforcement learning network model. Repeat steps (2) to (6) until the convergence condition is reached and the control strategy model is trained. The control strategy model finally trained reflects the strategy mapping relationship between the state variables and the control strategy that makes the reward value optimal. After the control strategy model is trained, it can be used in practice.
[0061] In one embodiment, considering that different navigation stages have different control requirements, different navigation stages correspond to different reward functions. In this embodiment, the navigation stage of the unmanned ship to be controlled can be determined according to the actual position and speed of the unmanned ship to be controlled, combined with the deviation between the actual position and speed and the target track and target speed of the unmanned ship to be controlled, including the navigation adjustment stage, the stable tracking stage and the navigation end stage:
[0062] (1) Stable tracking stage: When the position deviation value E between the actual position of the unmanned ship to be controlled and the target track pos is less than the track deviation threshold, and the speed deviation value E between the speed of the unmanned ship to be controlled and the target speed vel is less than the speed deviation threshold, and the track end point deviation value E between the real-time position of the unmanned ship to be controlled and the track end point of the target track dis When it is greater than the endpoint deviation threshold, it is determined that the unmanned ship to be controlled is in the stable tracking stage.
[0063] Among them, three thresholds can be customized. A common setting method is: the track deviation threshold is set to one times the ship length, the speed deviation threshold is set to 0.1 times the maximum speed, and the end point deviation threshold is set to 2 times the stopping distance.
[0064] (2) Navigation end stage: When the deviation value E between the real-time position of the unmanned ship to be controlled and the end point of the target track is dis When it is not greater than the terminal deviation threshold, it is determined that the unmanned ship to be controlled is in the end stage of navigation.
[0065] (3) When none of the above conditions are met, it is determined that the unmanned ship to be controlled is in the navigation adjustment stage.
[0066] Different navigation stages have different control requirements. The control requirements corresponding to the navigation adjustment stage give priority to the position deviation between the unmanned ship to be controlled and the target track. The control requirements corresponding to the stable tracking stage give priority to the navigation stability of the unmanned ship to be controlled. The control requirements corresponding to the end of navigation stage give priority to the arrival error of the unmanned ship to be controlled relative to the target point. Different reward functions are set for different navigation stages. Then, in the above step (4) of the training process, the reward value is calculated based on the measured data of the navigation state before and after the change of the sample unmanned ship according to the reward function corresponding to the current navigation stage of the sample unmanned ship. The control strategy model obtained by training reflects the strategy mapping relationship between the state variables and the control strategy that makes the reward value optimal in each navigation stage. Different navigation stages correspond to different strategy mapping relationships, and the same state variables correspond to different control strategies in different strategy mapping relationships.
[0067] The reward functions for different navigation phases include different evaluation indicators, and the evaluation indicators included in the reward function for each navigation phase match the control requirements of the navigation phase. The evaluation indicators used by the reward function corresponding to the navigation adjustment phase include position deviation value and speed deviation value. The position deviation value is the distance deviation value between the real-time position included in the measured data of the navigation status and the target track. The speed deviation value is the deviation value between the real-time speed included in the measured data of the navigation status and the target speed. The evaluation indicators used by the reward function corresponding to the stable tracking phase include heading deviation angle and rudder angle. The heading deviation angle is the deviation value between the heading included in the measured data of the navigation status and the heading of the target track. The evaluation indicators used by the reward function corresponding to the navigation end phase include track end point deviation value. The track end point deviation value is the distance deviation value between the real-time position included in the measured data of the navigation status and the track end point of the target track.
[0068] In one embodiment, different reward functions corresponding to different navigation phases include:
[0069] (1) Navigation adjustment phase, the corresponding reward function is:
[0070]
[0071] The control requirement of this navigation phase is to make the real-time position of the unmanned ship to be controlled close to the target track and the real-time speed close to the target track. Therefore, the evaluation indicators used in the corresponding reward function include the position deviation value E calculated based on the measured data of the navigation state before the sample unmanned ship changes. pos (t-1) and speed deviation value E vel (t-1), and the position deviation value E calculated based on the measured data of the navigation state after the sample unmanned ship changes pos (t) and speed deviation value E vel (t).
[0072] (2) In the stable tracking phase, the corresponding reward function is:
[0073]
[0074] The control requirement of this navigation phase is to make the heading of the unmanned ship to be controlled consistent with the target track direction and minimize the rate of change of the rudder angle. Therefore, the evaluation indicators used in the corresponding reward function include the heading deviation angle E calculated based on the measured data of the navigation state of the sample unmanned ship before the change. phi (t-1) and rudder angle δ(t-1), as well as the heading deviation angle E calculated based on the measured data of the navigation state after the sample unmanned ship changes phi (t) and rudder angle δ(t). The heading deviation angle is the deviation between the real-time heading of the unmanned ship to be controlled and the heading of the target track.
[0075] (3) At the end of the voyage, the corresponding reward function is:
[0076]
[0077] The control requirement of this navigation phase is to minimize the distance between the unmanned ship to be controlled and the end point of the target track. Therefore, the evaluation index used in the corresponding reward function includes the end point deviation value E calculated based on the measured data of the navigation state before the sample unmanned ship changes. dis (t-1), and the track end point deviation value E calculated based on the measured data of the navigation state after the sample unmanned ship changes dis (t).
[0078] Correspondingly, in the actual application process, the real-time position of the unmanned ship to be controlled included in the measured data of the navigation state is combined with the target track to determine the current navigation stage of the unmanned ship to be controlled, and then the control strategy corresponding to the state variable is determined and output according to the strategy mapping relationship corresponding to the navigation stage. A set of control strategies corresponding to the strategy mapping relationship of the state variables in each navigation stage are used to control the unmanned ship to be controlled to adjust the navigation state to give priority to the control requirements corresponding to the navigation stage.
[0079] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.
Claims
1. An intelligent matching method for navigation control parameters of an unmanned ship based on deep reinforcement learning, characterized in that: The method comprises: Obtaining measured data on the navigation status of the unmanned ship to be controlled and its environment; constructing state variables based on the measured navigation state data, and inputting the state variables into a control strategy model to obtain an output control strategy, wherein the control strategy model is a reinforcement learning network model obtained by pre-training based on a proximal strategy optimization method; Obtaining navigation control parameters of the unmanned ship to be controlled based on the control strategy, and controlling the unmanned ship to be controlled to navigate along a target track according to the navigation control parameters; The control strategy model outputs a control strategy according to the input state variables, including: According to the real-time position and real-time speed of the unmanned ship to be controlled included in the measured navigation status data, the current navigation stage of the unmanned ship to be controlled is determined in combination with the target track and target speed of the unmanned ship to be controlled. The navigation stage of the unmanned ship to be controlled includes a navigation adjustment stage, a stable tracking stage and a navigation end stage: when the deviation value between the real-time position of the unmanned ship to be controlled and the end point of the target track is When the end point deviation value between the real-time position of the unmanned ship to be controlled and the end point of the target track is less than the end point deviation threshold, it is determined that the unmanned ship to be controlled is in the end stage of navigation; when the end point deviation value between the real-time position of the unmanned ship to be controlled and the end point of the target track is less than the end point deviation threshold, it is determined that the unmanned ship to be controlled is in the end stage of navigation; The position deviation value between the actual position of the unmanned ship to be controlled and the target track is greater than the endpoint deviation threshold The speed deviation value between the speed of the unmanned ship to be controlled and the target speed is less than the track deviation threshold. When the speed is less than the speed deviation threshold, it is determined that the unmanned ship to be controlled is in the stable tracking stage; otherwise, it is determined that the unmanned ship to be controlled is in the navigation adjustment stage; the control strategy corresponding to the state variable is determined and output according to the strategy mapping relationship corresponding to the navigation stage; wherein different navigation stages correspond to different strategy mapping relationships, and the same state variable corresponds to different control strategies in different strategy mapping relationships, and a group of control strategies corresponding to the strategy mapping relationship of each navigation stage are used to control the unmanned ship to be controlled to adjust the navigation state to give priority to meeting the control requirements corresponding to the navigation stage; The method also includes: establishing a reinforcement learning network model based on a proximal strategy optimization method and initializing network parameters; obtaining sample state variables of a sample unmanned ship, the sample state variables including measured navigation state data of the sample unmanned ship; inputting the sample state variables of the sample unmanned ship into the constructed reinforcement learning network model to obtain a control strategy; obtaining navigation control parameters of the sample unmanned ship based on the control strategy and controlling the movement of the unmanned ship to be controlled to change the measured navigation state data, and calculating a reward value based on the measured navigation state data of the sample unmanned ship before and after the change according to a reward function; repeatedly executing the steps of inputting the sample state variables of the sample unmanned ship to construct multiple groups of sample data, each group of sample data including sample state variables, control strategies and corresponding reward values; using multiple groups of sample data to perform model training on the reinforcement learning network model until the control strategy model is obtained through training, wherein the control strategy model reflects the strategy mapping relationship between the state variables and the control strategies that optimize the reward value; The reward function is calculated based on multiple evaluation indicators of the sample unmanned ship, each evaluation indicator is calculated based on the measured data of the navigation status before and after the change of the sample unmanned ship, and different navigation stages correspond to different reward functions. The reward value is calculated based on the measured data of the navigation status before and after the change of the sample unmanned ship according to the reward function, including: the reward value is calculated based on the measured data of the navigation status before and after the change of the sample unmanned ship according to the reward function corresponding to the current navigation stage of the sample unmanned ship; the control strategy model obtained by training reflects the strategy mapping relationship between the state variables and the control strategy that makes the reward value optimal in each navigation stage; the reward functions of different navigation stages include different evaluation indicators, and the evaluation indicators included in the reward function of each navigation stage match the control requirements of the navigation stage: The evaluation indicators used in the reward function corresponding to the navigation adjustment stage include position deviation value and speed deviation value. The position deviation value is the distance deviation value between the real-time position included in the navigation status measured data and the target track. The speed deviation value is the deviation value between the real-time speed included in the navigation status measured data and the target speed. The reward function corresponding to the navigation adjustment stage is , It is the position deviation value calculated based on the measured data of the navigation status before the change. It is the speed deviation value calculated based on the measured data of the navigation status before the change. It is the position deviation value calculated based on the measured data of the changed navigation status. It is the speed deviation value calculated based on the measured data of the changed navigation status; The evaluation indicators used in the reward function corresponding to the stable tracking stage include the heading deviation angle and the rudder angle. The heading deviation angle is the deviation value between the heading included in the measured data of the navigation state and the heading of the target track. The reward function corresponding to the stable tracking stage is , It is the heading deviation angle calculated based on the measured data of the navigation status before the change. is the measured data of the navigation status before the change, including the rudder angle, It is the heading deviation angle calculated based on the measured data of the changed navigation status. is the rudder angle included in the measured data of the changed navigation state; The evaluation index used by the reward function corresponding to the end of the navigation phase includes the track end deviation value, which is the distance deviation value between the real-time position included in the measured data of the navigation state and the track end of the target track; the reward function corresponding to the end of the navigation phase is , It is the track end point deviation value calculated based on the measured data of the navigation status before the change. It is the track endpoint deviation value calculated based on the measured data of the changed navigation status.
2. The method according to claim 1, characterized in that The obtaining of the navigation control parameters of the unmanned ship to be controlled based on the control strategy includes: A fractional-order PID controller is constructed according to the control strategy, a navigation error is determined according to the measured navigation state data of the unmanned ship to be controlled and the target track, and the navigation control parameters of the unmanned ship to be controlled are obtained by using the fractional-order PID controller based on the navigation error.
3. The method according to claim 2, characterized in that Any time Navigation control parameters ; in, is the time contained in the measured data of the navigation status The navigation error between the real-time position of and the corresponding target position in the target track, It is the navigation error The integral amount, It is the navigation error The derivative of; parameter array used to construct the fractional-order PID controller According to the control strategy, is the proportional term of the constructed fractional-order PID controller, is the integral term of the constructed fractional-order PID controller, is the differential term of the constructed fractional-order PID controller, is the order of the integral term of the constructed fractional-order PID controller, is the order of the differential term of the constructed fractional-order PID controller.
